Free whitespace software for linux

HTML-Strip-Whitespace is a Perl module to strip whitespace out of HTML pages. As opposed to other solutions (like HTML-Clean), it does not touch whitespace inside < pre > tags (and in the future possibly other whitespace-aware markup.)

Using the HTML-Tidy (or otherwise the HTML Tidy library) for that may be preferable, because they are probably faster, but this module still exists and will be maintained.

jTokeniser project is a Java library for tokenising strings into a list of tokens.
Main features:
- WhiteSpaceTokeniser - this splits a string on all occurances of whitespace, which include spaces, newlines, tabs and linefeeds.
- StringTokeniser - this is basically the same as java.util.StringTokenizer with some extra methods (and extends from Tokeniser). Its default behaviour is to act as a WhiteSpaceTokeniser, however, you can specify a set of characters that are to be used to indicate word delimiters.
- RegexTokeniser - this tokeniser is much more flexible as you can use regular expressions to define a what a token is. So, "w+" means whenever it matches one or more letters, it will consider that a word. By default, it uses a regular expression equivalent to a whitespace tokeniser.
- RegexSeparatorTokeniser - this can be thought of as an advanced StringTokeniser. Whereas StringTokeniser is limited to defining delimiters as a set of individual characters, RegexSeparatorTokeniser can utilise regular expressions for a richer and more flexible approach.
- BreakIteratorTokeniser - one of the most sophisticated tokenisers in the library, although should only be used on natural language strings to isolate words. It also comes with built-in rules about how to find words, knowing how to disregard punctuation, etc.
- SentenceTokeniser - this also uses a BreakIterater like the above, but tuned towards finding sentence boundaries. The "tokens" in this tokeniser are in fact individual sentences.
Enhancements:
- This release includes an easy to use GUI front-end to use the tokenisers interactively, out-of-the-box.
- This is especially useful for experimenting with tokenisers, perhaps within a teaching environment.
- It is also handy for those without the Java experience to utilise the library API directly.

ShiftJIS::Regexp contains regular expressions in Shift-JIS.

SYNOPSIS

use ShiftJIS::Regexp qw(:all);

match($string, p{Hiragana}{2}p{Digit}{2});
match($string, pH{2}pD{2});
# these two are equivalent:

This module provides some functions to use regular expressions in Shift-JIS on the byte-oriented perl.

The legal Shift-JIS character in this module must match the following regular expression:

[x00-x7FxA1-xDF]|[x81-x9FxE0-xFC][x40-x7Ex80-xFC]

To avoid false matching in multibyte encoding, this module uses anchoring technique to ensure each matching position places at the character boundaries.

cf. perlfaq6, "How can I match strings with multibyte characters?"

Functions

re(PATTERN)

re(PATTERN, MODIFIER)

Returns a regular expression parsable by the byte-oriented perl.

PATTERN is specified as a string. MODIFIER is specified as a string. Modifiers in the following list are allowed.

i case-insensitive pattern (only for ascii alphabets)
I case-insensitive pattern (greek, cyrillic, fullwidth latin)
j hiragana-katakana-insensitive pattern (but halfwidth katakana
are not considered.)

s treat string as single line
m treat string as multiple lines
x ignore whitespace (i.e. [x20nrtf]) unless backslashed
or inside a character class; but comments are not recognized!

o once parsed (not compiled!) and the result is cached internally.
o modifier
while ( ) {
print replace($_, (perl), $1, igo);
}
is more efficient than

while ( ) {
print replace($_, (perl), $1, ig);
}

because in the latter case the pattern is parsed every time
whenever the function is called.

match(STRING, PATTERN)

match(STRING, PATTERN, MODIFIER)

An emulation of m// operator aware of Shift-JIS. But, to emulate @list = $string =~ m/PATTERN/g, the pattern should be parenthesized (capturing parentheses are not added automatically).

@list = match($string, pH, g); # wrong; returns garbage!
@list = match($string,(pH),g); # good
PATTERN is specified as a string. MODIFIER is specified as a string.
i,I,j,s,m,x,o please see re().

g match globally
z tell the function the pattern matches an empty string
(sorry, due to the poor auto-detection)

replace(STRING or SCALAR REF, PATTERN, REPLACEMENT)

replace(STRING or SCALAR REF, PATTERN, REPLACEMENT, MODIFIER)

An emulation of s/// operator but aware of Shift-JIS.
If a reference to a scalar is specified as the first argument, substitutes the referent scalar and returns the number of substitutions made. If a string (not a reference) is specified as the first argument, returns the substituted string and the specified string is unaffected.

MODIFIER is specified as a string.

i,I,j,s,m,x,o please see re().
g,z please see match().

jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING)

jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING, LIMIT)

An emulation of CORE::split but aware of Shift-JIS.
In scalar/void context, it does not split into the @_ array; in scalar context, only returns the number of fields found.
PATTERN is specified as a string. But as PATTERN has no special meaning; it splits the string on a single space similarly to CORE::split / /.

When you want to split the string on whitespace, pass an undefined value as PATTERN or use the splitspace() function.

jsplit(undef, " x81x40 This is x81x40 perl.");
splitspace(" x81x40 This is x81x40 perl.");
# (This, is, perl.)

If you want to pass pattern with modifiers, specify an arrayref of [PATTERN, MODIFIER] as the first argument. You can also use "Embedded Modifiers").

MODIFIER is specified as a string.

i,I,j,s,m,x,o please see re().

splitspace(STRING)

splitspace(STRING, LIMIT)

This function emulates CORE::split( , STRING, LIMIT). It returns a list given by split STRING on whitespace including "x81x40" (IDEOGRAPHIC SPACE). Leading whitespace characters do not produce any field.

Note: splitspace(STRING, LIMIT) is equivalent to jsplit(undef, STRING, LIMIT).

splitchar(STRING)

splitchar(STRING, LIMIT)

This function emulates CORE::split(//, STRING, LIMIT). It returns a list given by split of STRING into characters.
Note: splitchar(STRING, LIMIT) is equivalent to jsplit(, STRING, LIMIT).

wsdebug is a debugger for the more or less famous whitespace programming language, coming along with a rather fast interpreter (wsi).

Most programming languages like C or Perl do not care for white space characters (like tab, space or newline/linefeed). The whitespace programming language works just the other way round, dont care for any character but those white space ones.

On the whole its just another geeky language like Brainfuck and others, however more adicting.

If youve written a whole lot of instructions youll probably reach the point, where you get lost. Then just put your script into wsdebug and step through your bunch of whitespace instructions and watch how each command manipulates the stack (or heap).

DParser project is an simple but powerful tool for parsing. You can specify the form of the text to be parsed using a combination of regular expressions and grammar productions.
Because of the parsing technique (technically a scannerless GLR parser based on the Tomita algorithm) there are no restrictions.
The grammar can be ambiguous, right or left recursive, have any number of null productions, and because there is no seperate tokenizer, can include whitespace in terminals and have terminals which are prefixes of other terminals.
DParser handles not just well formed computer languages and data files, but just about any wacky situation that occurs in the real world.
Main features:
- Powerful GLR parsing
- Simple EBNF-style grammars and regular expression terminals
- Priorities and associativities for token and rules
- Built-in error recovery
- Speculative actions (for semantic disambiguation)
- Auto-building of parse tree (optionally)
- Final actions as you go, or on the complete parse tree
- Tree walkers and default actions (multi-pass compilation support)
- Symbol table built for ambiguous parsing
- Partial parses, recursive parsing, parsing starting with any non-terminal
- Whitespace can be specified as a subgrammar
- External (C call interface) tokenizers and external terminal scanners
- Good asymptotically efficiency
- Comes with ANSI-C, Python and Verilog grammars
- Comes with full source
- Portable C for easy compilation and linking
- BSD licence, so you can included it in your application without worrying about licensing
Enhancements:
- Removed call to exec in python interface (Brian Sabbey)
- Fix binary_op_left in python interface (Brian Sabbey)

XML::Filter::DataIndenter is a SAX2 Indenter for data oriented XML.

SYNOPSIS

use XML::Filter::DataIndenter;

use XML::SAX::Machines qw( Pipeline );

Pipeline( XML::Filter::DataIndenter => *STDOUT );

ALPHA CODE ALERT: This is the first release. Feedback and patches welcome.
In data oriented XML, leaf elements (those which contain no elements) contain only character content, all other elements contain only child elements and ignorable whitespace. This filter consumes all whitespace not in leaf nodes and replaces it with whitespace that indents all elements. Character data in leaf elements is left unmolested.

This filter assumes youre emitting data oriented XML. It will die if it sees non-whitespace character data outside of a leaf element. It also dies if it sees start-tag / end-tag mismatch, just as a service to the programmer.
Processing instructions and comments are indented as though they were leaf elements except when they occur in leaf elements.

Example:

This document:

< a>< ?A?>
< !--A-->< b>< ?B?>< !--B-->B< /b>
< !--A-->
< /a>

gets reindented as:

< a>
< ?A?>
< !--A-->
< b>< ?B?>< !--B-->B< /b>
< !--A-->
< /a>

(plus or minus a space in each PI, depending on your XML writer).

Geography::States is a Perl module with map states and provinces to their codes, and vice versa.

SYNOPSIS

use Geography::States;

my $obj = Geography::States -> new (COUNTRY [, STRICT]);

EXAMPLES

my $canada = Geography::States -> new (Canada);

my $name = $canada -> state (NF); # Newfoundland.
my $code = $canada -> state (Ontario); # ON.
my ($code, $name) = $canada -> state (BC); # BC, British Columbia.
my @all_states = $canada -> state; # List code/name pairs.

This module lets you map states and provinces to their codes, and codes to names of provinces and states.

The Geography::States - new ()> call takes 1 or 2 arguments. The first, required, argument is the country we are interested in. Current supported countries are USA, Brazil, Canada, The Netherlands, and Australia. If a second non-false argument is given, we use strict mode. In non-strict mode, we will map territories and alternative codes as well, while we do not do that in strict mode. For example, if the country is USA, in non-strict mode, we will map GU to Guam, while in strict mode, neither GU and Guam will be found.

The state() method

All queries are done by calling the state method in the object. This method takes an optional argument. If an argument is given, then in scalar context, it will return the name of the state if a code of a state is given, and the code of a state, if the argument of the method is a name of a state. In list context, both the code and the state will be returned.

If no argument is given, then the state method in list context will return a list of all code/name pairs for that country. In scalar context, it will return the number of code/name pairs. Each code/name pair is a 2 element anonymous array.

Arguments can be given in a case insensitive way; if a name consists of multiple parts, the number of spaces does not matter, as long as there is some whitespace. (That is "NewYork" is wrong, but "new YORK" is fine.)