Devel::Tokenizer::C 0.05

Devel::Tokenizer::C is a Perl module that can generate C source for fast keyword tokenizer.

SYNOPSIS

use Devel::Tokenizer::C;

$t = new Devel::Tokenizer::C TokenFunc => sub { "return U$_[0];n" };

$t->add_tokens(qw( bar baz ))->add_tokens([for]);
$t->add_tokens([qw( foo )], defined DIRECTIVE);

print $t->generate;

The Devel::Tokenizer::C module provides a small class for creating the essential ANSI C source code for a fast keyword tokenizer.

The generated code is optimized for speed. On the ANSI-C keyword set, its 2-3 times faster than equivalent code generated with the gprof utility.

The above example would print the following C source code:
switch (tokstr[0])
{
case b:
switch (tokstr[1])
{
case a:
switch (tokstr[2])
{
case r:
if (tokstr[3] == )
{ /* bar */
return BAR;
}

goto unknown;

case z:
if (tokstr[3] == )
{ /* baz */
return BAZ;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

case f:
switch (tokstr[1])
{
case o:
switch (tokstr[2])
{
#if defined DIRECTIVE
case o:
if (tokstr[3] == )
{ /* foo */
return FOO;
}

goto unknown;
#endif /* defined DIRECTIVE */

case r:
if (tokstr[3] == )
{ /* for */
return FOR;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

default:
goto unknown;
}

So the generated code only includes the main switch statement for the tokenizer. You can configure most of the generated code to fit for your application.