Free to measure software for linux

Text::NSP::Measures::2D is a Perl module that provides basic framework for building measure of association for bigrams.

SYNOPSIS

Basic Usage
use Text::NSP::Measures::2D::MI::ll;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ll_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n"";
}

This module is to be used as a foundation for building 2-dimensional measures of association. The methods in this module retrieve observed bigram frequency counts, marginal totals, and also compute expected values. They also provide error checks for these counts.

Measuring Buffer 20090628 provides you with a perfect and enhanced version of buffer which features display of throughput, network support, memory-mapped file I/O for huge buffers and multithreading. This will be your excellent choice.

Text::NSP::Measures::3D::MI is a Perl module that provides error checks and framework to implement Loglieklihood, Total Mutual Information, Pointwise Mutual Information and Poisson Stirling Measure for trigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::3D::MI::ll;

$ll_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n";
}

This module is the base class for the Loglikelihood and the True Mutual Information measures. All these measure are similar. This module provides error checks specific for these measures, it also implements the computations that are common to these measures.

Log-Likelihood measure is computed as
Log-Likelihood = 2 * [n111 * log(n111/m111) + n112 * log(n112/m112) +

n121 * log(n121/m121) + n122 * log(n122/m122) +
n211 * log(n211/m211) + n212 * log(n212/m212) +
n221 * log(n221/m221) + n222 * log(n222/m222)]

Total Mutual Information

tmi = [n111/nppp * log(n111/m111) + n112/nppp * log(n112/m112) + n121/nppp * log(n121/m121) + n122/nppp * log(n122/m122) + n211/nppp * log(n211/m211) + n212/nppp * log(n212/m212) + n221/nppp * log(n221/m221) + n222/nppp * log(n222/m222)]

Pointwise Mutual Information

pmi = log (n111/m111)

Poisson Stirling Measure

ps = n111 * ( log(n111/m111) - 1)

All these methods use the ratio of the observed values to expected values, for computations, and thus have common error checks, so they have been grouped togrther.

Text::NSP::Measures::2D::MI::ll is a Perl module that implements Loglikelihood measure of association for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::ll;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ll_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage();
}
else
{
print getStatisticName."value for bigram is ".$ll_value;
}

Text::NSP::Measures::2D::MI::ps is a Perl module that implements Poisson-Stirling measure of association for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::ps;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ps_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ps_value."n"";
}

The log-likelihood ratio measures the devitation between the observed data and what would be expected if < word1 > and < word2 > were independent. The higher the score, the less evidence there is in favor of concluding that the words are independent.

Assume that the frequency count data associated with a bigram < word1 >< word2 > as shown by a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

np1 * n1p
m11= ---------
npp

The poisson stirling measure is a negative lograthimic approximation of the poisson-likelihood measure. It uses the stirlings firmula to approximate the factorial in poisson-likelihood measure.

Posson-Stirling = n11 * ( log(n11) - log(m11) - 1)

which is same as

Posson-Stirling = n11 * ( log(n11/m11) - 1)

Methods

calculateStatistic() - This method calculates the ps value

INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.

RETURN VALUES : $poissonStirling .. Poisson-Stirling value for this bigram.

getStatisticName() - Returns the name of this statistic

INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::3D::MI::pmi is a Perl module that implements Pointwise Mutual Information for trigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::3D::MI::pmi;

$pmi_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$pmi_value."n";
}

The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

n1pp * np1p * npp1
m111= --------------------
nppp

Pointwise Mutual Information (pmi) is defined as the log of the devitation between the observed frequency of a trigram (n111) and the probability of that trigram if it were independent (m111).

PMI = log (n111/m111)

Methods

initializeStatistic() -Initialization of the pmi_exp parameter if required

INPUT PARAMS : none

RETURN VALUES : none

calculateStatistic() - This method calculates the pmi value

INPUT PARAMS : $count_values .. Reference of a hash containing the count values computed by the count.pl program.

RETURN VALUES : $pmi .. PMI value for this trigram.

getStatisticName() - Returns the name of this statistic

INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D::CHI::phi is a Perl module that implements Phi coefficient measure for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::CHI::phi;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$phi_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$phi_value."n"";
}