Free measures software for linux

Text::NSP::Measures is a Perl module for computing association scores of Ngrams. This module provides the basic framework for these measures.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::ll;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ll_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n"";
}

Text::NSP::Measures::3D is a Perl module that provides basic framework for building measure of association for trigrams.

This module can be used as a foundation for building 3-dimensional measures of association that can then be used by statistic.pl. In particular this module provides methods that give convenient access to 3-d (i.e., trigram) frequency counts as created by count.pl, as well as some degree of error handling that verifies the data.

Basic Usage

use Text::NSP::Measures::3D::MI::ll;

$ll_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n";
}

The methods in this module retrieve observed trigram frequency counts and marginal totals, and also compute expected values. They also provide support for error checking of the output produced by count.pl. These methods are used in all the trigram (3d) measure modules provided in NSP. If you are writing your own 3d measure, you can use these methods as well.

Text::NSP::Measures::3D::MI is a Perl module that provides error checks and framework to implement Loglieklihood, Total Mutual Information, Pointwise Mutual Information and Poisson Stirling Measure for trigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::3D::MI::ll;

$ll_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n";
}

This module is the base class for the Loglikelihood and the True Mutual Information measures. All these measure are similar. This module provides error checks specific for these measures, it also implements the computations that are common to these measures.

Log-Likelihood measure is computed as
Log-Likelihood = 2 * [n111 * log(n111/m111) + n112 * log(n112/m112) +

n121 * log(n121/m121) + n122 * log(n122/m122) +
n211 * log(n211/m211) + n212 * log(n212/m212) +
n221 * log(n221/m221) + n222 * log(n222/m222)]

Total Mutual Information

tmi = [n111/nppp * log(n111/m111) + n112/nppp * log(n112/m112) + n121/nppp * log(n121/m121) + n122/nppp * log(n122/m122) + n211/nppp * log(n211/m211) + n212/nppp * log(n212/m212) + n221/nppp * log(n221/m221) + n222/nppp * log(n222/m222)]

Pointwise Mutual Information

pmi = log (n111/m111)

Poisson Stirling Measure

ps = n111 * ( log(n111/m111) - 1)

All these methods use the ratio of the observed values to expected values, for computations, and thus have common error checks, so they have been grouped togrther.

Text::NSP::Measures::3D::MI::ps is a Perl module that implements Poisson Stirling Measure for trigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::3D::MI::ps;

$ps_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$ps_value."n";
}

The log-likelihood ratio measures the devitation between the observed data and what would be expected if < word1 >, < word2 > and < word3 > were independent. The higher the score, the less evidence there is in favor of concluding that the words are independent.

The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

n1pp * np1p * npp1
m111= --------------------
nppp

The poisson stirling measure is a negative lograthimic approximation of the poisson-likelihood measure. It uses the stirlings firmula to approximate the factorial in poisson-likelihood measure. It is computed as follows:
Posson-Stirling = n111 * ( log(n111) - log(m111) - 1)

Methods

calculateStatistic() - This method calculates the ps value
INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.

RETURN VALUES : $poissonStirling .. Poisson-Stirling value for this trigram.

getStatisticName() - Returns the name of this statistic

INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D::MI::ps is a Perl module that implements Poisson-Stirling measure of association for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::ps;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ps_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ps_value."n"";
}

The log-likelihood ratio measures the devitation between the observed data and what would be expected if < word1 > and < word2 > were independent. The higher the score, the less evidence there is in favor of concluding that the words are independent.

Assume that the frequency count data associated with a bigram < word1 >< word2 > as shown by a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

np1 * n1p
m11= ---------
npp

The poisson stirling measure is a negative lograthimic approximation of the poisson-likelihood measure. It uses the stirlings firmula to approximate the factorial in poisson-likelihood measure.

Posson-Stirling = n11 * ( log(n11) - log(m11) - 1)

which is same as

Posson-Stirling = n11 * ( log(n11/m11) - 1)

Methods

calculateStatistic() - This method calculates the ps value

INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.

RETURN VALUES : $poissonStirling .. Poisson-Stirling value for this bigram.

getStatisticName() - Returns the name of this statistic

INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::3D::MI::tmi is a Perl implementation for True Mutual Information for trigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::3D::MI::tmi;

$tmi_value = calculateStatistic( n111=>10,
n1pp=>40,
np1p=>45,
npp1=>42,
n11p=>20,
n1p1=>23,
np11=>21,
nppp=>100);

if( ($errorCode = getErrorCode()))
{
print STDERR $erroCode." - ".getErrorMessage()."n";
}
else
{
print getStatisticName."value for bigram is ".$tmi_value."n";
}

True Mutual Information (tmi) is defined as the weighted average of the pointwise mutual informations for all the observed and expected value pairs.

tmi = [n111/nppp * log(n111/m111) + n112/nppp * log(n112/m112) +
n121/nppp * log(n121/m121) + n122/nppp * log(n122/m122) +
n211/nppp * log(n211/m211) + n212/nppp * log(n212/m212) +
n221/nppp * log(n221/m221) + n222/nppp * log(n222/m222)]

PMI = log (n111/m111)

Here n111 represents the observed value for the cell (1,1,1) and m111 represents the expected value for that cell. The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

n1pp * np1p * npp1
m111= --------------------
nppp
Methods

calculateStatistic($count_values) - This method calculates the tmi value

INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.
RETURN VALUES : $tmi .. TMI value for this trigram.
getStatisticName() - Returns the name of this statistic
INPUT PARAMS : none
RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D::MI::tmi is a Perl module that implements True Mutual Information.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::tmi;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$tmi_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$tmi_value."n"";
}

Assume that the frequency count data associated with a bigram < word1 >< word2 > is stored in a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:

np1 * n1p
m11= ---------
npp

True Mutual Information (tmi) is defined as the weighted average of the pointwise mutual informations for all the observed and expected value pairs.

tmi = [n11/npp * log(n11/m11) + n12/npp * log(n12/m12) +
n21/npp * log(n21/m21) + n22/npp * log(n22/m22)]


PMI = log (n11/m11)

Methods

calculateStatistic() - This method calculates the tmi value
INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.
RETURN VALUES : $tmi .. TMI value for this bigram.

getStatisticName() - Returns the name of this statistic
INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D::Fisher::left is a Perl module implementation of the left sided Fishers exact test.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::Fisher::left;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$left_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage();
}
else
{
print getStatisticName."value for bigram is ".$left_value;
}