Free measures of semantic relatedness software for linux

Text::NSP::Measures::2D::Dice::dice is a Perl module to compute Dice coefficient for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::Dice::dice;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$dice_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$dice_value."n"";
}

Assume that the frequency count data associated with a bigram is stored in a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The Dice Coefficient is defined as :

2 * n11
---------
np1 + n1p

The Jaccard coefficient can also be computed by applying a transformation to the dice coefficient:

$jaccard = $dice/(2-$dice)

Methods

calculateStatistic() - method to calculate the dice coefficient value

INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.

RETURN VALUES : $dice .. Dice Coefficient value for this bigram.

getStatisticName() - Returns the name of this statistic

INPUT PARAMS : none

RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D::Fisher2::twotailed is a Perl module implementation of the two-sided Fishers exact test.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::Fisher2::twotailed;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$twotailed_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage();
}
else
{
print getStatisticName."value for bigram is ".$twotailed_value;
}

This module provides a naive implementation of the fishers twotailed exact tests. That is the implementation does not have any optimizations for performance. This will compute the factorials and the hypergeometric measures using direct multiplications.

This measure should be used if you need exact values without any rounding errors, and you are not worried about the performance of the measure, otherwise use the implementations under the Text::NSP::Measures::2D::Fisher module. To use this implementation, you will have to specify the entire module name. Usage:

statistic.pl Text::NSP::Measures::Fisher2::twotailed dest.txt source.cnt
Assume that the frequency count data associated with a bigram < word1 >< word2 > is stored in a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The fishers exact tests are calculated by fixing the marginal totals and computing the hypergeometric probabilities for all the possible contingency tables,
A twotailed fishers test is calculated by adding the probabilities of all the contingency tables with probabilities less than the probability of the observed table. The twotailed fishers test tells us how likely it would be to observe an contingency table which is less prpbable than the current table.

Text::NSP::Measures is a Perl module for computing association scores of Ngrams. This module provides the basic framework for these measures.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::MI::ll;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ll_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n"";
}

Mantissa is a collection of various mathematical tools aimed towards for simulation.
Mantissa contains a collection of algorithms, among which:
a small set of linear algebra classes
a least squares estimator
some curve fitting classes
several ordinary differentials equations integrators, either with fixed steps or adaptive stepsize control (see below)
vectors and rotations in a three dimensional space
algebra-related classes like rational and double polynomials
various orthogonal polynomials:
Chebyshev
Hermite
Laguerre
Legendre
some random numbers and vectors generation classes:
Robert M. Ziff four tap shift register (contributed by Bill Maier)
Makoto Matsumoto and Takuji Nishimura Mersenne twister
generators for vectors with correlated components
some basic (min, max, mean, standard deviation) statistical analysis classes
some optimization algorithms using direct search methods:
the Nelder-Mead simplex method
Virginia Torczons multi-directional method
Enhancements:
- For many basic objects provided by Mantissa like Vector3D, Rotation, and the various Polynomial classes, instances are now guaranteed to be immutable.
- This greatly simplifies safe sharing of instances without forcing users to either put severe restrictions on their use of Mantissa classes or make numerous copies just to make sure everything is safe.
- Since the change is a semantic change on the contract of the classes, this version introduces some incompatibilities with respect to previous ones.
- Upgrading to this version is not difficult, though.

WebService::GoogleHack::Rate is a Perl module that implements a simple relatedness measure and semantic orientation related type functions.

SYNOPSIS

use WebService::GoogleHack::Rate;

#GIVE PATH TO INPUT FILE HERE

my $INPUTFILE="";

#GIVE PATH TO TRACE FILE HERE

my $TRACEFILE="";

#create an object of type Rate

my $rate = WebService::GoogleHack::Rate->new();

$results=$rate->measureSemanticRelatedness1("dog", "cat");

#The PMI measure is stored in the variable $results, and it can also
#be accessed as $rate->{PMI};

$results=$rate->predictSemanticOrientation($INPUTFILE, "excellent", "bad",$TRACEFILE);

#The resutls can be accessed through
print $results->{prediction}."n";
$results->{PMI Measure}."n";
$rate->{prediction} &."n";
$rate->{PMI Measure}."n";

WebService::GoogleHack::Rate - This package uses Google to do some basic natural language processing. For example, given two words, say "knife" and "cut", the module has the ability to retrieve a semantic relatedness measure, commonly known as the PMI (Pointwise mututal information) measure. The larger the measure the more related the words are. The package can also predict the semantic orientation of a given paragraph of english text. A positive measure means that the paragraph has a positive meaning, and negative measure means the opposite.

Text::NSP::Measures::2D::CHI::tscore is a Perl module that implements T-score measure of association for bigrams.

SYNOPSIS

Basic Usage

use Text::NSP::Measures::2D::CHI::tscore;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$tscore_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$tscore_value."n"";
}

Assume that the frequency count data associated with a bigram < word1 >< word2 > is stored in a 2x2 contingency table:

word2 ~word2
word1 n11 n12 | n1p
~word1 n21 n22 | n2p
--------------
np1 np2 npp

where n11 is the number of times < word1 >< word2 > occur together, and n12 is the number of times < word1 > occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The T-score is defined as a ratio of difference between the observed and the expected mean to the variance of the sample. Note that this is a variant of the standard t-test that was proposed for use in the identification of collocations in large samples of text.

Thus, the T-score is defined as follows:

m11 = n1p * np1 / npp

T-score = (n11 - m11)/sqrt(n11)

calculateStatistic() - method to calculate the tscore Coefficient

INPUT PARAMS : $count_values .. Reference of an hash containing the count values computed by the count.pl program.
RETURN VALUES : $tscore .. tscore value for this bigram.
getStatisticName() - Returns the name of this statistic
INPUT PARAMS : none
RETURN VALUES : $name .. Name of the measure.

Text::NSP::Measures::2D is a Perl module that provides basic framework for building measure of association for bigrams.

SYNOPSIS

Basic Usage
use Text::NSP::Measures::2D::MI::ll;

my $npp = 60; my $n1p = 20; my $np1 = 20; my $n11 = 10;

$ll_value = calculateStatistic( n11=>$n11,
n1p=>$n1p,
np1=>$np1,
npp=>$npp);

if( ($errorCode = getErrorCode()))
{
print STDERR $errorCode." - ".getErrorMessage()."n"";
}
else
{
print getStatisticName."value for bigram is ".$ll_value."n"";
}

This module is to be used as a foundation for building 2-dimensional measures of association. The methods in this module retrieve observed bigram frequency counts, marginal totals, and also compute expected values. They also provide error checks for these counts.