MUSCLE Network of Excellence

Text Analysis Tools

Updatable Probabilistic Latent Semantic Indexing

AUTH, Constantine Kotropoulos

Probabilistic latent semantic indexing (PLSI) is a semantic space reduction method that folds documents and the concepts  that appear in them into a smaller dimensioned semantic space which can then be used to index and classify new  documents. Building a reduced semantic space is time consuming, order O(N^3). AUTH has implemented a new method for updating PLSI when new documents arrive. The new method incrementally adds the words of any new document in the term-document matrix  and derives the updating equations for the probability of terms given the class (i.e. latent) variables, as well those of documents given the latent variables. This quick updating would be useful in a web crawler where the term-document matrix must be refreshed very often.

Website: http://www.aiia.csd.auth.gr/EN/

Presentation in MUSCLE BSCW

SOMLib Java Package

TU-WIEN - IFS, Andreas Rauber

TU Vienna - IFS has developed a software for analyzing text documents and organizing them on a Self Organizing Map (SOM) - a representation of a reduced semantic dimension, bringing similar documents, or objects closer together on a two or three dimensional plane. The SOMLib Java Package is a collection of JAVA programs that can be used to create SOMLib library systems for organizing text collections. The package includes 

• Feature Extraction
• Feature space pruning
• Feature vector creation
• Feature vector normalization
• SOM training
• SOM Labeling
• libViewer template generation. 

Website: http://www.ifs.tuwien.ac.at/~andi/somlib/download/index.html

Quick Reference: http://www.ifs.tuwien.ac.at/~andi/somlib/download/java_package/

Semi-automated Corpus Annotator (CNRS LLACAN)

CNRS, Fathi Debili

Automatic versus Interactive Analysis of Arabic Corpora

The tools we present make it possible to interactively annotate large corpora of Arabic texts.
These annotations involve splitting text into words, lemmatization, vowellization, tagging, segmentation into nominal and verbal chains, along with the construction of dependency relations.

Within this process, interactive annotation serves automatic parsing by massively providing it with annotated texts from which rules can be learned and evaluated. Automatic parsing serves interactive annotation whose correlated performance is to be measured by comparison to manual processing and its cost.

The language tools of CNRS LLACAN relate to the automatic processing of the Arabic language. Based on a dictionary of forms, they enable morphological analysis, POS tagging, phrase chunking and dependency analysis of Modern Standard Arabic, with variable levels of coverage and performance.

The required production of large training corpora and the difficulties specific to the Arabic language led to the realization of interactive analysis tools.

These tools are now operating under MS Windows. An intranet version is being developed.

These tools have also been used in the preparation of a tagged corpus of about 250000 words (available from ELDA).

Presentation in MUSCLE BSCW

UTIA Text Classifier

UTIA, Jana Novovicova

Text categorization (also known as text classification) is the task of automatically sorting a set of documents into predefined classes based on its contents. Document classification is needed in many applications including e-mail filtering, mail routing, spam filtering, news monitoring, selective dissemination of information to information consumers, and automated indexing of scientific articles. The Prague-based team of UTIA has produced a method for text classification using Oscillating Search which, unlike traditional approaches, evaluates feature groups instead of individuals and which improves classification accuracy in experiments.

Paper describing the work: http://staff.utia.cas.cz/novovic/files/CIARP06_NSP.pdf

ECUE Spam Concept Drift Datasets

• SpamTraining.txt = all emails used as initial training data in the concept drift experiments performed using this dataset.
• NonspamTraining.txt = all emails used as initial training data in the concept drift experiments performed using this dataset.
• TestMMM99.txt = all emails used as 'test' data in the concept drift experiments using this dataset where MMM represents the month and 99 the year the emails were originally received.

TechTC - Repository of Text Categorization Datasets

1. Plain text  In plain text form, each dataset consists of a pair of files corresponding to the two categories comprising the dataset. Each file contains all the documents in one category in ASCII text format, which resulted from HTML-to-text conversion.
2. Preprocessed feature vectors. In this format, texts were only tokenized and digitized, but underwent no other preprocessing whatsoever.

• TechTC-300 - a collection of 300 datasets whose categorization difficulty (as measured by baseline SVM accuracy) is uniformly distributed between 0.6 and 1.0.
• TechTC-100 - a collection of 100 datasets whose categorization difficulty (as measured by baseline SVM accuracy) is uniformly distributed between 0.6 and 0.92.
  Note: TechTC-100 is a subset of TechTC-300.