Huge piles of information accumulate in numerous text repositories held at news agencies, libraries, corporations, individual PCs, and the Web. The amount of stored information proliferates at a disastrous rate, and the human eyes and brain are increasingly unable to meet the challenges of this growth. Mankind is searching for intelligent electronic assistants to help with text analysis projects.

One needs to:

• Distill the meaning of a text in a concise form
• View accurate summaries before plunging into full documents
• Efficiently navigate through large textbases
• Perform natural language information retrieval

These and many other tasks can be successfully tackled by TextAnalyst, a unique software tool for semantic analysis, navigation, and search of unstructured texts.

TextAnalyst Functionality

A synergy of unique linguistic and neural network technologies implemented in TextAnalyst ensures high speed and accuracy in the analysis of unstructured texts.

Distilling the meaning of a text
Formation and export of an accurate Semantic Network of the text or textbase. This network concisely represents the meaning of a text and serves as a basis for all further analysis.

Accurate summarization of texts
The quality of the summary is provided by a balanced combination of linguistic and neural network investigation methods. The size of the summary is controlled through the semantic weight threshold.

Subject-focused text exploration
User-specified dictionaries of excluded and included words allow the investigation to focus on a chosen subject.

Efficient navigation through a textbase
The knowledge base can be navigated with hyperlinks from concepts in the Semantic Network to sentences in the documents that contain the considered combination of concepts. Individual sentences are in turn hyperlinked to those places in original texts where they have been encountered.

Explication of the text theme structure
A tree-like topic structure representing the semantics of the investigated texts is automatically developed. The more important subjects are placed closer to the root of a tree.

Clustering of texts
Breaking links representing weak relations in the original Semantic Network enables clustering of the textbase.

Semantic information retrieval
Natural language queries are analyzed for semantically important words and all relevant sentences from the textbase documents are retrieved. In addition, a subtree of concepts related to the query is formed, which facilitates a simple search refinement.

Platforms: MS Windows

Users
Existing users of TextAnalyst include government offices, consulting and law firms, medical centers, scientific organizations, electronic book publishers, customer support centers, political institutions, and even college students.

Potential users:

• Journal Editors
• Press Analysts
• Researchers
• Scientists
• Political Analysts
• Investment Bankers
• Lawyers
• Venture Capitalists

The TextAnalyst COM software development kit contains:

• A library of software components (Inproc Servers)
• The corresponding documentation
• A set of basic dictionaries required for working with English texts
• A program for editing dictionaries DicEdit

The provided TextAnalyst COM modules deliver the following functionality:

• Word stemming and transforming to a generic form
• Creating a frequency-based dictionary of concepts from the text (including words and word combinations), with a filtration of words from a given stoplist
• Derivating of lists of words semantically related to the selected words in the investigated text
• Numerical evaluation of the semantic significance of different concepts, and the weights of relations between these words, in the investigated text
• Retrieving a set of sentences from the text, which contain the specified words
• Forming a text summary the list of most important sentences from a text

The principles and structure of TextAnalyst COM

• Document analysis scheme
• Text preprocessing module
• Text analysis module

All the functions of TextAnalyst COM are supported through the cooperation of two program components:

• Text preprocessing module Linguistic Processor (LP)
• Text analysis module Algorithmic Core (AC)

The components exchange services with each other, as well as with external applications through COM architecture, thus providing the necessary functionality through standardized intefaces.

The text from an analyzed document is fed to the LP module. There a sequence of words with their attributes is extracted from the text according to linguistic rules and specified dictionaries. Then this sequence of words is passed to the AC module, where semantic analysis is performed with the help of neural network technology. The AC module creates a knowledge base that contains all the information about the text structure, which serves as a basis for carrying out retrieval and analysisfunctions requested by the user. All further work is perform with this knowledge base rather than with original texts. This significantly increases the speed of the undertaken analysis.

Each module has a set of interface methods implementing separate functions involved in the analysis, controlling the utilized settings, and fetching the required information. Combining these methods the user can implement practically any semantic text analysis function required for an external application.