Department of Software Technology
Vienna University of Technology

The SOMLib Digital Library - Experiments - Der Standard

Overview

For the experiments described below we use a collection of articles from the daily Austrian Newspaper Der Standard . (Special thanks for providing us with the article collection and allowing us to use them for our experiments).

On this page:
  1. Data
  2. Textrepresentation
  3. Trained Growing hierarchical Self-Organizing Maps with Labels

Data

The complete text collection consists of almost 50.000 documents. The following experiments use a subset of all articles, covering the second quarter of 1999, i.e. the months April, May and June 1999. This subset consists of 11,627 articles. All HTML tags were removed from the articles to obtain text-only representations suitable for content analysis by the SOMLib system. Furthermore, experiments using the full dataset are presented.


Text Representation

To be used for map training, a vector-space representation of the single documents is created by full-text indexing. For each document collection a list of all words appearing in the respective collection is extracted while applying some basic word stemming techniques. Words that do not contribute to contents description are removed from these lists. Instead of defining language or content specific stop word lists, we rather discard terms that appear in more than 813 (7%) or in less than 65 articles (0.56%). We thus end up with a vector dimensionality of 3.799 unique terms. The individual documents are then represented by feature vectors using a tf x idf, i.e. term frequency times inverse document frequency, weighting scheme as described by Salton. This weighting scheme assigns high values to terms that are important as to describe and discriminate between the documents. The 11,627 articles thus are represented by automatically extracted 3,799-dimensional feature vectors of word histograms weighted by a tf x idf weighting scheme and normalized to unit length. The listing below provides the template vector, i.e. the list of words used for document representation, the list of removed "stop-words" as well as the feature vectors used for training the maps of the subset of the article collection.
Furthermore, for more extensive experiments, the complete collection of 47.684 articles was used. Words appearing in less than 0.9%, i.e. in less than 429 articles, or in more than 8% of all articles (i.e. in more than 3.814 articles) were removed from the featurelist, resulting in a 2.487-dimensional feature vector (down from an initial 438.313 words).


Document Clustering:

Trained Growing Hierarchical Self-Organizing Maps with Labels

With documents collections of this size, providing single, flat SOMs does not offer a convenient interface anymore as the resulting maps would become too large. We thus use our new Growing Hierarchical Self-Organizing Map (GHSOM) model to create a hierarchical representation of the document archive. Based on the document description as outlined above, we trained several growing hierarchical self-organizing maps to represent the contents of the document archive. Using the labelSOM method, characteristic keywords were automatically extracted from the trained maps, describing the various topical clusters.

GHSOMs with a subset of the article collection: 2. Quarter 1999


A brief discussion of the results

Training the GHSOM with parameters t1=0.035 and t2=0.0035 results in the more shallow hierarchical structure of up to 7 layers provided as GHSOM 2. The layer 1 map grows to a size of 7 x 4 units, all of which are expanded at subsequent layers. We find the most dominant branches to be, for example, Sports, located in the upper right corner of the map, Internal Affairs in the lower right corner, Internet-related articles on the left hand side of the map, to name but a few. However, due to the large size of the resulting first layer map, a fine-grained representation of the data is already provided at this layer. This results in some larger clusters to be represented by two neighboring units already at the first layer, rather than being split up in a lower layer of the hierarchy. For example, we find the cluster on Internal Affairs to be represented by two neighboring units. One of these, on position (6/4), covers solely articles related to the Freedom Party and its political leader Jörg Haider, representing one of the most dominant political topics in Austria for some time now, resulting in an accordingly large number of news articles covering this topic. The neighboring unit to the right, i.e.located in the lower right corner on position (7/4), covers other Internal Affairs, with one of the main topics being the elections to the European Parliament.

The topics of articles covered by these two maps are closely related. We also find, for example, articles related to the Freedom Party on the branch covering the more general Internal Affairs, reporting on their role and campaigns for the elections to the European Parliament. As might be expected these are closely related to the other articles on the Freedom Party which are located in the neighboring branch. Obviously, we would like them to be presented on the left hand side of this map, so as to allow the transition from one map to the next, with a continuous orientation of topics.
Due to the initialization of the added maps during the training process, this continuous orientation is preserved, as can easily be seen from the automatically extracted labels on the two maps. Continuing from the second layer map of unit (6/4) to the right we reach the according second layer map of unit (7/4) where we first find articles focusing on the Freedom Party, before moving on to the Social Democrats, the People's Party, the Green Party and the Liberal Party. As all units at layer two in these branches have a quantization error below 42.63, no unit is further expanded at a third layer.

A similar situation is encountered with the Sports cluster, being represented by 3 units in the upper right corner on the first layer map, which specialize on Tennis and International Soccer - UEFA-Cup on unit (5/1), Austrian Soccer on unit (6/1), and other sports articles, such as short result listings on unit (7/1). Again, we find the second layer maps organized accordingly, i.e. for example we find the reports on International Soccer located on the lower right side of the second layer map, neighboring the other second layer map representing solely Austrian Soccer.

GHSOMs with the complete article collection of 1999

As a more extensive experiment, we created a GHSOM of the full article collection, comprising 47.684 articles represented by 2.487-dimensional feature vectors.
The top-layer map evolved to a size of 2 x 3 units. Please note, how a very concise cluster of articles in the upper right corner is already sufficiently represented at the top-layer map, resulting in this cluster not being further expanded at subsequent layers. The articles on this cluster represent weather reports. In spite of their large number (and thus a rather unfavourably layout of the resulting table), they do not require further detailed representation.
Next to it, in the upper left corner, we find another very concise cluster, representing articles on cultural events, listings of theater performances ("watchlist" etc.). This cluster is represented in more detail on a second-layer map of 5 x 6 units. again, the degree of representation at this level is sufficient, and the second-layer map in this branch is not expanded any further.
A similar situation can be found with a cluster of articles on German national politics originating from the the lower right corner of the top-layer map, or a cluster of sports articles originating from the unit above it.
The majority of articles are represented in greater detail in the branch originating from the lower right corner of the top-layer map. This second layer map expanded into 6 x 6 units, of which most units were expanded into several further layers of the hierarchy, whereas others did not require further expansion.

Publications

Some more detailed descriptions of experiments using this data collection can be found in some of our publications.


Up to the SOMLib Digital Library Homepage
Comments: rauber@ifs.tuwien.ac.at