Creation process for the Map of Mozart

The complete works of Mozart have been transformed into a digital audio format. Then, an analytical method, which extracts specific characteristics from the audio data, was applied - the so-called Audio Feature Extraction. For the analysing process, we use solely acoustic characteristics - the method has no knowledge of notes, motives and other structures of music theory concerning the pieces. The features thus extracted correspond to a so-called rhythm pattern [1,2,3]. This rhythm pattern is also displayed graphically next to the Map of Mozart. The rhythm pattern contains not only, strictly speaking, rhythms, but also even the smallest fluctuations in high frequency bands.

In the subsequent step, the features extracted have been passed on to a self-organizing system, the so called Self-Organizing Map (SOM). Here, the computer learns from the features exhibited in the music pieces and tries to organize them step by step on a two-dimensional map until as coherent as possible a representation emerges. In this map, pieces of music with similar features are mapped close to each other, and such pieces with low similarity are in distant regions. The map itself can take any arbitrary structure, from rectangular to specific shapes, as, for that matter, the silhouette of Mozart (Mnemonic SOM) [4].

The Map of Mozart thus created allows a representation which quickly gives an overview of Mozart's works. Each square on the Map of Mozart contains a group of pieces of music which exhibit a strong musical similarity among each other. Neighboring squares are occupied by similar pieces of music. The further away two squares are from each other, the more divergent are their musical characteristics. 

A group of similar squares forms whole cluster (or islands) of similar pieces of music. As an example, almost all operas composed by Mozart are located in the lower part of the  Map of Mozart, while piano music can be found on the top edge. The operas are further divided into different regions, for example recitatives, located in the region of Mozart's neck. The cluster of the piano music contains piano sonatas and concerts (however, the map does not primarily try to recognize instruments, but analyzes only acoustic characteristics).

Transitions between musical characteristics are continuous, and so is the content of the squares of the map. At boundaries between vastly different regions also empty squares may appear. The mapping of pieces of music to the squares results from a computer algorithm based solely on the audio content of the music. Therefore it may also happen that a few pieces may not be mapped to exactly the position one would expect - humans also often do not reach an agreement for classifying music into categories!

Publications:

1. Thomas Lidy, Andreas Rauber.
   Evaluation of Feature Extractors and Psycho-acoustic Transformations for Music Genre Classification.
   Proceedings of the Sixth International Conference on Music Information Retrieval (ISMIR 2005), pp. 34-41, London, UK, September 11-15, 2005. [pdf]
2. A. Rauber, E. Pampalk, D. Merkl.
   Using Psycho-Acoustic Models and Self-Organizing Maps to Create a Hierarchical Structuring of Music by Musical Styles
   Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR 2002), pp.71-80, October 13-17, 2002, Paris, France. [pdf]
3. A. Rauber, and M. Frühwirth.
   Automatically Analyzing and Organizing Music Archives
   Proceedings of the 5. European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2001), Sept. 4-8 2001, Darmstadt, Germany. [pdf]
4. Rudolf Mayer, Dieter Merkl, Andreas Rauber.
   Mnemonic SOMs: Recognizable Shapes for Self-Organizing Maps
   Proceedings of the 5th Workshop On Self-Organizing Maps Paris (WSOM 2005), pp. 131-138, September 5-8 2005, Paris, France. [pdf]

Further publications are listed on the publication page of the mir group