TU Wien IFS is a partner of the MusicBricks project. Our aim is to provide innovative music technologies from European institutions to developers, hackers, digital makers and creative SMEs and companies.

"#MusicBricks are a compendium of both physical and virtual interfaces and APIs, that allow creators, developers and digital content makers easy access to core building blocks of music."

This is a collection of #MusicBricks tools provided by TU Wien:

• RP_extract: Rhythm & Timbre Feature Extraction from Music
• SMINT API & Search by Sound Music Similarity Retrieval system

RP_extract is a library that processes audio data (WAV, PCM, or MP3) as input and analyzes the spectral rhythmic and timbral information in the audio to create different audio descriptors (a.k.a. features).

These audio descriptors can be used to find similar sounding songs, create automatic playlists, make music recommendations etc. Depending on the needs, a  range of audio features is available:

Rhythmic descriptors:

• RP: captures Rhythm Patterns (see picture at the right - audible frequency vs. repetition frequency) and is able to find songs with similar rhythm
• RH: Rhythm Histogram: simplified rhythm descriptor (roughly containing rhythmic strength for different bpm values)

Timbral descriptors:

• SSD: Statistical Spectrum Descriptor: describing timbral aspects of the audio Sonogram; by that it is possible to find songs with similar timbral characteristics
• MVD: The Modulation Frequency Variance Descriptor measures variations over critical audible frequency bands for a specific rhythmic repetitions (derived from a rhythm pattern).

Temporal descriptors:

• TSSD: Timbral variations over time (based on SSD)
• TRH: Temporal Rhythm Histograms - rhythmic variations over time (based on RH)

Features can also be combined (e.g. to cover rhythmic and timbral aspects).

This library is available free and open source:

• Python: https://github.com/tuwien-musicir/rp_extract
• Matlab or Java: from our download page

Documentation:

• Quick description of audio features
  
• Scientific Background Paper on audio features
  
• Tutorial on Audio Feature Extraction (Python) (recommended)
• Tutorial on Audio Feature Extraction (Python) (slightly outdated)

Feel free to use it in your own projects (but let us know about it :-).

The Search by Sound system developed by Spectralmind (a TU Wien spin-out) can analyze music tracks and store their fingerprints (features) in a database. Via an API, one can query for similar sounding songs, using a combination of audio-feature based and meta-data query (e.g. "give me rhythmically similar songs from the genre 'Electronic'").

The system can be set up with a custom (or empty) music library where additional songs can be added via the SMINT API. We also provide a pre-analyzed library with 50,000 songs from freemusicarchive.org.

The backend analyzing all the songs run on our server. We provide a demo frontend usable in the Web browser and an API to add, delete and retrieve (similar) songs. 

The API methods are:         

• track/add ... Add a Track by sending a URL where the track a) can be downloaded or b) providing a local file location (on the same server as the API is running).         
• track/delete/:smint_track_id … Removes a track from the system.
• track/:smint_track_id … Returns a list of tracks that are similar to the given trackid.
• track_external_key/:external_key … Returns a list of tracks that are similar to the given external key. 
• version … Returns version information on the API. 

The API returns an HTTP status code and an XML document.

Documentation:

• API Documentation
• API Usage Tutorial (Python)
• Frontend Usage Tutorial
• Search by Sound Web Frontend (Demo containing 50k preanalyzed tracks from freemusicarchive.org)

Please contact us for the API URL and the Web frontend. ( lidy AT ifs.tuwien.ac.at )