Department of Software Technology
Vienna University of Technology

1. Obtaining the SOMeJB Prototype 1

2. Creating RAW files

2.1 Filenames Pre-Processing

#! /bin/bash
# changes Filenames for next Steps
# output: deletes all except: a-z0-9.
#
for file in *.mp3
do
  mv "$file" `echo "$file" | sed -e s/[^A-Za-z0-9\.]//g | tr "[:upper:]" "[:lower:]"`
done

In a second step Filenames are trimmed to a length of 15 characters.

#! /bin/bash
# shorts filenames to given length
# call this with *.mp3 to change mp3 Filenames
#
LENGTH=15

for i in "$@"; do
  extension=${i##*.}
  prefix=${i%%.*}
  new_name=`echo "$prefix" | cut -c -$LENGTH`.$extension
  echo [$new_name]
  mv $i $new_name
done

2.2 Create RAW files

After the preprocessing you have to run XMMS with a special plugin to get RAW file which are used for further processing. Start XMMS. Enable the "Dump 2 File" (anafile) Visualisation Plugin and close XMMS again. After that step you can now start XMMS with the music file you want to convert. While playing the mp3-file, the extracted frequency information is being written to disk. You will find resulting PCM and FRQ data for both channels in /tmp.

To convert a large Musiccollection run the following two scripts in the directory.

#! /bin/bash
# this file is called: "music2raw.help.sjb"
# script is called by do.raw to end xmms
# parameters: $1 (Seconds after which xmms is killed)
#
sleep $1
killall xmms

Script 2:

#! /bin/bash
# runs XMMS with all files in current directory
# output: filename.pcm/frq.links/rechts.bz2
# (355 is max length of music file in seconds)

mkdir ./raw

for FILE in *.mp3
do
  music2raw.help.sjb 355 &
  xmms $FILE
  cat /tmp/temp.frq.links | bzip2 >../raw/$FILE.frq.links.bz2
##  cat /tmp/temp.pcm.links | bzip2 >../raw/$FILE.pcm.links.bz2
#  cat /tmp/temp.frq.rechts | bzip2 >../raw/$FILE.frq.rechts.bz2
#  cat /tmp/temp.pcm.rechts | bzip2 >../raw/$FILE.pcm.rechts.bz2
done

rm /tmp/temp.frq.*
rm /tmp/temp.pcm.*

First edit script 2 and adjust the extension in the for-loop. Now change the parameter of "music2raw.help.sjb" and set it to the longest period of a music file. The Help script is needed, because XMMS has no command-line command to quit, so the process has to be killed after a specified time. Call Script 2 out from your music files directory within an xterm and have a cup of coffee :-) Find resulting RAW files in folder "./raw"

3. RAW2SPLIT

3.1 Cleaninf the RAW files

RAW Datafiles are in a unoptimized format, to clean them and adjust the file format for next steps SJBclean has to be used, which is part of the prototype 1 distribution. To download it individually: sjbclean-1.1.tar.gz. A handbook on SJBclean is available.

3.2 CLEAN 2 SPLIT

The cleaned RAW Datafiles can now be splittet into single frequency by applying SJBsplit, again part of the prototype 1 distribution. To download it individually: sjbsplit-1.0.tar.gz. A handbook on SJBsplit is available.

3.3 RAW 2 SPLIT

The following script shows a way to combine the steps mentioned above by using unix pipes.

#! /bin/bash
# clean Raw Data Files and split files into single frequency
# calls "sjbclean" and "sjbsplit"
# $1 is first frequency (eg: "1")
# $2 is take every $2 frequency (eg: "15")
# $3 is stop at that frequency (eg: "250")
#
# output: ./split/*.000 to *.255

mkdir ./split

for FILE in *.frq.*.bz2
do
  echo SJBclean FRQ /tmp/SJBclean, SJBsplit $FILE
  bunzip2 -k -c $FILE | sjbclean FRQ /temp/SJBclean | sjbsplit $FILE
  COUNTER=$1
  while [  $COUNTER -lt $3 ]; do
    cat $FILE.$COUNTER.temp | bzip2 >./split/$FILE.$COUNTER
#    rm $FILE.$COUNTER.temp
    let COUNTER=COUNTER+$2
  done
  COUNTER=0
  while [  $COUNTER -lt 256 ]; do
    rm $FILE.$COUNTER.temp
    let COUNTER=COUNTER+1
  done
done
# do not forget to remove not needed frequencys from ./split
rm /temp/SJBclean.1

4. SPLIT 2 TIME

5. TIME 2 FFT

5.1 Interpolation

5.2 FKT 2 FFT

5.3 TIME 2 FFT

The following script shows a way to combine the steps mentioned above by using unix pipes.

#! /bin/bash
# interpolates all files and does a fft with them
# calls "sjbfkt" and "sjbfft"
# $1 is full time in seconds (e.g.: 5)
# $2 are steps for FFT (e.g.: 256)
# $3 is weighted, non-weighted (e.g.: 1/0)
# CAUTION: steps must be a power of 2!
# FFT: gewichtet: 1, ungewichtet: 0
# output: *.fft

mkdir ../time

for FILE in *s
do
  echo sjbfkt $FILE $1 $2, sjbfft $3
  cat $FILE | sjbfkt $1 $2 | sjbfft $3 >$FILE.fft
  mv $FILE ../time
done
# leerdateien löschen!
echo now deleting empty files...
find -empty -exec rm '{}' ';'

6. Vector generation

6.1 Majorvector

After all the steps mentioned in the previous chapters there should now be a directory with a lot of *.fft files in it. To obtain a valid SOM Vectorfile you have to run now SJBmajorvector. To download it individually: sjbmajorvector-1.0.tar.gz. A handbook on SJBmajorvector is available. There is also a undocumented program for special vector generation called SJBvector. To download it individually:SJBvector

6.2 Positive values

The GHSOM does not allow negative values within the Vectorfile. Use SJBplus to get a clean input vectorfile containing only positive values. To download it individually: sjbplus-1.0.tar.gz. A handbook on SJBplus is available.

7. Train the GHSOM

Now run the GHSOM with the following property file: (also available for download: sjb_demo.prop together with a template vector file required for the GHSOM:sjb_v1.tv.gz, do not forget to unzip the template vector file! )

EXPAND_CYCLES=3
MAX_CYCLES=0
TAU_1=1
TAU_2=1
INITIAL_LEARNRATE=0.8
INITIAL_NEIGHBOURHOOD=20
HTML_PREFIX=seg_map
DATAFILE_EXTENSION=
randomSeed=17
inputFile=vectors/segment.in
descriptionFile=vectors/sjb_v1.tv
savePath=output
printMQE=false
normInputVectors=NONE
saveAsHTML=true
saveAsSOMLib=true
INITIAL_X_SIZE=22
INITIAL_Y_SIZE=22
LABELS_NUM=0
LABELS_ONLY=true
LABELS_THRESHOLD=0.35
ORIENTATION=true

8. Vector generation for 2-nd level clustering

To get a valid vectorfile out of the mapping SJBmatrix or SJBstrongmatrix are used. Both can be downloaded individually as: sjbmatrix-1.0.tar.gz and sjbstrongmatrix-0.9.tar.gz. Handbooks for both SJBmatix and SJBstrongmatix are available.

9.GHSOM for 2-nd level clustering