weekly update

This week we both read through the tutorials of PureData as indicated last week. Also the book of Puckette M. is not virgin anymore.
For the next week we'll implement some concrete effects. Concrete means that they really work with a given input, from input to output. If there's time we might also try an extraction algorithm already. But we have to keep it simple so as to achieve something concretely this week.
Things we talked about during our meeting:

• I tried already measuring the latency by just connecting the input of my external soundcard (EDIROL FA101) with the output through PureData. With a sample rate of 44,1kHz and quantizing at 24bit I reach a minimal latency of 37msec, these is too audible and 1msec less results in a big mess. This is not the case when using other profeesional audio software on my computer (adobe audition 2.0, Live 6.0.1) I didn't really search further, I ony tried it out once and it works at least. I know there exists a latency test patch for Pd on the internet. I'll search it the next time I do real input tests.
• Anyway, Sam says that an audible latency is OK as long as we have a realtime working effect. The latency of the program would be rather normal because of the complexity of the code.
  

matlab session on the signal routing for testing my delay and other effects

I worked on the file effectSystem.m and all it needs still is the output part: playing and plotting the results of the effect. And also the part where the different frames are glued together again to form a soundfile and the mixer part.
After that I can start implementing and testing effect files I already have, or new ones.
I did not yet pay attention in effectSystem.m to the feature extraction part.

Reading at home and matlabsession in ESAT on HPS and a interface between effects

I finally found an article on that specific thing we are working on. (uch, some articles):

• Implementation strategies for adaptive digital audio effects
• A-DAFX adaptive digital effects
• Adaptive digital audio effects (A-DAFX) : a new class of sound transformations
• Adaptive audio effects based on STFT, using a source filter model
• Driving pitch-shifting and time-scaling algorithms with adaptive and gestural techniques
• An interdisciplinary approach to audio effect classification

Very interesting of course.
In the first article they speak of mapping functions and a control curve. Also the control scaling part is nice.

I'm going to read the other ones now too, because I tried some other things in matlab, but they wouldn't work well: a little bit more explanation:

1. I implemented the basics of the HPS algorithm as promised yesterday. Basically it's allright, but I don't know if it works because firstly I didn't yet implement the deciding code (which decides which peak is the current pitch), secondly because I didn't think about frames or blocks (Maybe this is not necessary, but I didn't yet think about this), thirdly because there's a seemingly stupid thing happening in matlab: it doesn't recognize the file I want to play. (Solved: I have to give the attributes as a string, so encapsulated with ' ' )
2. I also tried to make an applyEffect.m-like file that selects the extraction algorithm and effect, and then reads the file, applies the effect with the parameters from the extraction algorithm and plays the file. This is complexer than I thought, so I need to read or think about the control curve some more.
   

weekly meeting

A very long meeting, many things are said. I'll try a conclusion:
In this stage we see too many possibilities, too many ways we can walk. That's normal.

• A good idea is to keep on swimming in this chaos for a few days and then try to decide on some things and ordering everything in our minds.
• We discussed a while on realtime issues and complexity issues:
  Following Toons' opinion there are two extremes for us. I made a little figure of them:
  
  The first one shows the most difficult one: extracting information from the signal upon which we want to apply the effect.
  
  The second one is complexer, but easier. Here we have the whole spectrum of chosen tracks we want to use to extract different features from the music. Best is to use them all at once as an input, then the algorithm can choose the tracks it needs for the solution. This feature vector is then used as an input for effects that are applied on different tracks of choice (not necessarily the same tracks that are used as input for the feature extraction.)
  
  -28feb2007-
  There is another well defined extreme I thought of: It's like the second idea, but using only the master tracks as an input for the extraction algorithms. This means you have your 'intelligent effect box' and you plug into it a stereo mic or the outputs of the stereo masters of the mixer. The algorithms have to segment the music then or at least they have to do their thing without knowing specific knowledge about the input. They must be able to deal with noise from the public. This is a problem we won't handle.
  
• I talked also a lot about the interface of the effect blocks and extraction blocks. Probably the best and only choice is to use a modular structure. But in that case I want to have as soon as possible a standard interface.
• We will never reach the stage where we implement the effect in C or something like that, the farthest we'll get might be simulink (modular and realtime). It might be the right time to switch to simulink sometimes already.
• Obviously I forgot a lot of details here... someone?
  

matlab session on pitch detection, but more about ring modulator etc.

Also today I do a little warming up: I'd like to give a it a shot and implement a ring modulator (DAFX p76). The ring modulator of my yamaha MO8 has the following parameters:

• oscillator frequency coarse (0.5 - 5 kHz)
  
• oscillator frequency fine (0 - 127)
  
• LFO wave (tri, sine)
  
• LFO depth (0 - 127)
  
• LFO speed (0.0 - 39.70 Hz)
  
• HPF cutoff frequency
• LPF cutoff frequency
• dry/wet balance
• EQ low frequency
• EQ low gain
• EQ high frequency
• EQ high gain

Only the first five comprise the ring modulation, 6 and 7 are filter parameters, 8 is a mixer parameter and the rest are EQ parameters.
Some remarks:

1. I used
   

   modSound = sin(2*pi*OSCfreq*[0:1/Fs:Nbits]);
   

   but the Nbits indicates the quantization depth (16 bits when used with the wave file flute2.wav in my folder). Instead I have to use the length of the sound vector (as seen in the algorithm) when I want the modulating signal to be as long as the soundFile input.
   What I still don't get is why I have to divide the length of the vector sound with Fs. It is a kind of normalization, and if I don't do it, it wouldn't work. But I don't know why.
   solution: Try it with a ridiculous 2Hz sampling frequency and 10 seconds music sample. You'll see that if you want as many samples on the sine wave as on the input sample you'll have to divide the length of your input minus 1 through 2 to get to the 10 seconds.
   
2. I spent a reasonable amount of time in generating different signals to be used as modulators; sine wave, triangular wave (special case of:), sawtooth, square wave. Everything works allright and the explanations are to be found in the m-file.
3. The algorithm:
   

   function [output] = ringmod(soundFile, OSCfreq, LFOwave, LFOdepth, LFOspeed)
   % ring modulator
   % y(n) = x(n).m(n)
   % OSCfreq: oscillator frequency (best between )
   % LFOwave: type of wave that modulates the audio signal (sine, triangle, square, sawtooth with standard width)
   
   [sound,Fs,Nbits] = wavread(soundFile);
   
   % defining the modulating wave type 'LFOwave'
   % SINE
   if (strcmp(LFOwave, 'sine')) % use strcmp instead of == to compare strings
   % sampled taking steps of Ts or 1/Fs and this until length(sound)/Fs samples are
   % calculated
   % number of vector values 'length(sound)' must be normalized by Fs
   % because ... (?)
   modSound = sin(2*pi*OSCfreq*[0:1/Fs:(length(sound)-1)/Fs]);
   end
   % SAWTOOTH
   if (strcmp(LFOwave, 'sawtooth'))
   width = 0.9 % this defines where the max of the wave is situated in the interval between 0 and 2*pi
   modSound = sawtooth(2*pi*OSCfreq*[0:1/Fs:(length(sound)-1)/Fs],width);
   end
   % TRIANGLE
   if (strcmp(LFOwave, 'triangle'))
   %triangle wave (width = 0.5)
   modSound = sawtooth(2*pi*OSCfreq*[0:1/Fs:(length(sound)-1)/Fs],0.5);
   end
   % SQUARE
   if (strcmp(LFOwave, 'square'))
   modSound = square(2*pi*OSCfreq*[0:1/Fs:(length(sound)-1)/Fs]);
   end
   
   %wavplay(modSound, Fs); %for testing
   %sound(1:100)
   %modSound(1:100)
   %length(sound)
   %length(modSound)
   output = sound .* modSound';
   
   wavplay(output, Fs);
   

4. This effect is perceived as nice eg. when using a voice as input, sine or other wave as modulation and low frequencies (~50Hz)
5. I had some problems with the multiplication, but after trying it with small matrices I saw why:
   
   • I had to use a dot-product for elementwise multiplication .*
   • I had to have two columnmatrices or vectors. The modSound waves are on the contrary row matrices, so I had to transpose them: modSound'
     

OK, a warmup during 3 hours, typical...
So now a little bite and then: pitch extraction!

Hm, another two hours spent on surfing the internet and finding sites we could use. Look at the posts involved: Links, just links , some links for beat extraction.
Also I started a post that will contain ideas we have during our work. This way we have a leading trail when experimenting later on.
OK, another try towards pitch detection.

I'll talk about pages 336 and further in DAFX.
Pitch extraction is the same as estimating the fundamental frequency f0 and then possible postprocessing like pitch tracking and taking into account frequency relationships.
I'll be reading the interesting things and tomorrow I'll try the matlab code.

Weekly meeting

We agreed on many ideas: after experimenting with effects in matlab, we should now start doing matlab on extraction of features in audio. That is what we're going to do first. Then the next step would be to connect extraction features with effects. All this might be done in matlab first.
Of course we really want this to be done in real time (not only to do easy testing, but also for client purposes) and modular.
If we could make standard effect blocks and standard extraction blocks in simulink that would be nice. Simulink might indeed be the best next step after matlab, eg to test in realtime (although this might require some reprogramming of the m-functions).
We also talked about the idea to turn everything in VST plugins or direct X plugins as a final step, but we might lack time for that.
TODO: Our promotor M. Moonen would like us to do an intermediate presentation. Here we might explain some theoretical issues about effects and extraction algorithms we already tried and also we can explain our project status and the next steps towards our goal.

warming up session 2007

Exams finished, vacation finished. As for me, I'm almost healthy again.
Today Tony and I met to set things straigth and restart our discovery.
Our plan for the nearby future:
This week we will finish the basic literaturestudy (as in, we will read the DAFX book and surf across the internet a lot)
So next week we can start trying out some feature extraction methods (as we already did with effects)

On monday 19 feb we hope to meet with our assistants.

matlab session on my own

To login to ESAT from a remote computer I use Filezilla:
host: login.esat.kuleuven.be
port: 22
servertype: SFTP using SSH2
user: rvandenb
Here I hold the folder 'Thesis', where all the documents are placed.

To give labels to axes:
xlabel('Time');
ylabel('Frequency (Hz)');
I tried understanding the command spectrogram, but...
Even the chirp example given by matlab I don't understand. I now try searching google on it.

Found an at first sight, very exciting URL: http://www.nd.edu/~nkottens/
Some other m-files about audio on the mathworks file-exchange also with some effects!

There is a spectrogram function on this site with better visual effect.
I'll study this one in another post