Drawing audio data from within control

[chaz13]

Hi!

I'm trying to create a control, based off of the mouse control example. I want it to be able to draw the waveform of the audio (the past so many samples or whatever). So you'll attach/create the control like any other, and you can interact with it to change a parameter, but it'll do some drawing stuff too. There's a couple of things I'm not certain on the best way to do:

1) How do you draw things like lines from within a control class? Where should the drawing code go, and how do you make it update every frame?

2) How do you get the sample data from within the control class? I presume I'll need to save a buffer from within the audio processing function, but I'm not sure how I'd then access that from a control.

Thanks for any help!

[random_id]

You can start by making a control that inherits IControl. Take a look at the IPlugMultiTargets project in WDL-OL. There is a VU meter in there.

You need to create your own functions for Draw() and SetValueFromPlug(). The SetValueFromPlug() is how you send data to the control from the audio processing thread. This gets the data into the control. In the Draw() function, you then have to figure out how to visualize the data.

I have a waveform history view in my plugins. It basically draws a series of vertical lines from left to right in the Draw() function. The height is based on the audio data. You will also need some type of array/vector to hold previous data.

There is a lot of little things that need to be done, especially when you start thinking about "what happens if I want to change the scale (vertically/horizontally)". If you start from the perspective that the control is just a VU meter that keeps some history, it should get you going.

I hope that helps.

[Add9]

Originally Posted by random_id

You can start by making a control that inherits IControl. Take a look at the IPlugMultiTargets project in WDL-OL. There is a VU meter in there.

Also check out the IPlugOpenGL project... OpenGL is really powerful and you can create simple visuals but also all kinds of crazy sophisticated stuff too if you understand how to manipulate the 3D space (which I don't, but I've seen some cool examples :P)

[chaz13]

Originally Posted by random_id

You can start by making a control that inherits IControl. Take a look at the IPlugMultiTargets project in WDL-OL. There is a VU meter in there.

You need to create your own functions for Draw() and SetValueFromPlug(). The SetValueFromPlug() is how you send data to the control from the audio processing thread. This gets the data into the control. In the Draw() function, you then have to figure out how to visualize the data.

I have a waveform history view in my plugins. It basically draws a series of vertical lines from left to right in the Draw() function. The height is based on the audio data. You will also need some type of array/vector to hold previous data.

There is a lot of little things that need to be done, especially when you start thinking about "what happens if I want to change the scale (vertically/horizontally)". If you start from the perspective that the control is just a VU meter that keeps some history, it should get you going.

I hope that helps.

Thanks for the suggestions, that's great info and I've nearly got it working. I'm not 100% sure how to use the SetValueFromPlug. I have a function by that name in the waveform control class, like so:

	void SetValueFromPlug(double sample) {
		for (int i = 0; i <399; i++) {
			lastsamples[i] = lastsamples[i + 1];
		}
		lastsamples[399] = sample;
		OutputDebugString(std::to_string(sample).c_str());
	}

But I'm not too sure how to call it from within the audio processing function.

[random_id]

You would have a pointer to your control, and then use something like this after your audio processing occurs:

double outMax = std::max(std::abs(*out1), std::abs(*out2));
pMyControl->SetValueFromPlug(outMax);

Of course, you could always do something other than the max/abs stuff if you were graphing + and -.

One thing to think about is that instead of shuffling your values in the array, you could always use a circular buffer with an index. Something like:

void SetValueFromPlug(double sample) {
    idx++;
    if (idx > 399) ixd = 0;
    lastsample[idx] = sample;
}

Then when you Draw, you just use idx+1 as the oldest value, and cycle through it.

[chaz13]

Originally Posted by random_id

You would have a pointer to your control, and then use something like this after your audio processing occurs:

double outMax = std::max(std::abs(*out1), std::abs(*out2));
pMyControl->SetValueFromPlug(outMax);

Of course, you could always do something other than the max/abs stuff if you were graphing + and -.

One thing to think about is that instead of shuffling your values in the array, you could always use a circular buffer with an index. Something like:

void SetValueFromPlug(double sample) {
    idx++;
    if (idx > 399) ixd = 0;
    lastsample[idx] = sample;
}

Then when you Draw, you just use idx+1 as the oldest value, and cycle through it.

Thanks! That was super helpful, I have a beautiful waveform drawing going on now

I have a bit of an extension to my question. I'll outline what I want to achieve in general, because there's probably both good and bad ways to go about doing it..

I'd like this waveform drawer to take mouse input which can change a parameter. That parameter is also linked to a knob, so I want the knob to be able to change the parameter (and changes reflect in the waveform control), and vice versa. So basically, I want to share a parameter between two controls - will this work?

I'm not sure how to get/set the value of the parameter from my waveform control I'm writing. Right now I'm just trying to get the value, so I can draw the threshold line on the waveform.

I've tried using mPlug->GetParam(kThresh)->Value(), but this gives an error that kThresh1 is an undeclared identifier - although it's initialised before this control is made so I don't fully understand why that is. If I replace it with a 0 it works, though, which is something. I presume I need to pass kThresh1 into the control during it's creation, but I'm not 100% on how to actually do that.

EDIT: Woo, after some tinkering I have it nearly working. I pass the parameter id's into the waveform control. I can now read the parameter values and draw the lines I wanted perfectly. I'm still not sure how to set the parameters from the waveform control, though. Using:

mPlug->GetGUI()->SetParameterFromPlug(thresh1_idx, 0.5, 0);

Does set the parameter, but it does not invoke the on parameter change function in the main .cpp file.

[random_id]

In my waveform viewer control, I have an adjustment for vertical and horizontal spacing. This impacts the decibel scale, as well as how much "history" is shown. What I did (and there are probably many ways to do it), is to have the constructor of the control take the parameter index of the other controls. Then, I periodically check the value of the vertical and horizontal controls to see if they changed. If so, the control recalculates the vertical and horizontal scale.

The other thing you could do is to do everything from within the control. Using OnMouseDown() and all of the other IControl functions, you can determine what happens when the user clicks on the control. You can get as complicated as you want by determining where the user clicked within the control.

[chaz13]

That's really helpful, thanks!

One question I have on SetValueFromPlug - this is expecting a double, but I'm hoping to pass in two values. I've tried changing it to pass an array but that wasn't working out - is there a better way to do this?

[Xenakios]

Originally Posted by chaz13

That's really helpful, thanks!

One question I have on SetValueFromPlug - this is expecting a double, but I'm hoping to pass in two values. I've tried changing it to pass an array but that wasn't working out - is there a better way to do this?

2 consecutive calls into SetValueFromPlug? How else would you hope it to work?

[chaz13]

I should clarify - two unrelated values (like min and max of the signal, for examples), not two of the same thing. So I'd ideally want to pass two doubles, or an array of doubles. Perhaps SetValueFromPlug is the wrong function to be using here!

[random_id]

I think you should just make your own method that accepts multiple values. Since you have a pointer to the control, you should be able to call it without problems from ProcessDoubleReplacing().

[chaz13]

That just occurred to me too. I got stuck before because I didn't have that pointer so couldn't get my own method to work - but that should do the trick! Thanks for all your help

EDIT: Just having a little trouble calling my own method. I have a pointer to the control, but apparently that's not what I need. This is what I'm doing currently:

	  IControl* control = GetGUI()->GetControl(mInputMeterIdx);
	  control->UpdateValues(inmin, inmax);

Where update values is a method I've defined in the meter class. I get the "IControl has no member "UpdateValues"" error, which makes sense, but I'm not sure how to get the instance of the "IWaveFormThresh" which inherits from IControl, instead of the IControl.

[Tale]

You can do it like this:

IWaveFormThresh* control = (IWaveFormThresh*)GetGUI()->GetControl(mInputMeterIdx);
control->UpdateValues(inmin, inmax);

Of course you will have to be sure that mInputMeterIdx really points to a IWaveFormThresh.

[random_id]

I misspoke. I shouldn't have said "a pointer to your control". I should have said "a pointer to your derived control class". A pointer to IControl will only give you access to the base class functions. If you have a pointer to the derived class, you can use/create whatever functions you want.

You can always cast it like Tale said, too.

[chaz13]

That's working great now, thanks

I have a bit of a new problem, though. It's more to do with the actual drawing, but I figured I may as well continue asking here instead of making a new thread for a related issue.

The drawing of my waveform is really.. flickery? It's a little difficult to tell whether it's an optical illusion/the result of aliasing, but it feels like sometimes the drawing isn't performed on some refreshes so the waveform flickers.

In addition to this, my waveform drawing sometimes extends beyond the bounds it should when another control is tweaked, as you can see here:



It only happens for a split second (took a long while to get a screenshot!) when one of the two knobs to the right of the waveform are being changed. It's also strange because the boundaries of the waveform control definitely end at the end of the box on the GUI, and the drawing is 4x the length of the waveform array, which is 160 samples, giving a length the same as that of the control/box in the GUI, so it shouldn't ever be able to extend beyond that range (in theory..).

Here's how the drawing is done. I know I could optimise and only have one of the rectangle's being drawn (as one will have the bottom higher than the top and not been drawn), but I'm not really worried about that right now - the issue persists if I comment one of those out.

	bool Draw(IGraphics* pGraphics)
	{

		double thresh1 = mPlug->GetParam(thresh1_idx)->Value();
		double thresh2 = mPlug->GetParam(thresh2_idx)->Value();

		//Draw waveform
		int count = 0;
		for (int i = idx + 1; i < idx + 160; i++) {

			double stretch = 4;

			IColor linecol;
			linecol = COLOR_WHITE;
			
			double minval = lastsamplesmin[(i) % 159];
			double maxval = lastsamplesmax[(i) % 159];

			IRECT filledBit;

				filledBit = IRECT(mRECT.L + count * stretch, (mRECT.B + mRECT.T) / 2, mRECT.L + count * stretch + stretch-1, (mRECT.B + mRECT.T) / 2 + (minval * mRECT.H() / 2));
				pGraphics->FillIRect(&linecol, &filledBit);

				filledBit = IRECT(mRECT.L + count * stretch, (mRECT.B + mRECT.T) / 2 + (maxval * mRECT.H() / 2), mRECT.L + count * stretch + stretch-1, (mRECT.B + mRECT.T) / 2);
				pGraphics->FillIRect(&linecol, &filledBit);


			count++;
		}

//Threshold lines
		pGraphics->DrawLine(&COLOR_RED, mRECT.L, (mRECT.B + mRECT.T) / 2 + thresh1*mRECT.H()/2, mRECT.R, (mRECT.B + mRECT.T) / 2 +thresh1*mRECT.H()/2);
		pGraphics->DrawLine(&COLOR_RED, mRECT.L, (mRECT.B + mRECT.T) / 2 - thresh1*mRECT.H() / 2, mRECT.R, (mRECT.B + mRECT.T) / 2 - thresh1*mRECT.H() / 2);

		pGraphics->DrawLine(&COLOR_GREEN, mRECT.L, (mRECT.B + mRECT.T) / 2 + thresh2*mRECT.H() / 2, mRECT.R, (mRECT.B + mRECT.T) / 2 + thresh2*mRECT.H() / 2);
		pGraphics->DrawLine(&COLOR_GREEN, mRECT.L, (mRECT.B + mRECT.T) / 2 - thresh2*mRECT.H() / 2, mRECT.R, (mRECT.B + mRECT.T) / 2 - thresh2*mRECT.H() / 2);

		//Changing parameter
		//mPlug->GetParam(thresh1_idx)->Set(0.5);
		//mPlug->GetGUI()->SetParameterFromPlug(thresh1_idx, 0.5, false);

		return true;
	}

Is it bad to draw a bunch of rectangles this way in a loop, or something like that?

[random_id]

I am not sure what is happening, but I suspect it has something to do with idx * stretch. What is the max value of idx? You could always use the BOUND macro within IPlug, but I think figuring out what is calculating incorrectly would be better in the long run.

I am taking a guess with the flickering, but I wonder if it is happening when your buffer is reset, looped, whatever you call it. I would debug and step through the code specifically around the point when the buffer goes 158, 159, 0, to see if it is doing what you want it to. Not only with the draw method, but the method that is adding the audio data to the buffer.

It could also be a problem where the control thinks it isn't dirty, so it is not drawn for a frame or two. You should be able to set mDirty to true and see if that is part of the problem. You could do that temporarily within the call to add the audio data to see if it changes anything.

And, I don't think your way of drawing rectangles is bad.

[chaz13]

Originally Posted by random_id

I am not sure what is happening, but I suspect it has something to do with idx * stretch. What is the max value of idx? You could always use the BOUND macro within IPlug, but I think figuring out what is calculating incorrectly would be better in the long run.

I am taking a guess with the flickering, but I wonder if it is happening when your buffer is reset, looped, whatever you call it. I would debug and step through the code specifically around the point when the buffer goes 158, 159, 0, to see if it is doing what you want it to. Not only with the draw method, but the method that is adding the audio data to the buffer.

It could also be a problem where the control thinks it isn't dirty, so it is not drawn for a frame or two. You should be able to set mDirty to true and see if that is part of the problem. You could do that temporarily within the call to add the audio data to see if it changes anything.

And, I don't think your way of drawing rectangles is bad.

Thanks for the suggestions! Gave me a lot to think about, and there were a few bugs with the modulo stuff, although that wasn't causing the issue.

The MAIN issue causing the drawing out of bounds was that the variable idx (i.e position in the circular buffer) was changing DURING the drawing loop, so I guess the audio thread was updating the samples during the drawing, which is obviously not ideal... I've fixed that by using a static value for the loop, start_index, which fixes the flashing and the drawing where it shouldn't be, like so:

		//Draw waveform
		//int count = 0;
		int start_index = idx;
		for (int i = start_index + 1; i < start_index + 160; i++) {

			int modind = i % 160;
			int count = i - (start_index + 1);
			double stretch = 3;

			IColor linecol;
			linecol = COLOR_WHITE;

			double minval = lastsamplesmin[(i) % 160];
			double maxval = lastsamplesmax[(i) % 160];

			IRECT filledBit;

			filledBit = IRECT(mRECT.L + count * stretch, (mRECT.B + mRECT.T) / 2, mRECT.L + count * stretch + stretch-1, (mRECT.B + mRECT.T) / 2 + (minval * mRECT.H() / 2));
			pGraphics->FillIRect(&linecol, &filledBit);
			filledBit = IRECT(mRECT.L + count * stretch, (mRECT.B + mRECT.T) / 2 + (maxval * mRECT.H() / 2), mRECT.L + count * stretch + stretch-1, (mRECT.B + mRECT.T) / 2);
				pGraphics->FillIRect(&linecol, &filledBit);
			//count++;
		}

Problem now is I still get "jumps" in the waveform every second or so. You'd expect a sine wave to kinda "crawl" across the screen in one direction, which is mostly how it works, but it'll occasionally make a big jump in one direction. I'm assuming this is due to the updates happening faster, or during, the drawing, but I'm not too sure what I can do about that. Here's how the updates work:

Method in the waveform control class:

	void UpdateValues(double samplemin, double samplemax) {

		lastsamplesmin[idx] = samplemin;
		lastsamplesmax[idx] = samplemax;

		idx++;
		if (idx > 159) 
			idx = 0;
	}

Which is called in the main audio method like:

		if (GetGUI())
		{

			IWaveFormThresh* c = (IWaveFormThresh*)GetGUI()->GetControl(mInputMeterIdx);
			c->UpdateValues(inmin, inmax);

			//IControl* control = GetGUI()->GetControl(mOutputMeterIdx);
			//control->SetValueFromPlug(outmean);
		}

For each sample in the buffer. inmax and inmin are used because I will eventually take a window of the buffer, but I stopped doing that for now for simplicity, so inmax and inmin are just the current sample value right now.

EDIT Caught the glitch in action:

[random_id]

I think the problem might be with the way you are looping through the buffer.

What about something like this (untested)

int start_index = idx;
int xPos = mRECT.L;
int stretch = 3;

IColor linecol;
linecol = COLOR_WHITE;

for (int i = 0; i < 160; i++){
	
	start_index++;
	if(start_index > 159) start_index = 0;
	
	double minval = lastsamplesmin[start_index];
	double maxval = lastsamplesmax[start_index];
	
	IRECT filledBit;
	filledBit = IRECT(xPos * stretch, (mRECT.B + mRECT.T) / 2, xPos * stretch + stretch-1, (mRECT.B + mRECT.T) / 2 + (minval * mRECT.H() / 2));
	pGraphics->FillIRect(&linecol, &filledBit);

	filledBit = IRECT(xPos * stretch, (mRECT.B + mRECT.T) / 2 + (maxval * mRECT.H() / 2), xPos * stretch + stretch-1, (mRECT.B + mRECT.T) / 2);
	pGraphics->FillIRect(&linecol, &filledBit);
	
	xPos++;
	
}

Then you don't have to deal with the modulus stuff

The other thing you could do is test with a sin wave, and add an assert line to trigger when the previous and next values are widely different. Then you could look in the lastsamplemin/lastsamplemax arrays to see if they have the correct values, or it is the Draw().

[chaz13]

Originally Posted by random_id

I think the problem might be with the way you are looping through the buffer.

What about something like this (untested)

int start_index = idx;
int xPos = mRECT.L;
int stretch = 3;

IColor linecol;
linecol = COLOR_WHITE;

for (int i = 0; i < 160; i++){
	
	start_index++;
	if(start_index > 159) start_index = 0;
	
	double minval = lastsamplesmin[start_index];
	double maxval = lastsamplesmax[start_index];
	
	IRECT filledBit;
	filledBit = IRECT(xPos * stretch, (mRECT.B + mRECT.T) / 2, xPos * stretch + stretch-1, (mRECT.B + mRECT.T) / 2 + (minval * mRECT.H() / 2));
	pGraphics->FillIRect(&linecol, &filledBit);

	filledBit = IRECT(xPos * stretch, (mRECT.B + mRECT.T) / 2 + (maxval * mRECT.H() / 2), xPos * stretch + stretch-1, (mRECT.B + mRECT.T) / 2);
	pGraphics->FillIRect(&linecol, &filledBit);
	
	xPos++;
	
}

Then you don't have to deal with the modulus stuff

The other thing you could do is test with a sin wave, and add an assert line to trigger when the previous and next values are widely different. Then you could look in the lastsamplemin/lastsamplemax arrays to see if they have the correct values, or it is the Draw().

Thanks for the help (as always). I've re-written the code as you suggested, and no change in the problem. Triggering a stop in the code when the value changes a LOT does seem to catch the issue when it happens, but there's no obvious time/index/position at which it happens.

I tried re-writing the code to use a normal buffer instead of a circular one to entirely rule out that being the issue, like so:

void UpdateValues(double sample) {

	for (int ii = 0; ii < 159; ii++)
		lastsamples[ii] = lastsamples[ii + 1];

	lastsamples[159] = sample;

}

	//Draw waveform
	int xPos = mRECT.L;
	int stretch = 3;

	IColor linecol;
	linecol = COLOR_WHITE;

	for (int i = 0; i < 160; i++) {

		double val = lastsamples[i];

		IRECT filledBit;
		filledBit = IRECT(xPos + i* stretch, (mRECT.B + mRECT.T) / 2, xPos + i* stretch + stretch - 1, (mRECT.B + mRECT.T) / 2 + (val * mRECT.H() / 2));
		pGraphics->FillIRect(&linecol, &filledBit);

		xPos++;

	}

But that worked exactly the same as the original version - with the same jumping issue. So I guess that means something else is going on and I'm suspecting it's to do with how/when the graphics and audio threads are communicating, although I'm currently at a loss to an exact reason and how I'd go about fixing that! I'm going to re-write the class and everything involving it from scratch to see if that magically helps, but I'm not too hopeful aha.

P.S, I've checked out your site and I'm in awe - really great site, and really cool plugins. Nice!