A Real-Time JSFX Audio Denoiser for REAPER

In this article, I’ll show you how to build a nice real-time denoiser for REAPER using JSFX, its built-in programming language. We’ve used this denoiser in production at the Game Music Ensemble at UCLA, where I used it to remove background noise for the Celeste Celeste Collections videos, as well as in several published pieces from our 2018 International Games Day concert at Powell Library.

Imagine you have an audio signal with noise, and you’d like to remove the noise from the audio, given a clip of the noise. For instance, suppose you record a guitar player with a power supply humming in the background, or a violin player with ambient room tone (which could include the sound of anything from air conditioning units, to people in other rooms, to Brownian noise in the air). Ideally, for the noise you can’t avoid, you’d like to be able to take a small portion of the audio where the artist isn’t playing, and use that as data to recover the full audio track.

In the digital audio workstation REAPER, one frequent workflow for denoising using its built-in tools involves using ReaFIR’s Subtract mode to build a noise profile from the noise sample and to then subtract the magnitude of each frequency band of the noise profile from the magnitude of each frequency band of the source audio. (Here‘s a walkthrough showing how to do that.)

REAFir, REAPER’s built-in FFT tool, denoising audio by subtracting two spectrograms. The dark yellow graph represents the spectrum of the input audio, the red graph represents the captured noise profile, and the bright yellow graph is the spectrum of the output audio.

While this works well for noise samples such as electrical hums (since this in effect removes those particular frequencies from the source audio), this often produces artifacts when parts of of the spectrum of the input audio are at or below the spectrum of the noise. (This can happen even when frequencies of the signal are strictly louder than frequencies of the noise, as the noise phases in and out of sync with the signal as time progresses.)

As it turns out, we can do better using the same amount of data, avoiding many of these artifacts while still producing a clean, denoised signal, by modifying how we combine the source audio and noise profile spectrograms.

Finally, this denoiser is free and open-source – and you can get it and start denoising audio right away at https://github.com/nbickford/REAPERDenoiser! See the instructions on that page for how to install this plugin.

Note – as of July 1st, 2020, the version of REAPERDenoiser on the GitHub page has been slightly modified from the version presented here in order to be compatible with ReaPlugs, which (as of July 1st, 2020) doesn’t have the fft_real function. If you’re following along with this article, you will probably want to follow the version of ReaperDenoiser here – or, for a bit of a challenge, you can follow along with the updated version! This blog post will be updated in the future to guide you through the updated version.

How this Denoiser Works

Here’s the idea behind this denoiser, which is based on Norbert Weiner’s deconvolution algorithm: for each frequency band, we compute its signal-to-noise ratio, and then attenuate it based on how noisy it is. While the previous denoiser would take any bands with a signal-to-noise ratio less than 1 and set their magnitude to 0, this denoiser uses a smooth falloff for all signal-to-noise ratios.

Here’s a bit of a longer explanation. Suppose we have a recording of some audio with noise y(t), and we know the power spectrum of the noise, |N(f)|.

Aside: What’s a power spectrum? What’s a Fourier transform?

We usually think of a signal as something that varies in time. But we can also think of a signal as a sum of components with different frequencies, from low-frequency components (i.e. low sounds) to high-frequency components (i.e. high sounds).

If we have a discrete signal defined on n points (for instance, a second of monophonic audio recorded at a sample rate of 44,100 Hz is a signal defined on 44,100 points), then we can express that signal as a sum of n sine waves of different frequencies, each with their own phase. By doing so, we separate the signal into n frequency bands. The amplitude of each band’s sine wave is then the magnitude of that band, or equivalently the magnitude of that frequency in the signal. We can also decompose many real continuous-time signals (i.e. real-valued functions defined over the real numbers) into an infinite number of sine waves, although there are some restrictions on when the infinite sum of sine waves converges everywhere to the original function.

A Fourier transform takes a signal and expresses it as a sum of sine waves, or as a sum of complex oscillators. (The definition of a particular Fourier transform depends on whether the signal is discrete or continuous, real-valued or complex-valued, the sampling rate if the signal is discrete, and potentially other factors as well, which is why I’m defining it in such generality.)

Fourier transforms are named after Jean-Baptiste Joseph Fourier, but earlier authors dating back to Babylonian mathematics used approximations of functions by sums of sine waves to analyze orbits using epicycles, the motion of vibrating strings, or even heat diffusion. Similar systems have also been used to predict tides, for instance. It turns out we can also use them to separate audio signals into their frequency-space components, or to separate images into low-frequency and high-frequency features.

If you’ve worked with audio processing tools before, you might know about high-pass and low-pass filters, which isolate the high-frequency and low-frequency components of an audio signal. Fourier transforms are even more precise: they separate audio signals into potentially thousands of individual frequency bands, which can be individually controlled. (Usually, in real-time applications and some offline applications, we split apart the audio into many individual clips, and then compute the Fourier transform of each clip independently. There are some subtleties to this, such as window functions, which we’ll talk about later.) After modifying the output of the Fourier transform, we can use an inverse Fourier transform to (conceptually) take all of the individual waves and sum them back together to get a modified signal. They’re one of my favorite audio processing tools, because of this flexibility and versatility that they provide.

Finally, the power spectrum of a signal is a function that takes a frequency and returns the squared magnitude of that frequency band in the signal. These are the squares of what you’re seeing when you look at 1D audio spectrograms in a digital audio workstation, for instance! (2D audio spectrograms plot how the power spectrum of small sections of a signal changes over time.)

Note that there are many different ways to decompose a function (an audio signal) into individual parts, and Fourier transforms include just some of these ways. We could use basis functions other than sine waves, for instance (usually we’ll want these to be orthogonal functions, for computational reasons, but you can also work with non-orthogonal bases – I should probably point to Thomas Roughton’s paper on an efficient method for this that just came out). As one example, wavelet transforms (pioneered by Ingrid Daubechies) separate signals into a sum of wavelets: concentrated, oscillatory functions (often compactly supported – i.e. only nonzero over a finite region, as opposed to sine waves, which are nonzero over almost the entire real line) that trade off lower frequency resolution for higher spatial resolution at higher frequencies.

Aside: Why sine waves?

As mentioned, the Fourier transforms we’ll use express real-valued functions as sums of sine waves. In fact, the Fourier transforms we’ll use operate on functions defined on bounded domains (e.g. clips of audio, instead of audio of infinite length), which means that we’ll actually use a specific set of sine functions – specifically, a set of those that wrap around perfectly if we assume that our domain is periodic. For instance, our Fourier transform would take a signal f(t) that’s 1 second long and sampled at 44,100 Hz and find constants so that

f(t) = c_0 + s_1 \sin(t) + c_1 \cos(t) + s_2 \sin(2t) + c_2 \cos(2t) + \dots + s_{22049} \sin(22049t) + c_{22049} \cos(22049t) + c_{22050} (-1)^t.

This has some interesting issues when attempting to express functions such as sin(2.5t) – this is why we use windowing functions, which we’ll talk about later.

We use these sine functions for two main reasons. The first is that in some sense, sine waves are an idealized form of oscillation that closely matches some natural phenomena. Sine waves appear exactly in the coordinates of objects rotating around an axis or in simple spring-mass systems, for instance; the angle of an ideal pendulum over time is also almost exactly a sine wave, for small angles (its actual motion can be expressed in terms of the Jacobi sn function, or by its Fourier series). Many instruments also have waveforms (at least, in their sustained section) that look like sine functions when magnified, since at some level some part of them acts like a simple harmonic oscillator. However, upon closer inspection, these waveforms are better expressed using sums of sine functions, where the frequencies of the sine functions follow in an arithmetic progression (in other words, in a harmonic series). We usually also speak of inharmonicities, which can be thought of as sine-wave components of these instruments’ waveforms that don’t follow their harmonic series. These overtones (and sometimes undertones), as they’re called, are important; they contribute to the timbre of each instrument that distinguishes each instrument’s sound. (I’ll just note here that ignoring their volume dynamics, ocarinas are actually really close to sine waves; that’s because they’re shaped to try to cancel out any harmonics that could occur, according to ocarinist Devons Mo.)

A frequency spectrum and part of the waveform of an ocarina being played, with some noise in the background. Note how much of the energy of the sound is concentrated in a single peak, while the rest of the signal is more than 30 dB below that!

The second is that sine waves (as the real and complex parts of the complex exponential e^{ift}) are really nice to work with from a mathematical perspective; they’re orthogonal functions, which makes defining the Fourier transform (which decomposes a function into sine waves) from the inverse Fourier transform (which sums sine waves) straightforward; series of complex exponential functions are often easy to sum and often show up in solutions to differential equations; every square-integrable function over an interval has a Fourier series that converges at almost every point on that interval (it will not converge at discontinuities or singularities, for instance); and because exponential functions are nice to work with, the discrete Fourier transform over n points can be computed using a fast Fourier transform in time proportional to n \log{n} instead of using the equation for the discrete Fourier transform directly, which would take time proportional to n^2. (I should note that fast wavelet transforms can improve on fast Fourier transforms by a factor of \log{n} to a time proportional to n, but assume a different structure – this is a topic for another article, though.)

Previously, we tried to denoise y(t) by subtracting the noise spectrum |N(f)| from the input spectrum |Y(f)|, while keeping the phase the same:

\begin{array}{lcl} |X(f)| & = & \text{max}(|Y(f)|-|N(f)|, 0) \\ \text{arg}(X(f)) & = & \text{arg}(Y(f)) \end{array}

where X(f) is the discrete Fourier transform of x(t), the output of the denoiser. We can also write that like this:

X(f) = Y(f) \frac{\text{max}(|Y(f)|-|N(f)|, 0)}{|Y(f)|}

Norbert Wiener’s deconvolution algorithm assumes we have the power spectrum of the original signal, S(f). If we had that, then we could compute a denoised signal as follows:

X(f) = Y(f) \frac{|S(f)|^2}{|S(f)|^2 + |N(f)|^2}.

But since we don’t have S(f), we’ll assume that a constant multiple of Y(f) is a good approximation to it. So, we’ll replace S(f) by Y(f)/k (where k > 0 is some constant, usually around 1) to get

X(f) = Y(f) \frac{|Y(f)|^2/k^2}{|Y(f)|^2/k^2 + |N(f)|^2}.

Finally, we’ll simplify this a bit to get the equation we’ll use for denoising:

\boxed{X(f) = Y(f) \frac{|Y(f)|^2}{|Y(f)|^2 + k^2|N(f)|^2}}

We can think about this as attenuating each band depending on each band’s signal-to-noise ratio. If the signal-to-noise ratio is small (i.e. N(f) is large compared to Y(f)), then we greatly attenuate the band, while if the signal-to-noise ratio is large (which is good), then we leave the band mostly as is. Most importantly, in the case where N(f) = Y(f), this model multiplies band f by \frac{1}{1+k^2}, while the previous model set Y(f) to 0.

Aside: Are there better approximations to S(f) than Y(f)/k, given |N(f)|?

It’s entirely possible! In the process of implementing this denoiser, I tried a few other replacements for S(f), some of which took N(f) into account. In the end, I decided to stick with the more classical substitution. However, when implementing your denoiser, you might want to try a few variants to see what you can find – maybe you’ll find an expression that you like better! In fact, the denoiser for the Celeste Celeste Collections video actually used

X(f) = Y(f) \frac{|Y(f)|}{|Y(f)| + k |N(f)|}

because I was ignoring the squared terms in the denoiser formula.

By the way, here’s a simple puzzle with a surprisingly complex answer (to the best of my knowledge). Take two sine waves with the same frequency, (uniform) random phases, and amplitudes A and B, respectively, and add them together. You’ll get another sine wave with the same frequency, but with a different phase and amplitude. Given A and B, and the fact that the phases are uniformly randomly distributed, what’s the expected amplitude of the sum? (As an example, when A is large and B is small, we’d expect an amplitude close to A.)

Denoiser Implementation: Outline

JSFX effects are usually real-time, so in order to implement the denoiser above, we’ll need to have some method of changing the frequency response of the audio as it streams in (as opposed to an audio effect that takes in an audio file and outputs a different audio file). We’ll have two modes – one mode will record a sample of the noise, while the other will denoise audio using data from this noise sample.

To do this, imagine taking the input audio (which for now we’ll treat as a monophonic signal – i.e. as having only one audio channel) and splitting it into overlapping tiles. When we overlap and add the tiles’ audio signals, we recover the original audio.

Top: the original audio. Middle: Splitting the audio into overlapping tiles. Bottom: With a suitable envelope, the sum of the audio in the tiles is the original audio.

Now consider a single tile. We can use a fast Fourier transform (FFT) to take its audio and express it as a sum of sine waves as described above. If we’re recording the noise sample, we’ll store the FT of the noise in an auxiliary buffer. Then to denoise the audio, we’ll take the FT of the audio and the FT of the noise, apply our transformation to get the FT of the noise, and then apply an inverse fast Fourier transform (IFFT) to get the denoised signal for this tile. When we add together all of the denoised tiles’ audio, we can get a denoised signal.

One might be wondering why it’s necessary to overlap the tiles, instead of partitioning the audio into separate, non-overlapping chunks, transforming each one, and then concatenating the results. The problem is that the FT assumes that the input data is periodic (i.e. loops cleanly). Most chunks’ waveforms will have different starting and ending values, though, so the FT will see a discontinuity at the beginning of every chunk, which will translate to artifacts in the result. We could adjust chunk lengths so that the audio at the beginning and end of every chunk lines up, but this then has problems with FTs being most efficient with power-of-two sizes. Windowing also helps prevent audio at the end of a tile from influencing the audio at the start of the tile, which would lead to discontinuities in the output (although we show an example at the end of this article where you can produce these artifacts in any case to produce interesting effects).

The magnitudes of the Fourier transform don’t care about cyclic rotations of the input signal. So to the magnitudes of the Fourier transform, this waveform that doesn’t loop cleanly (top-left) is the same as a waveform that has a discontinuity (top-right) which sounds like a high-pitched click. You can see this in the spectrograms in the bottom row.
If we apply an envelope such as the Hann window, the signal becomes cyclically continuous, and the spurious overtones (the original signal was a sine wave, so there shouldn’t be any) are drastically reduced!

Instead, we’ll overlap the tiles and apply an envelope to the audio within each tile, so that the audio gracefully fades out and in from and to 0 at the beginning and end of each tile. This adds some ringing to the Fourier transform, but we can minimize this with a good envelope. Here, we’ll use the Hann window,

w(x) = 0.5 - 0.5 \cos(\frac{2 \pi x}{S-1})

where S is the number of samples per tile

for its simplicity, but there are many other windows we could use (see https://en.wikipedia.org/wiki/Window_function for more info on this). (In the plugin, we’ll approximate S-1 by S, since it makes the window functions sum up to 1; otherwise, we’d have to do something such as dividing by the sum of the two window functions.)

By Olli Niemitalo – Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=77818388

Suppose our tiles each have S audio samples. For simplicity when computing the FFT, we’ll want S to be a power of two at least 2 (although fast Fourier transforms exist for arbitrary sizes, JSFX’s fft function is limited to a small set of power-of-two sizes). Larger values of S trade off temporal sharpness for a greater number of frequency bands, so in many plugins, it makes sense to let the user set S – but since this adds some additional complexity to the plugin, we’ll hardcode S to be 2^{14}=16384 samples (or about 1/3rd of a second at a sample rate of 44,100 samples per second). Since we’re using the Hann window, our windows will overlap by S/2 samples, and each input and output sample will be used in two tiles.

This means that every input sample will write to two input buffer tiles, and every output (denoised) sample will read from two output (denoised) buffer tiles. If we’re clever about reusing tiles, this means that we only need to allocate memory for 4 tiles (as well as one more tile for our noise buffer).

Here’s how we do this: Every sample, we read an input sample into the two input tiles and write an output sample from the two output tiles. So then, as soon as we finish reading into an input tile, we can apply the envelope, transform it with the noise data, and write those results into its corresponding output tile. The very next sample, we start reading from the data we just wrote, and the cycle begins again. We have to transform a tile every S/2 samples, and the total latency of this process is S samples. Here’s a diagram demonstrating how this works:

Finally, JSFX zeros out its memory and all of its variables whenever it reloads a script (e.g. when the user resets the position of the playhead in REAPER or when you re-open a project). However, it would be nice if the user didn’t have to re-record their noise sample every time they reloaded the project or rewound the playhead. To do this, we’ll show how to serialize the noise buffer so that it gets saved and reloaded properly.

Let’s Implement a Denoiser! A Step-By-Step Tutorial

Now that we’ve covered how the denoiser works, let’s implement it in a JSFX plugin! You can type in the code below, or read along with the final plugin at https://github.com/Nbickford/REAPERDenoiser/blob/master/REAPERDenoiser. The entire plugin’s about 218 lines long, but more than half of that is whitespace and comments. For this, you’ll need a version of REAPER made after about 2012 (or another JSFX host, although I’ve only tested this in REAPER).

Safety tip: If you’re wearing headphones or earphones, it’s a good idea to make sure they’re not on your ears while you’re writing this effect! It’s too easy to accidentally create positive feedback loops, which will result in your plugin generating lots of loud noise. REAPER will detect this and auto-mute the track, but you still want to take good care of your hearing! By not wearing headphones while you’re writing this, you’ll be able to tell when you’ve accidentally made a feedback loop without damaging your ears – you’ll usually hear sound noise coming out of your headphones when this is the case. For this reason, you’ll often see audio effect creators wearing their headphones around their necks when writing new effects.

Let’s get started!

Step 1/7: Creating a new JS Effect

First, open REAPER, create a new track (by right-clicking on the empty space on the left and clicking “Insert new track”, or by pressing Ctrl-T), and add some audio that you’d like to denoise. (You can use any noisy audio of your own, or here’s some audio from the improvisation sessions for the short film Ovation recorded on a study room piano with an AC unit in the background.)

Open the FX Browser by clicking on the “FX” button on the track, or pressing Ctrl-F. It should look a bit like this:

Then in the menu bar, click “FX” > “Create new JS FX…”, and enter a name for your effect, which you can name almost whatever you like. (Some possible names are REAPERDenoiser, Ratio Denoiser, or MyDenoiser.) After you click OK, you’ll see that you’ve added an effect to the track that looks like this:

Let’s start modifying it! Click on the “Edit…” button in the plugin to open REAPER’s JS Development Environment, which will display the code of the plugin on the left, and all of the plugin’s variables (updating in real-time) on the right.

Clicking on “Open in external editor” will also open the plugin in a text editor, such as Notepad on Windows.

It’s worth noting that you’ve just made a plugin file; if you’d like to move this file to another system, for instance, you can find your plugin in the Effects folder of REAPER’s’ resource folder, which you can find by going to the main REAPER window and clicking on Options > “Show REAPER’s resource path in explorer/finder…”. You may have configured it to be somewhere else, for instance, but on Windows, this is usually in %APPDATA%\REAPER\Effects. Your plugin will be in a file corresponding to whatever you named it above, containing plain text.

Finally, let’s start anew, by selecting all of the code in the JS Development Environment and deleting it.

Step 2/7: Creating a Panel

First, we need to specify the name of the plugin. We do this using the desc: tag. (In JSFX, lines starting with // are comments; the program skips over them, but they can help document what you’re doing.)

desc:Ratio Denoiser (github.com/nbickford/REAPERDenoiser) // That's the description of the plugin. This is how it'll show up in the effect // search dialog, as well as the text at the start of its user interface. We use // it as the first line of the script per the JSFX documentation's // recommendation (https://www.reaper.fm/sdk/js/js.php#js_file)

We put this line at the top of the file, because it allows REAPER to quickly identify the file as a JSFX file (since it can check to see if the first five letters are “d”, “e”, “s”, “c”, and “:”).

Once you’ve done that, press Ctrl-S (Command-S on Mac) to save the file. You should see a cyan bar saying RECOMPILED OK! show up at the bottom-left corner. That’s a good thing! JSFX is a compiled language, so instead of reading each line of the program as it goes, it tries to compile the entire program into an intermediate representation before running it.

Now if you go back to the FX window, you should see that your plugin’s panel has the description you typed in:

Next, we’ll add two controls, a combo box (to allow the user to switch between recording noise and denoising audio) and a slider that controls the amount of denoising, k, described above. Both of these are technically sliders in JSFX, since there a combo box is a slider where you can only choose a few options, and each of the options are named.

// This defines a combo box that allows the user to select "Denoise Input" or // "Record Noise Sample". The default value is 0 (Denoise Input). The maximum // value is 1 (Record Noise Sample), and it increases in steps of 1. slider1:0<0,1,1{Denoise Input, Record Noise Sample}>Noise Collection Mode // This defines a slider that can be varied between 0.0 and 10.0 in steps of // 0.001, with default value 1.0. (If slider2 is equal to 0.0, this plugin // shouldn't really do anything to the input audio.) slider2:1<0.0,10.0,0.001>Noise Scale

Save this again (which will recompile the plugin again) and look back at the plugin window. You should now see a combo box and a slider underneath the description!

Let’s do one more thing for the plugin’s user interface: at the moment, you’ll notice that REAPER thinks the plugin takes MIDI input. That’s because we haven’t told REAPER that the plugin has any input or output audio channels, or pins in JSFX terminology. We can add them by adding in_pin and out_pin labels, one for each of the left and right channels:

// Here we can label our input and output pins. This also tells REAPER how many // channels we can handle. In this case, the plugin is stereo (a monophonic // plugin would be simpler, but I almost always use this to denoise stereo // audio), so we define two input and output pins. in_pin:Noisy Audio 1 in_pin:Noisy Audio 2 out_pin:Denoised Audio 1 out_pin:Denoised Audio 2

Save the JSFX file again. You should now see that the plugin window says “2 in 2 out” where it used to say “MIDI”, and if you click on “2 in 2 out”, you should see the labels we’ve given to the pins, above the Routing Matrix:

Aside 3/7: Some Notes on JSFX

JSFX is an imperative programming language, like C, C++, C#, Python, or Javascript, so the syntax might look familiar if you’ve used a programming language before. Each line instructs the program to do something (or sometimes multiple things), and you can control which instructions get run next using control flow statements, such as branches (“if x is less than 2, do this, otherwise do that”) or for loops (“do this 50 times”). The syntax for control flow statements can be a bit unusual, so we’ll discuss that when we come to it.

Interestingly, in JSFX, plugins have a small and fixed amount of memory. There’s nothing in the standard library for allocating and deallocating memory, although libraries such as CookDSP provide memory managers that can do this. Instead, by default, a JSFX plugin’s memory consists of just over 8 million (specifically, 2^{23}=8,388,608 double-precision floating point memory slots, numbered 0, 1, …, 8388607. However, if 8 million memory slots aren’t enough, you can request up to 32 million (probably 2^{25}=33,554,432, though I haven’t checked this) memory slots using the maxmem option. Still, this could be considered small compared to other types of programs, for instance.

In any case, this means that we don’t have to allocate memory; all the memory we’ll use is already allocated for us!

All variables in JSFX are double-precision (64-bit) IEEE 754 floating-point numbers (one sign bit, so they can be positive or negative, 11 bits of exponent, so they can represent values from 2.23 * 10^{-308} to 1.80 * 10^{308} , and 52 fraction bits, so they have about 16 decimal digits of precision). This also means that any integer between -2^{53}-1 and 2^{53}+1 can be represented exactly, so in practice you usually don’t often have to worry about differences between integer and floating-point arithmetic.

JSFX does a lot of conversion implicitly; for instance, suppose we have a variable, foo, equal to 100.0. If we write

x = foo[3];

x will be set to the value in memory slot 100.0 + 3.0 = 103. JSFX does this by taking foo (100.0), adding it to the numeric value 3 (3.0) to get 103.0, converting this to an integer to get 103, and then getting that address in memory. (I’m guessing this might be the reason JSFX plugins are limited by default to 2^{23} memory addresses; 32-bit floating-point numbers, which will be involved at the end of this article, can represent numbers up to 2^{24} exactly, so perhaps the sizes are related.)

Finally, almost all variables in JSFX are global (except for local function variables, which we’ll talk about later) and are defined automatically. This means that you have to be careful about your spelling: for instance, the JSFX code

numSamples = 100; munSmaplss = numSamples + 1;

will create two variables, numSamples and munSmaplss; numSamples will be set to 100, while munSmaplss will be 101 (numSamples+1), even though the author probably meant to write

numSamples = 100; numSamples = numSamples + 1;

which would have set numSamples to 101. This won’t produce any errors (both are valid JSFX code samples), so it’s good to be aware about this!

Step 4/7: Initialization

Initialization takes place inside the @init block. This is where we’ll set the default values of all of our variables, including those that point to locations in memory. First, let’s define the size of the FFT we’ll be using (I’m using all-caps here to denote that this value should be constant, since JSFX doesn’t have a const keyword):

@init // On initialization, initialize all of our variables. // The FFT size will always be constant. SIZE = 16384;

Now let’s write out where our audio buffers will be stored. Since each channel uses 5 audio buffers of length SIZE, we’ll have 10 audio buffers of length SIZE in total. I’ll use separate variables for their names for simplicity, but you could definitely do more sophisticated things here!

// We don't do any allocation in this plugin, since we know we start out with 8M // words of memory. So all we need to do is construct some pointers to memory, // where we'll store our data. // Since we have two channels, we'll have 10 buffers of length SIZE. bufferI1L = 0; // The left input tile 1 buffer starts at memory address 0. bufferI2L = SIZE; // The left input tile 2 buffer starts at memory address SIZE. bufferO1L = 2*SIZE; // The left output tile 1 buffer starts at address 2*SIZE. bufferO2L = 3*SIZE; // And so on noiseBufferL = 4*SIZE; // The noise buffer starts at memory address 4*SIZE bufferI1R = 5*SIZE; // Right channels bufferI2R = 6*SIZE; bufferO1R = 7*SIZE; bufferO2R = 8*SIZE; noiseBufferR = 9*SIZE;

We can be respectful of memory usage and further tell REAPER that we’ll only use the first 10*SIZE items of memory – this allows REAPER to free the rest of the memory and let other plugins use it. The documentation contradicts itself slightly on whether this is 0-indexed or 1-indexed, so I’ll add 1 slot (64 bits) here for safety:

freembuf(10*SIZE + 1);

When we’re processing an audio sample, we’ll need to know which sample we’re processing. We’ll keep track of this using samplesCollected, which will hold our position in the last of the two tiles:

// samplesCollected will be our position in the last of the two tiles. // As such, it'll range from 0 to (SIZE/2) - 1. // (In other words, our position in the first tile will be // samplesCollected + SIZE/2, and our position in the second tile will be // samplesCollected) samplesCollected = 0;

Finally, let’s give REAPER more information about the plugin. Our algorithm has a delay of SIZE samples, but if we tell REAPER that we have that much of a delay, REAPER can automatically run the plugin SIZE samples ahead of time and make it sound like there’s no delay at all! We also have to tell REAPER that the channels that will be delayed are channels 0 and 1 (i.e. all channels from 0 to 2, not including 2):

// Finally, the algorithm we use outputs modified audio SIZE samples after we // input it. If we tell REAPER that the plugin has a delay of SIZE samples, // REAPER can automatically compensate for this and make it appear as if there's // no delay at all. pdc_delay = SIZE; pdc_bot_ch=0; pdc_top_ch=2;

Technically, this means that you can vary the delay of the plugin programmatically, from code, as the plugin is running! However, the JSFX documentation would like you not to change this too often (for instance, every sample).

Step 5/7: Processing Audio

Now for the main part of the plugin! The code in the @sample block will be executed for each audio sample that passes through the plugin (e.g. 44,100 times per second if the audio is running at 44.1 kHz).

First of all, let’s define a function that computes the attenuation factor in the denoising algorithm,


This function takes 5 arguments, all of which are real numbers: the real and imaginary parts of Y, the real and imaginary parts of N, and k^2. You can also see that we tell JSFX that this function will use two local variables, yNorm and nNorm. This makes it so that if we accidentally had another variable named yNorm or nNorm outside of this function, we wouldn’t overwrite the global variable yNorm with the value of the local variable yNorm. (This is important if you’re making a library, for instance; you don’t want to have to tell users “because you’re using this library, you can’t name your variables anything in the following list”!)

You’ll also notice that we have a factor of \frac{1}{2*\text{SIZE}} in computeAttenuation. Fourier transforms usually have normalization factors somewhere that have to multiply together to make \frac{1}{SIZE}. For instance, Mathematica usually applies a factor of \frac{1}{\sqrt{SIZE}} to both its Fourier transform and inverse Fourier transform, while we apply the entire factor of \frac{1}{SIZE} in a single place. I’m not sure why we need the extra factor of 2 here, to be honest – I’m guessing it has to do that we’re using the version of JSFX’s FFT that takes a real audio signal instead of a complex audio signal. This was determined empirically by looking at the input and output values that this plugin produces.

@sample // We'll use this function to compute the attenuation factor used in Norbert // Weiner's denoising algorithm, // }Y(f)|^2/(|Y(f)|^2 + k^2 |N(f)|^2), // given the real and imaginary parts of Y(f), the real and imaginary parts of // N(f), and k^2. This also tells the JSFX compiler that we'll use two local // variables, yNorm and nNorm. This function also includes a factor of // 1/(2*SIZE) that we appear to need to normalize the FFT. function computeAttenuation(yReal yImag nReal nImag kSquared) local(yNorm nNorm) ( // Compute |Y(f)|^2 = real(Y(f))^2 + imaginary(Y(f))^2 yNorm = sqr(yReal) + sqr(yImag); // The same for the noise component: nNorm = sqr(nReal) + sqr(nImag); // The function returns the last thing computed in it: yNorm / (2.0 * SIZE * (yNorm + kSquared * nNorm)); );

Now, we start out by getting the samples from our left and right audio channels.

// Read our input audio and write it into the input buffers. // Channels are zero-indexed. sampleLeft = spl0; // This might look like "sample 10", but it's "SamPLe 0", // i.e. the first (left) channel. sampleRight = spl1; // Similarly, this is sample 1, the second (right) channel.

Next, let’s compute our positions in the audio buffers in tile 1 and tile 2 (you’ll remember that we have two tiles per audio channel, so we have both left and right tile 1s and left and right tile 2s):

// Compute our positions in tile 1 and tile 2 for conciseness tilePos1 = samplesCollected + SIZE/2; tilePos2 = samplesCollected;

Then we compute the envelope that we’re applying for each tile, and finally multiply our audio samples by the envelopes and store the results in the tile buffers.

// We'll apply each tile's envelope as we write the sample into // the tile's buffer. // See https://en.wikipedia.org/wiki/Window_function#Hann_and_Hamming_windows hannWindowTile1 = 0.5 - 0.5 * cos(2*$pi*tilePos1/SIZE); hannWindowTile2 = 0.5 - 0.5 * cos(2*$pi*tilePos2/SIZE); // Write into the input buffers: bufferI1L[tilePos1] = sampleLeft * hannWindowTile1; bufferI2L[tilePos2] = sampleLeft * hannWindowTile2; bufferI1R[tilePos1] = sampleRight * hannWindowTile1; bufferI2R[tilePos2] = sampleRight * hannWindowTile2;

At the moment, we aren’t outputting any audio at all. You could make sure that the envelopes sum up correctly by writing some code like this:

spl0 = sampleLeft * (hannWindowTile1 + hannWindowTile2); spl1 = sampleRight * (hannWindowTile1 + hannWindowTile2);

Instead, we’ll output audio by summing the results from our processed tiles:

// For the output audio, read from the two tiles and sum their results. spl0 = bufferO1L[tilePos1] + bufferO2L[tilePos2]; spl1 = bufferO1R[tilePos1] + bufferO2R[tilePos2];

Then increment the number of samples collected by 1, since we’ve just read and written a sample:

samplesCollected += 1;

Now for the part that does the denoising! Every SIZE/2 samples, we denoise the tiles that we just finished. We do this inside @sample using the following if statement: when samplesCollected is equal to SIZE/2, we reset samplesCollected to 0, and then do a lot of processing (below). (This is roughly equivalent to if(samplesCollected == SIZE/2) { samplesCollected = 0; ...} in C.)

// When we finish a tile, samplesCollected is equal to SIZE/2. // When that happens, we transform the contents of tile 1 and write them to // output tile 1. Then we swap tiles 1 and 2 for both the input and output tiles // and set samplesCollected to 0, so that we're back to where we started. samplesCollected == SIZE/2 ? ( samplesCollected = 0;

We’ve just finished reading tile 1 for the left and right channels. Let’s take the Fourier transform of it:

// This computes the real FFT of each buffer in-place: fft_real(bufferI1L, SIZE); fft_real(bufferI1R, SIZE); // The different frequency bins are now stored in permuted order. We need to // call fft_permute to get them in order of their frequencies. // See https://www.reaper.fm/sdk/js/advfunc.php#js_advanced for more info. fft_permute(bufferI1L, SIZE/2); fft_permute(bufferI1R, SIZE/2); // Our buffers now look like // [band 0 amplitude, band SIZE/2-1 amplitude, // real part of band 1, imaginary part of band 1, // real part of band 2, imaginary part of band 2, // ... // real part of band SIZE/2-2, imaginary part of band SIZE/2-2]

If Slider 1 is set to “Record Noise Sample”, then the variable slider1 will be 1 (as we specified when laying out the combo box). So we can say “if slider1 is greater than 0.5, copy the Fourier transforms of the buffers into noiseBufferL and noiseBufferR“:

// If slider1 is greater than 0.5 (i.e. the user selected "Record Noise // Sample", we store the FFTs of each of these buffers. slider1 > 0.5? ( // memcpy(A, B, C) copies C words from the memory starting at B into the // memory starting at A. memcpy(noiseBufferL, bufferI1L, SIZE); memcpy(noiseBufferR, bufferI1R, SIZE); );

Now we prepare to apply the filtering algorithm by computing k^2 ahead of time:

// Apply Norbert Weiner's filtering algorithm, // X(f) = Y(f) * (|Y(f)|^2)/(|Y(f)|^2 + k^2 |N(f)|^2) // sqr() computes the square of a number, and abs() computes the absolute // value of a number. kSquared = sqr(slider2); // slider2 is the Noise Scale from above.

Now we get to use the computeAttenuation function we defined earlier! I mentioned before that JSFX lays out its FFT bands in a slightly unusual order. The values of the Fourier transform for band 0 (the DC coefficient, the sum of all of the values in the audio buffer) and band SIZE/2-1 (the result of alternately adding and subtracting the elements in the audio buffer) are both real, so JSFX can store them in the first two entries of the FFT result. The rest of the bands, from 1 to SIZE/2-2, are complex-valued, so JSFX stores their real and imaginary part right next to each other, in pairs. In practice, this means that we need to handle bands 0 and SIZE/2-1 as special cases, since they’re the only bands that use only one element. They have 0 imaginary part, so we pass 0 to yImag and nImag. Notice that we also copy data from our input tiles to our output tiles here.

// First, handle band 0 and band SIZE/2-1 separately, since as we've seen, // they're laid out in memory differently than the rest of the bands. // They also have zero imaginary part. bufferO1L[0] = bufferI1L[0] * computeAttenuation(bufferI1L[0], 0, noiseBufferL[0], 0, kSquared); bufferO1R[0] = bufferI1R[0] * computeAttenuation(bufferI1R[0], 0, noiseBufferR[0], 0, kSquared); bufferO1L[1] = bufferI1L[1] * computeAttenuation(bufferI1L[1], 0, noiseBufferL[1], 0, kSquared); bufferO1R[1] = bufferI1R[1] * computeAttenuation(bufferI1R[1], 0, noiseBufferR[1], 0, kSquared);

Now we attenuate the rest of the frequency bands. JSFX’s main control flow structure for executing loops is loop, which runs a block of code a certain number of times. (JSFX also has while, which can also be useful). Here, we’ll iterate over the SIZE/2-1 bands from band 1 to SIZE/2-1, call computeAttenuation to determine how much each band should be attenuated, and use this to copy data from the input buffer to the output buffer.

// Now, loop over bands 1 to SIZE/2-1. bandIndex = 1; loop(SIZE/2-1, // Compute output index outIndex = bandIndex * 2; // Compute attenuation factor and apply it to the output: attenuationFactor = computeAttenuation( bufferI1L[outIndex], bufferI1L[outIndex+1], noiseBufferL[outIndex], noiseBufferL[outIndex+1], kSquared ); bufferO1L[outIndex] = bufferI1L[outIndex] * attenuationFactor; bufferO1L[outIndex+1] = bufferI1L[outIndex+1] * attenuationFactor; // Do the same thing again, but for the right channel: attenuationFactor = computeAttenuation( bufferI1R[outIndex], bufferI1R[outIndex+1], noiseBufferR[outIndex], noiseBufferR[outIndex+1], kSquared ); bufferO1R[outIndex] = bufferI1R[outIndex] * attenuationFactor; bufferO1R[outIndex+1] = bufferI1R[outIndex+1] * attenuationFactor; // Next band bandIndex += 1; );

Now we’ve got data in our output buffers, but we need to convert them from the frequency domain to the time domain! Since we went from the time domain to the frequency domain by calling fft_real and then fft_permute, we reverse this to call fft_ipermute and then ifft_real. (One analogy is that this is like wrapping a book with wrapping paper and a bowtie; to wrap the book, you wrap it with paper and then tie the bowtie, while to unwrap the book you untie the bowtie and then unwrap the paper.)

// Convert back from the frequency domain to the time domain: fft_ipermute(bufferO1L, SIZE/2); fft_ipermute(bufferO1R, SIZE/2); ifft_real(bufferO1L, SIZE); ifft_real(bufferO1R, SIZE);

Finally, we’ve finished processing tile 1. We’re now going to write into the second half of what was tile 2 and start overwriting tile 1, so we swap tiles 1 and 2 as described above. Finally, we add a closing parenthesis to end our if statement (samplesCollected == SIZE/2 ?) that we started earlier.

// Finally, swap our tiles: temp = bufferI1L; bufferI1L = bufferI2L; bufferI2L = temp; temp = bufferI1R; bufferI1R = bufferI2R; bufferI2R = temp; temp = bufferO1L; bufferO1L = bufferO2L; bufferO2L = temp; temp = bufferO1R; bufferO1R = bufferO2R; bufferO2R = temp; )

Step 6/7: Test the denoiser!

If you’ve entered in the code correctly, you should now be able to run and test your denoiser!

First, save your code and fix any compilation errors that the JS Development Environment finds. Then, making sure that you’ve turned your volume down, move the play cursor in REAPER to a section without any input audio, and press the play button in REAPER. You shouldn’t hear anything out of the speakers – if this is the case, congrats! Otherwise, if you hear feedback out of the speakers even though no audio’s coming in (or REAPER has automuted one of the tracks for getting too loud), check your code against the reference at https://github.com/Nbickford/REAPERDenoiser/blob/master/REAPERDenoiser and see if you can spot any differences. (https://www.diffchecker.com/ is a useful tool for comparing files if you don’t have a diff tool installed. I had a problem like this earlier; it was because of the buffer swapping at the end of the plugin accidentally swapping some of the input buffers with the output buffers!)

Next, slide the Noise Scale slider to its maximum value, go back to the start of the track, and press play. Select “Record Noise Sample” in the plugin during a part with only noise to collect a noise sample, then switch it back to “Denoise Input” just before the rest of the audio enters and listen to the result. The noise should be greatly attenuated, but you should also hear some artifacts; dial back the Noise Scale slider to reach a value that balances between these two.

You can also automate Noise Collection Mode using its track envelope to automatically capture a noise sample! This helps because at the moment, REAPER resets all of the audio buffers – including the buffers for the Fourier transforms of the noise samples! – when we reset the play cursor. In fact, this automation was how I denoised tracks using this plugin for a while, before I learned about JSFX serialization!

Step 7/7: Serialize the noise buffers!

When the user resets the position of the play cursor or saves the file, for instance, we’d like to write out the entries of the noise buffers. Then when the plugin gets reloaded, we’d like to reload the entries of the noise buffers from the place where we stored them.

JSFX has a nice API for this – we can use the same functions for both writing data from memory to a stream, and for reading from a stream to memory! Here’s how you can think of it working:

When writing to a stream (e.g. saving a project):

  • REAPER opens a stream for writing.
  • REAPER writes all of the sliders’ values to the stream
  • REAPER calls the plugin’s @serialize section.
    • For instance, here the denoiser would write the contents of noiseBufferL and noiseBufferR.
  • REAPER closes the stream.

When reading from a stream (e.g. loading a project):

  • REAPER opens a stream for reading.
  • REAPER reads all of the sliders’ values from the stream.
  • REAPER calls the plugin’s @serialize section.
    • For instance, here the denoiser would read the contents of noiseBufferL and noiseBufferR.
  • REAPER closes the stream.

(Note that REAPER might call @init before or after @serialize.)

Here, we’ll use the file_mem function to write or read data from memory. When writing to a stream, file_mem copies a certain amount of data from the memory pointed to by its second argument to the stream. But when reading from a string, file_mem copies a certain amount of data from the stream to the memory pointed to by its second argument. So, here’s all we need to do for file serialization:

@serialize // Sliders are serialized automatically, so all we have to serialize is the two // noise buffers. JSFX's serialization works in a clever way: when reading the // state of the plugin from a serialized version, these functions copy data into // noiseBufferL and noiseBufferR. But when writing out the state of the plugin, // they work the other way, copying data out of noiseBufferL and noiseBufferR. file_mem(0, noiseBufferL, SIZE); file_mem(0, noiseBufferR, SIZE);

The noise buffers should now have their state stored and loaded as needed, and you’ll no longer have to program automation to capture noise samples.

Finally, here’s one interesting thing about this serialization procedure: although JSFX’s numbers are all double-precision floating-point numbers, serialized values are generally stored as 32-bit single-precision floating-point numbers! In other words, some quantization happens when serializing values, like taking a 24-bit audio file and converting it to 16-bit. However, this error seems to be small enough in our case that we can ignore it.

Concluding Notes

By this point, you’ve made a working denoiser – congratulations! Here are some final notes before the post wraps up.

Denoiser limitations: If you run this denoiser with a high Noise Scale, you might notice that you seem to get ghostly echoes before certain transients (i.e. starts of notes). This is an artifact of this denoiser that the current approach doesn’t solve.

You can hear this artifact a bit in the GME Celeste Celeste Collections video above – listen for the slight pre-echo around 0:38 or 1:45. I decided to leave this in, because I sort of liked this ghostly effect in this context. This video also used an earlier version of the denoiser,


since I originally ignored the squared terms in the denoiser formula.

To see why we hear this echoing artifact, first note that you can think of the current denoising technique as circularly convolving each tile’s audio with an impulse response that depends on the tile’s audio and the noise profile. In a convolution reverb effect, for instance, this impulse response would start off with a peak, then decay down to 0. However, this denoiser doesn’t take the direction of time into account, so it’s perfectly happy to create echoes before the transient as it is to create echoes after the transient! However, the echoes after the transient usually sound a lot like the instrument’s natural reverberation, so we only hear the echoes before the transient.

One solution to this would be to compute the impulse response that’s being applied to each tile, and then to process the impulse response to remove these pre-echoes (at the expense of adding in some additional noise). I’m not sure what the best way to do this is, however. You’d probably also want to modify the algorithm to perform a standard convolution instead of a circular convolution, which would require larger tile sizes.

Exceeding slider bounds: One fun thing about JSFX sliders is that REAPER doesn’t automatically check to make sure that the values of the sliders are in range. Since we also didn’t add any logic to make sure the value of the Noise Scale slider, for instance, is between 0 and 10, this means that we can type in any value we want into the text box to the right of the Noise Scale slider, and have the plugin use that value! This can sometimes cause JSFX plugins to do really interesting things.

Here’s what this plugin sounds like denoising a piece of audio using a Noise Scale of 1000 and S=32768. At this level, each tile’s audio is replaced by a sum of its most prominent sine waves – since sine waves by themselves don’t have any envelopes, tiles now have discontinuities with their neighbors, and you get an interesting artificial bell-like effect:

Here’s what the spectrogram of the processed audio looks like. Time reads from left to right, while the y axis charts low frequencies to high frequencies. You can see that most frequencies have been significantly attenuated, and that there are tonal discontinuities where tiles overlap.

Finally, as mentioned above, you can find a downloadable copy of the denoiser and its associated files at https://github.com/nbickford/REAPERDenoiser. It’s available under the MIT License, so you can use it for any commercial or non-commercial project (although without any warranty).