Audio denoise

Real-time speech denoising — suppress background noise (keyboards, fans, traffic, room hum) from a microphone while keeping the speaker’s voice clean.

It runs as a separate pipeline from the video effect: an AudioWorklet on the audio render thread, in parallel with the GPU video work. The two share nothing but the output MediaStream, so denoise adds no load to the video pipeline.

Hear it

The same ~10-second clip — a noisy microphone recording, then cleaned by each model. Headphones recommended.

Noisy input

DFN3 — dfn (high)

DFN3 int8 — dfnint8 (mid)

RNNoise — rnnoise (low)

Enabling it

Pass audio: 'denoise'. The input stream’s microphone track is denoised in place; pipeline.stream carries the cleaned audio track.

const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true })

const pipeline = new EffectsPipeline(stream, {
  background: 'blur',
  audio: 'denoise',
})

videoEl.srcObject = pipeline.stream   // denoised audio + processed video

pipeline.ready resolves on the first video frame and does not wait on audio — the mic passes through until the denoiser has loaded (a beat after init), then switches over automatically.

Models

Three models cover the hardware range. The default, 'auto', runs a tiny weight-free probe at init: it times the real network on the user’s device without downloading the multi-MB weights, then picks the best fit and downloads only that one.

modeltierwhat it is
'dfn'highDeepFilterNet3, full f32 — best quality
'dfnint8'midDeepFilterNet3 with int8 GRUs — smaller download, faster on weak hardware
'rnnoise'lowclassic RNNoise — tiny, and the fallback where wasm SIMD isn’t available

The two DeepFilterNet models are Longpipe’s hand-written WebAssembly port of DeepFilterNet3; RNNoise is Xiph’s RNNoise.

Force a specific model (or tier shorthand) to skip the probe:

new EffectsPipeline(stream, { audio: { denoise: { model: 'dfn' } } })    // explicit model
new EffectsPipeline(stream, { audio: { denoise: { model: 'high' } } })   // tier: high | mid | low

Options

interface DenoiseOptions {
  model?:          'auto' | 'high' | 'mid' | 'low' | 'rnnoise' | 'dfn' | 'dfnint8'  // default 'auto'
  postFilterBeta?: number    // DFN post-filter — extra suppression of residual noise (default 0.03)
  gruLeak?:        number     // DFN recurrent-state drift bound (default 0.995)
  enabled?:        boolean    // start denoising vs. passthrough (default true)
}

postFilterBeta and gruLeak tune the DeepFilterNet models ('dfn' / 'dfnint8') only; RNNoise ignores them.

Runtime control

pipeline.setDenoise(false)                      // passthrough — stop denoising (cheap to re-enable)
pipeline.setDenoise(true)                       // resume
pipeline.setDenoise({ postFilterBeta: 0.05 })   // tune DFN params live

Inspect what’s running and how it performs:

const s = pipeline.getAudioStats()
// { model: 'dfn', p50Ms: 0.4, p95Ms: 0.6, latencyMs: 12.5, sampleRate: 48000, active: true }

p50Ms / p95Ms are per-hop compute time (the budget is one 2.67 ms audio render quantum); latencyMs is the buffering the denoiser adds.

Audio-only (without the video pipeline)

EffectsPipeline needs a video track. For an audio-only stream — a participant with their camera off, or a mic-settings dialog where you only have the audio track — use the denoiser directly from its own entry point. It brings none of the video pipeline with it: no worker, no GPU, just the AudioWorklet.

import { AudioDenoiser } from 'longpipe/audio'

const mic = await navigator.mediaDevices.getUserMedia({ audio: true })

const denoiser = new AudioDenoiser(mic.getAudioTracks()[0], {
  model: 'auto',                  // same options as DenoiseOptions above
  onError: (msg) => console.warn(msg),
})

peerConnection.addTrack(denoiser.outputTrack)   // live immediately (passthrough until loaded)

const loaded = await denoiser.ready             // true = denoising, false = stayed in passthrough

outputTrack is live as soon as the constructor returns: it carries the raw mic until the model has loaded, then switches to denoised audio. ready resolves to true once denoising is live, or false if the model couldn’t load (onError says why). In that case audio keeps flowing as passthrough, so it never rejects.

The rest of the surface is what EffectsPipeline uses internally:

denoiser.setEnabled(false)                     // bypass (cheap to re-enable)
denoiser.setConfig({ postFilterBeta: 0.05 })   // tune DFN params live
denoiser.getStats()                            // same shape as getAudioStats()
await denoiser.resume()                        // resume a context the browser started suspended
denoiser.destroy()                             // closes its AudioContext

Browsers’ autoplay rules can start an AudioContext suspended until the user interacts with the page, and a suspended context processes no audio. The denoiser tries to resume on construction. If you construct it before any user interaction, also call denoiser.resume() from a click handler, such as your join or unmute button. It does nothing if the context is already running. If the context is still suspended shortly after construction, the denoiser calls onSuspended and resumes by itself on the next tap, click or key press. With EffectsPipeline, call EffectsPipeline.unlockAudio() first thing in the click that starts the call (before any await), or pipeline.resumeAudio() later; while suspended, the output carries the original mic audio (onAudioTrackChanged) and onError reports audio-suspended (non-fatal).

Assets come from the same default CDN as the video weights. If you self-host, pass weightsBaseUrl.

Sample rate

The models run at 48 kHz. If the browser gives the AudioContext a different rate, the SDK resamples in and out automatically — you don’t need to do anything. getAudioStats().sampleRate reflects the actual context rate.

Notes