API reference
EffectsPipeline
The single public class. Hand it a MediaStream, get a MediaStream back with the effect applied. The pipeline owns its own worker, weight loading, autotune, frame transport selection, audio passthrough, and adaptive preset swaps — designed so you don’t have to wire any of that yourself.
Construction returns synchronously: pipeline.stream is wired immediately and emits the unprocessed input until the model is ready (~1–3s on cold start, depending on hardware). Use await pipeline.ready if you want to wait for the effect to actually be live before consuming the output.
const pipeline = new EffectsPipeline(inputStream, options?)
Constructor parameters
inputStream
Type: MediaStream · Required
Any video stream — webcam from getUserMedia(), screen capture from getDisplayMedia(), or a captured stream from a <video> or <canvas> element via .captureStream(). Audio tracks are passed through unchanged by default; pass audio: 'drop' to strip them, or audio: 'denoise' to clean them (see Audio denoise).
options
Type: PipelineOptions · Optional
Shape:
interface PipelineOptions {
background?: BackgroundInput // default: 'blur'
model?: 'longpipe' | 'mediapipe-selfie'
| 'mediapipe-selfie-square' | 'mediapipe-selfie-landscape' // default: 'longpipe'
touchup?: TouchupOptions // default: off
reframe?: boolean | ReframeOptions // default: off
preset?: PresetName | ManualPreset // default: 'auto'
weightsBaseUrl?: string // default: cdn.longpipe.dev/.../v/0.0.5/
audio?: 'passthrough' | 'drop' | 'denoise' | { denoise: DenoiseOptions } // default: 'passthrough'
enabled?: boolean // default: true
outputResolution?: { w: number, h: number } // default: input track size
adaptive?: boolean // default: true
onReady?: (info: ReadyInfo) => void
onError?: (err: PipelineError) => void
onRecovered?: (info: RecoveredInfo) => void
onTrackChanged?: (change: TrackChange) => void
onAudioTrackChanged?: (change: AudioTrackChange) => void
}
Each property in detail:
Blur
To add a blur, you can either set background to 'blur' to use default blur settings,
{
background: 'blur'
}or you can specify an object
{
background: {
blur: {
strength?: number //0 to 1
}
}
}Image
You can set background to a url, an ImageBitmap or an <img> element
{
background: string | HTMLImageElement | ImageBitmap
}Video
You can also set a video background by specifying a video url, or a <video> element, which will play in a loop with each frame being set as the background as the video plays in a loop.
{
background: string | HTMLVideoElement
}Swappable at runtime via pipeline.setBackground(...) — no flicker, no re-init. See Backgrounds for the full input surface.
Which segmentation model runs inside the pipeline. Everything else — the worker, frame transport, WebGPU / WebGL2 backends, zero-copy input and the compositor — is the same either way.
new EffectsPipeline(stream, { model: 'mediapipe-selfie', background: 'blur' })'longpipe'(default): Longpipe’s own models (five tiers, see Presets). Every effect: backgrounds, touch-up, auto-reframe, optical-flow temporal stabilization.'mediapipe-selfie': Google’s MediaPipe Selfie Segmenter. Background effects only. The square (256×256) or landscape (256×144) variant is picked from the output aspect ratio;'mediapipe-selfie-square'/'mediapipe-selfie-landscape'force one.
With a MediaPipe model, touchup, reframe and any preset other than 'auto' throw at construction (…requires a Longpipe model; set model: 'longpipe'), and setTouchup(), setReframe(), reframe() and setPreset() later reject with 'control-failed'. Autotune and the adaptive controller don’t run. See Models.
Skin smoothing. Composes with the background rather than replacing it — both ride the same encoder pass, so enabling it adds the landmark model and the smoothing passes, not a second inference.
new EffectsPipeline(stream, { background: 'blur', touchup: true })
new EffectsPipeline(stream, { touchup: { strength: 0.4, style: 'bilateral' } })strength (0–1, default 0.6) is the knob to expose to users. amount, detail and style tune the look. Smooths every face in frame (up to four, on medium and above).
Needs three extra files from weightsBaseUrl on first enable — see Touch-up. Requires model: 'longpipe'.
Crop and zoom so the subject is framed — “centre stage”. Needs no extra assets or model; it rides the face keypoints the network already produces.
new EffectsPipeline(stream, { reframe: true }) // auto-follow
new EffectsPipeline(stream, { reframe: { auto: false } }) // solve once, then freezeauto: true (default) follows the subject with a deadband — hold, deliberate move, hold. auto: false is manual: it frames once and holds until you call pipeline.reframe(), which is the mode for a reframe button in your own UI.
Applies to you, not to a virtual background. See Auto-reframe. Requires model: 'longpipe'.
Model size selection. 'auto' runs an init-time microbenchmark and picks the largest preset whose per-frame model cost fits the frame budget (~half a 30fps frame) on the user’s hardware; while in 'auto' mode the adaptive controller (see below) can swap up or down at runtime. Explicit choices — 'fast', 'balanced', 'quality', or a tier name 'xs'…'xl' (0.0.20+) — are always respected and never auto-overridden. Pass a ManualPreset object to pin a specific model + dtype + resolution combination. See Presets & autotune. Presets pick a Longpipe tier: with model: 'mediapipe-selfie' only 'auto' (ignored) is accepted.
Where the SDK fetches model weight files. The default points at a public, versioned CDN (https://cdn.longpipe.dev/models/v/0.0.5/) — fine for prototyping. Override to self-host: offline use, locked-down corporate networks, or to avoid the public CDN dependency. The SDK fetches files of the form model_${preset}.bin. With model: 'mediapipe-selfie' the default is https://cdn.longpipe.dev/models/mediapipe/selfie/v1/; if you set weightsBaseUrl, the MediaPipe files are fetched from it too. See Self-hosting weights.
Controls the output stream’s audio. 'passthrough' forwards the input’s audio tracks unmodified — useful for video-call scenarios where audio is part of the same stream. 'drop' strips audio, useful when you only care about the video (e.g., screen recording, frame export, server-side processing). 'denoise' runs the input microphone through the real-time speech denoiser — a separate AudioWorklet pipeline, independent of the video work — and outputs the cleaned track; pass { denoise: { model, postFilterBeta, gruLeak, enabled } } to configure it. See Audio denoise.
Whether the effect is active. false makes the pipeline emit the raw input stream verbatim — no model load, no inference, no compositing. ready still resolves, with preset: null; the first setEnabled(true) fetches the weights. Toggle at runtime via pipeline.setEnabled(true | false) — cheap to re-enable since the worker stays alive.
Dimensions of the output canvas. By default matches the input video track’s intrinsic size — preserves aspect ratio and avoids pointless rescale. Falls back to 1280×720 if the track hasn’t reported its size yet. Set explicitly to force a specific output dimension regardless of input.
Whether the SDK auto-adjusts the preset at runtime when conditions change. Only applies when preset: 'auto' — explicit preset choices are always respected, and calling setPreset() turns it off. When enabled, downgrades to a smaller preset if framerate drops below target, and upgrades when modelMs shows consistent headroom (WebGPU only — WebGL upgrades are too expensive to swap live). The target is the camera’s measured frame rate, capped at 30 fps, so a 15 fps camera isn’t mistaken for an overloaded GPU.
Fires once, when the network is attached and the effect is live (the next frame through the pipe is the first with the effect). With no usable GPU it fires once raw passthrough has settled, with info.backend === 'none'. Same moment pipeline.ready resolves, and with the same ReadyInfo:
interface ReadyInfo {
backend: 'webgpu' | 'webgl2' | 'none'
dtype: 'f16' | 'f32' | null
preset: string | null // resolved model, e.g. 'small'; null with no backend or enabled: false
} Fires for every async / runtime failure, as a typed PipelineError with a type and a fatal flag. That covers no GPU, weight 404, bad backgrounds, GPU device or context loss, worker crashes and hangs, the camera ending, failed or timed-out control calls (including setBackground()), adaptive swap failures and audio problems. The output track keeps carrying video through all of them (the raw camera when the effect can’t run), except 'camera-ended'. Synchronous construction errors (no input video track, transport setup throws) propagate out of new EffectsPipeline(...) directly — wrap construction in try/catch if you want one handler for both.
Fires after the SDK has recovered automatically from a GPU device/context loss or a worker crash: the worker was restarted, the model reloaded, your settings re-applied, and the effect is live again. info is { reason: 'backend-lost' | 'worker-crash', backend, attempts }. Until then the output carried the raw camera.
Fires when the output video track had to be replaced. This only happens on the transfer-capture path (Safari), after a worker crash or GPU loss: that path’s canvas belongs to the worker and can’t be taken back. pipeline.stream is already updated in place. Call sender.replaceTrack(change.videoTrack) on your RTCRtpSender. change is { stream, videoTrack, previousTrack, reason }. Never fires on Chromium or Firefox, where the raw fallback stays on the original track. See Browser support.
Fires when the output audio track changes: if the denoise AudioContext can’t run (autoplay policy, no audio output device), the output switches to your original mic audio, and back to the denoised track once the context runs. pipeline.stream is already updated; call sender.replaceTrack(change.audioTrack). change is { audioTrack, previousTrack, denoised }.
Properties
pipeline.stream: MediaStream— the output stream. Available synchronously; emits passthrough video until the effect is live, and the raw camera whenever the effect can’t run.pipeline.ready: Promise<ReadyInfo>— resolves when the network is attached (not on the first composited frame), or withbackend: 'none'when there’s no usable GPU. Rejects only on a fatal error before that point; later failures never touch it. Optional to await.pipeline.backend: 'webgpu' | 'webgl2' | 'none' | null— the active GPU backend (nulluntil resolved).
Methods
All control methods below return a Promise and follow the same rules (see Control calls): calls made before ready queue; a failure rejects that call with a typed LongpipeError, also goes to onError, and leaves the previous settings running.
pipeline.setBackground(bg: BackgroundInput): Promise<void>— swap the background at runtime. Same input shape as the option. No flicker, no re-init. A bad URL or undecodable image rejects withtype: 'background-load'and keeps the previous background.pipeline.setEnabled(enabled: boolean): Promise<void>— toggle the effect.falseputs the pipeline in passthrough mode.pipeline.setTouchup(opts: TouchupOptions | null): Promise<void>— enable, update or disable skin smoothing at runtime.nulldisables. First call fetches the landmark assets; later calls are instant, so it’s safe on a slider. See Touch-up.pipeline.setReframe(cfg: boolean | ReframeOptions | null): Promise<void>— enable, update or disable auto-reframe.null(orfalse) disables. No fetch. See Auto-reframe.pipeline.reframe(): Promise<void>— recompute the frame now and hold it. The manual-mode entry point — wire it to a “reframe” button. A no-op inautomode, where the camera already follows.pipeline.setDenoise(input: boolean | DenoiseOptions): void— toggle or reconfigure audio denoise at runtime.false= passthrough; an object updates the DFN params live. Only effective whenaudio: 'denoise'was set at construction. See Audio denoise.EffectsPipeline.unlockAudio()(static) — call it first thing in the click that starts your call, before anyawait(e.g.getUserMedia), so the denoiseAudioContextcan start. Firefox otherwise keeps it suspended.pipeline.resumeAudio(): Promise<void>— resume the denoiseAudioContextif the autoplay policy suspended it. Call it from a user gesture such as a join button; the SDK also resumes on the first gesture by itself. No-op withoutaudio: 'denoise'.pipeline.getAudioStats(): AudioStats | null— current denoise telemetry (model, per-hop p50/p95 ms, buffered latency, sample rate).nullwhen denoise isn’t active.pipeline.setPreset(preset: PresetName | ManualPreset, weights?: ArrayBuffer): Promise<void>— switch to a fixed preset ('fast','balanced','quality', a tier name or aManualPreset; not'auto'). Downloads the preset’s weights fromweightsBaseUrlunless you pass them. Disables the adaptive controller from then on. Rejects with a MediaPipemodel.pipeline.setInputStream(stream: MediaStream): Promise<void>— switch the video source (e.g. a camera switch) without changing the output track. Call it before stopping the old camera. Only the video track is switched; audio stays as configured.pipeline.getStats(): Promise<PipelineStats>— runtime counters:fps,modelFps,modelMs,displayMs,skipped,preset,enabled,inputPath,outputPath, plusbackend('webgpu' | 'webgl2' | 'none'),status('starting' | 'running' | 'recovering' | 'passthrough' | 'ended' | 'destroyed'),inputFps(measured camera rate) andrestarts. One postMessage round-trip; never rejects.pipeline.attachPreview(canvas: HTMLCanvasElement): Promise<void>— attach a second canvas for previewing candidate effects. See Preview.pipeline.setPreview({ background, fps? }): Promise<void>— render a candidate effect to the preview canvas. See Preview.pipeline.clearPreview(): Promise<void>— stop previewing.pipeline.destroy(): void— terminates the worker and releases GPU resources. The pipeline is not reusable after destroy.
Preview
Show a user what a candidate effect would look like — e.g. while they browse a background menu — without changing the effect on the outgoing stream. The main output keeps its currently-applied background; a second canvas renders the candidate. The network and matte are computed once and shared between the two, so previewing adds only a second composite (not a second inference).
// 1. Attach a canvas you own. Called ONCE — control is transferred to the
// worker, so the element can't be drawn to by the page afterward. The
// worker sizes the backing to the output resolution; CSS-scale it for display.
pipeline.attachPreview(document.querySelector('#preview'))
// 2. Preview a candidate. `background` is the identical wide surface as the
// main effect (keyword, URL, ImageBitmap, { color }, { blur }, …). 'none'
// previews the no-effect (raw) option. fps throttles the preview (default 15;
// the main output stays the priority).
await pipeline.setPreview({ background: 'https://example.com/beach.jpg', fps: 15 })
// 3. Stop previewing (the canvas freezes on its last frame — hide it in your UI).
pipeline.clearPreview()
To apply a previewed candidate to the outgoing stream, just call pipeline.setBackground(sameInput) — there’s no separate “commit” call.
The preview canvas is local display only; it is not a MediaStream and is not transmitted. On WebGPU it composites directly to your canvas; on WebGL2 it’s presented via a per-frame snapshot of the render canvas (a little more expensive, which is why preview is throttled).
Error handling
The policy is fail open: the output video track always carries something (the effect if possible, otherwise the raw camera frames), and every failure reaches onError as a typed error. Longpipe never causes black video or silent audio.
interface PipelineError {
type: PipelineErrorType // what happened — branch on this
fatal: boolean // true → effects won't come back without a new pipeline
message: string
source: 'worker' | 'backend-lost' | 'adaptive' | 'background' | 'audio' | 'camera' | 'control' | 'unknown'
recoverable: boolean // deprecated: same as !fatal
cause?: unknown
}
Rejected control calls throw a LongpipeError, exported from longpipe. It’s an Error subclass with the same fields, so err instanceof LongpipeError && err.type === 'background-load' works.
type | Fatal | What happened | Output video |
|---|---|---|---|
'no-backend' | no | No usable GPU (no WebGPU, no WebGL2) at startup. ready resolves with backend: 'none'. | Raw camera |
'init-failed' | yes | Weights download, autotune or worker init failed. ready rejects. | Unprocessed frames |
'background-load' | no | A background image/video failed to load or decode (or timed out after 15 s). The previous background stays ('none' at startup). | Unchanged |
'backend-lost' | no | WebGPU device lost / WebGL context lost. The SDK restarts the worker with a fresh backend (WebGL2 after a second WebGPU loss), then fires onRecovered. | Raw camera until recovered |
'worker-crash' | no | The worker errored, stopped answering (no heartbeat for 5 s) or stalled. Restarted as above. | Raw camera until recovered |
'recovery-failed' | yes | Automatic recovery gave up (3 restarts within a minute, or no GPU any more). | Raw camera |
'camera-ended' | yes | The input video track ended (unplugged, permission revoked, stop()). | Ended |
'control-failed' | no | A control call failed. That call rejects; previous settings stay. | Unchanged |
'control-timeout' | no | A control call got no answer in time (10 s; 30 s if queued before ready). | Unchanged |
'adaptive' | no | An adaptive preset swap failed; stays on the current preset. | Unchanged |
'audio' | no | Audio denoise failed. The original audio passes through. | Unaffected |
'audio-suspended' | no | The denoise AudioContext is suspended (autoplay policy, or no audio output device). Call EffectsPipeline.unlockAudio() in your click before any await; the SDK also resumes on the next gesture. | Audio: original mic until it resumes (onAudioTrackChanged) |
Only a fatal error before ready settles rejects ready. Nothing after that, including a failed control call, touches it.
onError does not fire for synchronous constructor errors — those propagate out of new EffectsPipeline(...) directly. Wrap in try/catch if you want a single handler for both.
Control calls
setBackground, setEnabled, setPreset, setTouchup, setReframe, reframe, attachPreview, setPreview and clearPreview all behave the same way:
- Before
ready, or while the SDK is recovering: the call queues, then applies once the pipeline is running. If the pipeline can’t get there (no GPU, init failed, destroyed), the call rejects with that reason’stype. - On failure: the promise rejects with a
LongpipeError('control-failed','control-timeout'or'background-load'),onErrorfires with the same non-fatal error, and the pipeline keeps running with the previous settings. No call can hang. - Across a restart: your settings (background, enabled, touch-up, reframe, a preset you picked) are re-applied to the restarted worker. A call that was in flight when the worker died rejects with
'worker-crash', and its setting is still applied after recovery. The preview canvas can’t be carried over, so attach a new one.
Rejections are pre-handled, so ignoring a returned promise never produces an “Uncaught (in promise)” warning; the error still reaches onError.
See Browser support & reliability for per-browser behaviour, including when the output track can change (onTrackChanged, Safari only).