API reference

EffectsPipeline

The single public class. Hand it a MediaStream, get a MediaStream back with the effect applied. The pipeline owns its own worker, weight loading, autotune, frame transport selection, audio passthrough, and adaptive preset swaps — designed so you don’t have to wire any of that yourself.

Construction returns synchronously: pipeline.stream is wired immediately and emits the unprocessed input until the model is ready (~1–3s on cold start, depending on hardware). Use await pipeline.ready if you want to wait for the effect to actually be live before consuming the output.

const pipeline = new EffectsPipeline(inputStream, options?)

Constructor parameters

inputStream

Type: MediaStream  ·  Required

Any video stream — webcam from getUserMedia(), screen capture from getDisplayMedia(), or a captured stream from a <video> or <canvas> element via .captureStream(). Audio tracks are passed through unchanged by default; pass audio: 'drop' to strip them, or audio: 'denoise' to clean them (see Audio denoise).

options

Type: PipelineOptions  ·  Optional

Shape:

interface PipelineOptions {
  background?:        BackgroundInput              // default: 'blur'
  model?:             'longpipe' | 'mediapipe-selfie'
                    | 'mediapipe-selfie-square' | 'mediapipe-selfie-landscape'   // default: 'longpipe'
  touchup?:           TouchupOptions               // default: off
  reframe?:           boolean | ReframeOptions     // default: off
  preset?:            PresetName | ManualPreset    // default: 'auto'
  weightsBaseUrl?:    string                       // default: cdn.longpipe.dev/.../v/0.0.5/
  audio?:             'passthrough' | 'drop' | 'denoise' | { denoise: DenoiseOptions }   // default: 'passthrough'
  enabled?:           boolean                      // default: true
  outputResolution?:  { w: number, h: number }     // default: input track size
  adaptive?:          boolean                      // default: true
  onReady?:           (info: ReadyInfo) => void
  onError?:           (err: PipelineError) => void
  onRecovered?:       (info: RecoveredInfo) => void
  onTrackChanged?:    (change: TrackChange) => void
  onAudioTrackChanged?: (change: AudioTrackChange) => void
}

Each property in detail:

background BackgroundInput default 'blur'

Blur

To add a blur, you can either set background to 'blur' to use default blur settings,

{
  background:  'blur'
}

or you can specify an object

{
  background: {
    blur: {
      strength?: number //0 to 1
    }
  }
}

Image

You can set background to a url, an ImageBitmap or an <img> element

{
  background: string | HTMLImageElement | ImageBitmap
}

Video

You can also set a video background by specifying a video url, or a <video> element, which will play in a loop with each frame being set as the background as the video plays in a loop.

{
  background: string | HTMLVideoElement
}

Swappable at runtime via pipeline.setBackground(...) — no flicker, no re-init. See Backgrounds for the full input surface.

model 'longpipe' | 'mediapipe-selfie' | … default 'longpipe'

Which segmentation model runs inside the pipeline. Everything else — the worker, frame transport, WebGPU / WebGL2 backends, zero-copy input and the compositor — is the same either way.

new EffectsPipeline(stream, { model: 'mediapipe-selfie', background: 'blur' })
  • 'longpipe' (default): Longpipe’s own models (five tiers, see Presets). Every effect: backgrounds, touch-up, auto-reframe, optical-flow temporal stabilization.
  • 'mediapipe-selfie': Google’s MediaPipe Selfie Segmenter. Background effects only. The square (256×256) or landscape (256×144) variant is picked from the output aspect ratio; 'mediapipe-selfie-square' / 'mediapipe-selfie-landscape' force one.

With a MediaPipe model, touchup, reframe and any preset other than 'auto' throw at construction (…requires a Longpipe model; set model: 'longpipe'), and setTouchup(), setReframe(), reframe() and setPreset() later reject with 'control-failed'. Autotune and the adaptive controller don’t run. See Models.

touchup TouchupOptions default off

Skin smoothing. Composes with the background rather than replacing it — both ride the same encoder pass, so enabling it adds the landmark model and the smoothing passes, not a second inference.

new EffectsPipeline(stream, { background: 'blur', touchup: true })
new EffectsPipeline(stream, { touchup: { strength: 0.4, style: 'bilateral' } })

strength (0–1, default 0.6) is the knob to expose to users. amount, detail and style tune the look. Smooths every face in frame (up to four, on medium and above).

Needs three extra files from weightsBaseUrl on first enable — see Touch-up. Requires model: 'longpipe'.

reframe boolean | ReframeOptions default off

Crop and zoom so the subject is framed — “centre stage”. Needs no extra assets or model; it rides the face keypoints the network already produces.

new EffectsPipeline(stream, { reframe: true })              // auto-follow
new EffectsPipeline(stream, { reframe: { auto: false } })   // solve once, then freeze

auto: true (default) follows the subject with a deadband — hold, deliberate move, hold. auto: false is manual: it frames once and holds until you call pipeline.reframe(), which is the mode for a reframe button in your own UI.

Applies to you, not to a virtual background. See Auto-reframe. Requires model: 'longpipe'.

preset 'auto' | 'fast' | 'balanced' | 'quality' | 'xs' | 'small' | 'medium' | 'large' | 'xl' default 'auto'

Model size selection. 'auto' runs an init-time microbenchmark and picks the largest preset whose per-frame model cost fits the frame budget (~half a 30fps frame) on the user’s hardware; while in 'auto' mode the adaptive controller (see below) can swap up or down at runtime. Explicit choices — 'fast', 'balanced', 'quality', or a tier name 'xs'…'xl' (0.0.20+) — are always respected and never auto-overridden. Pass a ManualPreset object to pin a specific model + dtype + resolution combination. See Presets & autotune. Presets pick a Longpipe tier: with model: 'mediapipe-selfie' only 'auto' (ignored) is accepted.

weightsBaseUrl string default cdn.longpipe.dev/...

Where the SDK fetches model weight files. The default points at a public, versioned CDN (https://cdn.longpipe.dev/models/v/0.0.5/) — fine for prototyping. Override to self-host: offline use, locked-down corporate networks, or to avoid the public CDN dependency. The SDK fetches files of the form model_${preset}.bin. With model: 'mediapipe-selfie' the default is https://cdn.longpipe.dev/models/mediapipe/selfie/v1/; if you set weightsBaseUrl, the MediaPipe files are fetched from it too. See Self-hosting weights.

audio 'passthrough' | 'drop' | 'denoise' | { denoise: DenoiseOptions } default 'passthrough'

Controls the output stream’s audio. 'passthrough' forwards the input’s audio tracks unmodified — useful for video-call scenarios where audio is part of the same stream. 'drop' strips audio, useful when you only care about the video (e.g., screen recording, frame export, server-side processing). 'denoise' runs the input microphone through the real-time speech denoiser — a separate AudioWorklet pipeline, independent of the video work — and outputs the cleaned track; pass { denoise: { model, postFilterBeta, gruLeak, enabled } } to configure it. See Audio denoise.

enabled boolean default true

Whether the effect is active. false makes the pipeline emit the raw input stream verbatim — no model load, no inference, no compositing. ready still resolves, with preset: null; the first setEnabled(true) fetches the weights. Toggle at runtime via pipeline.setEnabled(true | false) — cheap to re-enable since the worker stays alive.

outputResolution { w: number, h: number } default input track size

Dimensions of the output canvas. By default matches the input video track’s intrinsic size — preserves aspect ratio and avoids pointless rescale. Falls back to 1280×720 if the track hasn’t reported its size yet. Set explicitly to force a specific output dimension regardless of input.

adaptive boolean default true

Whether the SDK auto-adjusts the preset at runtime when conditions change. Only applies when preset: 'auto' — explicit preset choices are always respected, and calling setPreset() turns it off. When enabled, downgrades to a smaller preset if framerate drops below target, and upgrades when modelMs shows consistent headroom (WebGPU only — WebGL upgrades are too expensive to swap live). The target is the camera’s measured frame rate, capped at 30 fps, so a 15 fps camera isn’t mistaken for an overloaded GPU.

onReady (info: ReadyInfo) => void

Fires once, when the network is attached and the effect is live (the next frame through the pipe is the first with the effect). With no usable GPU it fires once raw passthrough has settled, with info.backend === 'none'. Same moment pipeline.ready resolves, and with the same ReadyInfo:

interface ReadyInfo {
  backend: 'webgpu' | 'webgl2' | 'none'
  dtype:   'f16' | 'f32' | null
  preset:  string | null      // resolved model, e.g. 'small'; null with no backend or enabled: false
}
onError (err: PipelineError) => void

Fires for every async / runtime failure, as a typed PipelineError with a type and a fatal flag. That covers no GPU, weight 404, bad backgrounds, GPU device or context loss, worker crashes and hangs, the camera ending, failed or timed-out control calls (including setBackground()), adaptive swap failures and audio problems. The output track keeps carrying video through all of them (the raw camera when the effect can’t run), except 'camera-ended'. Synchronous construction errors (no input video track, transport setup throws) propagate out of new EffectsPipeline(...) directly — wrap construction in try/catch if you want one handler for both.

onRecovered (info: RecoveredInfo) => void

Fires after the SDK has recovered automatically from a GPU device/context loss or a worker crash: the worker was restarted, the model reloaded, your settings re-applied, and the effect is live again. info is { reason: 'backend-lost' | 'worker-crash', backend, attempts }. Until then the output carried the raw camera.

onTrackChanged (change: TrackChange) => void

Fires when the output video track had to be replaced. This only happens on the transfer-capture path (Safari), after a worker crash or GPU loss: that path’s canvas belongs to the worker and can’t be taken back. pipeline.stream is already updated in place. Call sender.replaceTrack(change.videoTrack) on your RTCRtpSender. change is { stream, videoTrack, previousTrack, reason }. Never fires on Chromium or Firefox, where the raw fallback stays on the original track. See Browser support.

onAudioTrackChanged (change: AudioTrackChange) => void

Fires when the output audio track changes: if the denoise AudioContext can’t run (autoplay policy, no audio output device), the output switches to your original mic audio, and back to the denoised track once the context runs. pipeline.stream is already updated; call sender.replaceTrack(change.audioTrack). change is { audioTrack, previousTrack, denoised }.

Properties

Methods

All control methods below return a Promise and follow the same rules (see Control calls): calls made before ready queue; a failure rejects that call with a typed LongpipeError, also goes to onError, and leaves the previous settings running.

Preview

Show a user what a candidate effect would look like — e.g. while they browse a background menu — without changing the effect on the outgoing stream. The main output keeps its currently-applied background; a second canvas renders the candidate. The network and matte are computed once and shared between the two, so previewing adds only a second composite (not a second inference).

// 1. Attach a canvas you own. Called ONCE — control is transferred to the
//    worker, so the element can't be drawn to by the page afterward. The
//    worker sizes the backing to the output resolution; CSS-scale it for display.
pipeline.attachPreview(document.querySelector('#preview'))

// 2. Preview a candidate. `background` is the identical wide surface as the
//    main effect (keyword, URL, ImageBitmap, { color }, { blur }, …). 'none'
//    previews the no-effect (raw) option. fps throttles the preview (default 15;
//    the main output stays the priority).
await pipeline.setPreview({ background: 'https://example.com/beach.jpg', fps: 15 })

// 3. Stop previewing (the canvas freezes on its last frame — hide it in your UI).
pipeline.clearPreview()

To apply a previewed candidate to the outgoing stream, just call pipeline.setBackground(sameInput) — there’s no separate “commit” call.

The preview canvas is local display only; it is not a MediaStream and is not transmitted. On WebGPU it composites directly to your canvas; on WebGL2 it’s presented via a per-frame snapshot of the render canvas (a little more expensive, which is why preview is throttled).

Error handling

The policy is fail open: the output video track always carries something (the effect if possible, otherwise the raw camera frames), and every failure reaches onError as a typed error. Longpipe never causes black video or silent audio.

interface PipelineError {
  type:        PipelineErrorType  // what happened — branch on this
  fatal:       boolean            // true → effects won't come back without a new pipeline
  message:     string
  source:      'worker' | 'backend-lost' | 'adaptive' | 'background' | 'audio' | 'camera' | 'control' | 'unknown'
  recoverable: boolean            // deprecated: same as !fatal
  cause?:      unknown
}

Rejected control calls throw a LongpipeError, exported from longpipe. It’s an Error subclass with the same fields, so err instanceof LongpipeError && err.type === 'background-load' works.

typeFatalWhat happenedOutput video
'no-backend'noNo usable GPU (no WebGPU, no WebGL2) at startup. ready resolves with backend: 'none'.Raw camera
'init-failed'yesWeights download, autotune or worker init failed. ready rejects.Unprocessed frames
'background-load'noA background image/video failed to load or decode (or timed out after 15 s). The previous background stays ('none' at startup).Unchanged
'backend-lost'noWebGPU device lost / WebGL context lost. The SDK restarts the worker with a fresh backend (WebGL2 after a second WebGPU loss), then fires onRecovered.Raw camera until recovered
'worker-crash'noThe worker errored, stopped answering (no heartbeat for 5 s) or stalled. Restarted as above.Raw camera until recovered
'recovery-failed'yesAutomatic recovery gave up (3 restarts within a minute, or no GPU any more).Raw camera
'camera-ended'yesThe input video track ended (unplugged, permission revoked, stop()).Ended
'control-failed'noA control call failed. That call rejects; previous settings stay.Unchanged
'control-timeout'noA control call got no answer in time (10 s; 30 s if queued before ready).Unchanged
'adaptive'noAn adaptive preset swap failed; stays on the current preset.Unchanged
'audio'noAudio denoise failed. The original audio passes through.Unaffected
'audio-suspended'noThe denoise AudioContext is suspended (autoplay policy, or no audio output device). Call EffectsPipeline.unlockAudio() in your click before any await; the SDK also resumes on the next gesture.Audio: original mic until it resumes (onAudioTrackChanged)

Only a fatal error before ready settles rejects ready. Nothing after that, including a failed control call, touches it.

onError does not fire for synchronous constructor errors — those propagate out of new EffectsPipeline(...) directly. Wrap in try/catch if you want a single handler for both.

Control calls

setBackground, setEnabled, setPreset, setTouchup, setReframe, reframe, attachPreview, setPreview and clearPreview all behave the same way:

Rejections are pre-handled, so ignoring a returned promise never produces an “Uncaught (in promise)” warning; the error still reaches onError.

See Browser support & reliability for per-browser behaviour, including when the output track can change (onTrackChanged, Safari only).