Video Effects SDK 3.12.0 for Web

Models Updated#

• New models have been introduced for the Balanced Landscape and Quality Landscape presets.
• The new models provide better segmentation quality while using fewer system resources.

ONNX Runtime Upgrade, Pipeline Stability, and Better Error Reporting#

This release upgrades the inference runtime to ONNX Runtime 1.27, fixes multiple scenarios that could silently stop the video pipeline, and continues the migration to the new error reporting model.

If you host the SDK files yourself or configure wasmPaths, note that the runtime and model files have changed - see Migration.

ONNX Runtime 1.27#

• Updated the bundled ONNX Runtime to version 1.27.
• The runtime now ships as a single WASM file - ort-wasm-simd-threaded.asyncify.wasm. By default it is loaded from the same directory where the SDK script is placed; a custom URL can be provided via the SDK config (see Migration).
• All model files have been updated and now use new versioned filenames. Self-hosted integrations need to host the updated model files (see Migration).
• Inference for both the wasm and webgpu providers now runs explicitly in a dedicated SDK worker. Previously, this processing was handled internally by ONNX Runtime, which also executed it in a separate worker thread. If the worker cannot be created, the SDK falls back to inference on the main thread and reports it as a warning. The worker is used unless it is turned off in the config:

sdk.config({
  proxy: false // run inference on the main thread; the worker is used by default
});

Migration#

This section applies if you host the SDK files yourself, or if you configure wasmPaths explicitly.

WASM runtime

Previous versions loaded several ONNX Runtime files. Now there is a single one: ort-wasm-simd-threaded.asyncify.wasm. Host it in the same directory as the SDK script and it is picked up automatically, or pass an explicit URL:

sdk.config({
  wasmPaths: {
    wasm: 'https://domain.com/sdk/ort-wasm-simd-threaded.asyncify.wasm'
  }
});

The shape of wasmPaths changed with it. The previous per-file map is no longer recognized:

// No longer works - the setting is ignored and the default location is used instead.
wasmPaths: {
  'ort-wasm.wasm': '...',
  'ort-wasm-simd.wasm': '...'
}

Smaller WASM File

• The new WASM file is significantly smaller than the previous version.
• If you host the WASM file yourself, we recommend enabling Brotli compression on your web server.
• With Brotli enabled, the transferred file size is less than 2 MB.

The NPM package does not include the WASM file. In this repository, the file is available in ./sdk/, so self-hosted integrations should copy it from there into their static assets. Note that the "same directory as the SDK script" default does not apply to bundled ESM builds: without wasmPaths the file is loaded from the Effects SDK CDN.

Models

The model files have been updated and now use new versioned filenames. The old model versions are no longer requested by the SDK.

The updated model files are also included in this repository under: ./sdk/models/

If you host the models yourself, replace the existing files with the new set in the models subfolder of your sdk_url:

• model-1.2.2.tsvb — default segmentation model
• model-1.3.2.tsvb, model-5.5.2.tsvb, model-3.1.2.tsvb, model-4.0.2.tsvb — quality, balanced, speed, and lightning presets
• m-model-1.0.2.tsvb, m-model-5.5.2.tsvb, m-model-3.1.2.tsvb, m-model-4.0.2.tsvb — portrait presets
• model-fd-1.1.2.tsvb — face detector

sdk.config({
  sdk_url: 'https://domain.com/sdk/' // models are expected in https://domain.com/sdk/models/
});

Bug Fixes#

• Fixed multiple scenarios where an internal error could silently stop the video pipeline. Such errors are now reported through onError while the pipeline keeps running.
• Fixed a ReferenceError on browsers without the VideoFrame API (Firefox before 130, Safari 16.3 and older).
• Stopping or restarting the pipeline no longer stops the audio tracks of the caller's input stream.
• On some browsers (e.g. Safari, Firefox) the stream returned by getStream() had a frame rate ceiling that ignored setFpsLimit(), so a higher limit never reached the output. The ceiling now follows the limit set before the first getStream() call, and is never lower than it used to be.
• Fixed the "Critical dependency: the request of a dependency is an expression" warning from consumer bundlers: the webpackIgnore and @vite-ignore pragmas on the dynamic import() calls (used by the Avatars Effect to load its modules at runtime) are now preserved in the shipped JS files.
• The SDK now falls back to the global timers when its timer worker cannot be created (usually a Content-Security-Policy blocking blob: workers) instead of failing. The fallback is reported once as a TIMERS_FALLBACK warning.
• A throwing onError subscriber no longer breaks the SDK call that reported the error and no longer prevents the remaining subscribers from being notified.

Error Reporting#

The SDK continues migrating its public API from ErrorBus notifications to thrown SdkError exceptions. The ErrorBus (onError) remains the channel for errors that happen inside the frame pipeline, where there is no caller to throw to. The codes and events below are described in more detail in the documentation.

New error codes:

• VIDEO_PLAYBACK_ISSUE = 1031 - the input video element failed to start playback.
• TIMERS_FALLBACK = 1065 - the timer worker is unavailable and the SDK fell back to the global timers, which the browser throttles in a hidden tab.

Methods that previously did not throw and now can:

• setFpsLimit() - throws SdkError.
• getStream() - on some browsers (e.g. Safari, Firefox) throws SdkError with code RENDERING_ISSUE when the output stream cannot be captured from the canvas.
• applyConstraints() on the output video track - now returns the real result of the underlying track; previously it always resolved successfully and dropped failures.
• getSnapshot() - now rejects with the underlying error; previously it could stay pending forever after a failure.