Video Effects SDK 2.6 for Windows and macOS: thread-safe pipeline, new effects, 4–18× faster NV12

Video Effects SDK 2.6 is a major desktop release. The C++ SDK for Windows and macOS (and the Objective-C/Swift API it shares with iOS) gets two new effects, reworked blur, beautification, color correction and low light, a thread-safe pipeline, new segmentation models and GPU-side NV12 conversion that makes 4K processing about 10× faster. Versions 2.4 and 2.5 were iOS-only, so this is the first feature release for desktop since 2.3.

Full release notes · Download from GitHub · C++ API reference · Swift API reference

Thread-safe pipeline#

IPipeline and IReplacementController can now be called from any thread without your own locks. The only rule left: do not call IPipeline::process() while another process() call is running. Loading a model or creating the engine no longer blocks process(), so turning an effect on does not stall the video.

The same applies to the Swift Pipeline: one process call at a time, and ReplacementController.background can be set from any thread. Enabling an effect may load a model, so call enable* methods off the UI thread; parameter setters such as setColorCorrectionPower are cheap.

New and reworked effects#

• Noise Suppression (new) removes grain from the camera image and keeps noisy areas stable instead of shimmering from frame to frame. It replaces the old background denoise effect.
• Portrait Lighting (new) lights up the person with separate controls for the light strength and for darkening the background.
• Color Correction (reworked) analyzes the scene automatically and adds manual white balance, exposure, temperature, tint, vibrance and contrast controls.
• Background Blur is faster and now matches the Web SDK.
• Beautification is reworked.
• Low Light brightens dark scenes without an ML model.

Faster#

Other performance changes:

• On DirectX, NV12 is converted on the GPU, and background images are uploaded to the GPU without a CPU copy.
• On Apple platforms a CVPixelBuffer background is used by Metal directly, and Core ML may now run segmentation on the GPU as well (mlBackendGPU, on by default).
• Output frames and copied input frames come from a memory pool (CVPixelBufferPool on Apple).

New API#

• IFrame::setColorRange and setColorMatrix (BT.601, BT.709); on Apple they are read from the CVPixelBuffer.
• Swift: FrameFactory.newFrame(pixelBuffer:metalCompatible:), enableNoiseSuppression, enablePortraitLighting, typed colorCorrection and portraitLighting settings, isSegmentationOnGPUEnabled, and read-only blur and replacement state.
• New PipelineErrorCode values: notSupportedByBackend, enabledUnsupportedFeatures, invalidUsage.
• The SDK can now be linked directly: sdk_factory.h declares createSDKFactory() and tsvb.lib is shipped. Loading the DLL at runtime still works.

Breaking changes — check before upgrading#

• Beautification, Color Correction and Low Light no longer run on the CPU backend: enabling them there returns notSupportedByBackend (previously engineInitializationError).
• The speed and lightning segmentation presets are removed.
• The old background denoise API (enableDenoiseBackground, setDenoisePower and related) is removed — use Noise Suppression.
• setBackgroundImage / ReplacementController.background no longer copy the frame: do not release a frame created with makeCopy = false while it is the background.
• macOS x86_64 now needs macOS 11.0, the same as arm64.
• invalidArguemnt is renamed to invalidArgument; enableColorCorrectionWithReferance is deprecated in favor of enableColorCorrectionWithReference.
• On Windows, background replacement without an image now gives a black background for NV12 frames, as on Apple.

Fixes#

Among others: createNV12 with makeCopy could read past the UV plane; tsvb.dll now unloads with FreeLibrary; Smart Zoom works again on Windows x86; the CPU backend keeps the input frame rotation; a socket leak on authorization (Apple) and several Objective-C memory leaks are fixed; await factory.auth(customerID:) no longer hangs when authorization fails early.

New sample app#

The desktop sample is rebuilt with Qt QML. Camera capture uses Media Foundation on Windows and AVFoundation on macOS, and frames go to the SDK and to the screen without CPU conversions or copies (NV12 on Windows, CVPixelBuffer on macOS) — the path we recommend for production.

Getting started: Windows and macOS guide · Start a free trial