Video Effects SDK 2.6.0.118490 for Windows & macOS

Video Effects SDK 2.6#

Release 2.6.0 for Windows and macOS. Versions 2.4 and 2.5 were iOS-only releases of the Swift package.

IPipeline and IReplacementController are thread-safe#

Any method may be called from any thread without synchronization, except that IPipeline::process must not be called while another process() call is running. Loading a model or creating the engine no longer holds up process(). The same holds for the Obj-C/Swift Pipeline: only one of its process methods may run at a time, and ReplacementController.background may be set from any thread. Enabling an effect may take a while (it may load a model), so avoid calling enable* methods on the UI thread. Parameter setters such as setColorCorrectionPower are cheap.

Effects#

• New feature Noise Suppression: Improves video quality by reducing grainy noise in the video and keeps noisy regions stable instead of letting them shimmer from frame to frame. See IPipeline::enableNoiseSuppression.
• New feature Portrait Lighting: Highlights the person in the video, with separate controls of the light strength and of the background darkening. See IPipeline::enablePortraitLighting, setPortraitLightingParam/getPortraitLightingParam with PortraitLightingParam.
• Reworked Background Blur: faster and aligned with the Web SDK.
• Reworked Beautification.
• Reworked Color Correction for more natural results: automatic scene analysis adapts the image to the lighting and color balance, and manual controls of white balance, exposure, temperature, tint, vibrance and contrast fine-tune it. See IPipeline::setColorCorrectionParam/getColorCorrectionParam with ColorCorrectionParam.
• Reworked Low Light: brightens dark scenes without an ML model.

Changes#

• New segmentation models, better and faster: the Quality preset is about 1.75x faster (3.0 ms to 1.7 ms per frame on our test CPU).
• DirectX: NV12 conversion is performed on GPU now. process() of an NV12 frame on our test discrete GPU (with background replacement enabled): 1920x1080 3.4 ms instead of 15.7 ms, 3840x2160 5.0 ms instead of 54 ms, 7680x4320 12.5 ms instead of 230 ms.
• Obj-C/Swift: Added the ability to create Frame with CVPixelBuffer, see FrameFactory.newFrame(pixelBuffer:metalCompatible:).
• Windows: Improved performance of IReplacementController::setBackgroundImage: the background is uploaded to GPU without a copy on CPU, NV12 backgrounds are converted on GPU.
• Apple: Improved performance of ReplacementController.background: a CVPixelBuffer background is used by Metal directly, without a copy, when possible; other frames are converted once per change.
• Apple: The Metal pipeline supports RGBA frames created from memory; the processed frame is in memory too, its toCVPixelBuffer returns nil.
• Added Color Range and Color Matrix (BT.601, BT.709) properties in IFrame: IFrame::setColorRange with ColorRange, IFrame::setColorMatrix with ColorMatrix. On Apple they are taken from the CVPixelBuffer.
• Apple: Added MLBackend::mlBackendGPU, enabled by default: Core ML may run segmentation on the GPU too. To keep the CPU and the Neural Engine only, pass mlBackendCPU | mlBackendAppleNeuralEngine to IPipelineConfiguration::setSegmentationMLBackends (Obj-C/Swift: isSegmentationOnGPUEnabled false).
• Frame memory is re-used now: output frames of IPipeline::process() and the copies that IFrameFactory::createBGRA/createNV12 make when makeCopy is true come from a pool (on Apple, a CVPixelBufferPool).
• New error codes in PipelineErrorCode: notSupportedByBackend, enabledUnsupportedFeatures and invalidUsage.
• Windows and macOS: The SDK can be linked directly: sdk_factory.h declares createSDKFactory(), tsvb.lib is shipped. Loading the library at runtime still works.
• Obj-C/Swift: Added enableNoiseSuppression, enablePortraitLighting, effect settings as typed properties colorCorrection (ColorCorrectionSettings) and portraitLighting (PortraitLightingSettings), and isSegmentationOnGPUEnabled.
• Obj-C/Swift: Added read-only isBlurBackgroundEnabled, blurBackgroundPower and isReplaceBackgroundEnabled.
• The API reference is available at https://effectssdk.ai/sdk/desktop/docs/ (C++) and https://effectssdk.ai/sdk/ios/documentation/tsvb/ (Obj-C/Swift).
• Apple: Removed excess dependencies and improved model packing: size of the binaries drastically reduced. macOS libtsvb.dylib: arm64 14.0 MB to 2.05 MB, x86_64 17.2 MB to 2.08 MB.

Breaking changes#

• Misspelled enableColorCorrectionWithReferance deprecated. Use enableColorCorrectionWithReference.
• Removed the segmentationPresetSpeed and segmentationPresetLightning values of SegmentationPreset (Obj-C/Swift: speed and lightning of SegmentationPreset).
• Removed the old background denoise effect, replaced by Noise Suppression: enableDenoiseBackground, disableBackgroundDenoise, setDenoisePower/getDenoisePower, setDenoiseWithFace/getDenoiseWithFace and getDenoiseBackgroundState of IPipeline (Obj-C/Swift: enableDenoiseBackground, disableDenoiseBackground, denoisePower and denoiseWithFace).
• Beautification, Color Correction, Low Light do not work on CPU anymore: enabling them on the CPU backend returns notSupportedByBackend. Code that checked for engineInitializationError there has to check for notSupportedByBackend now.
• macOS x86_64: The minimum macOS version is 11.0 instead of 10.15, as for arm64.
• PipelineErrorCode::invalidArguemnt is renamed to invalidArgument.
• IReplacementController::setBackgroundImage (Obj-C/Swift background) does not copy the frame anymore: don't release a frame created with makeCopy false while it is the background.
• Windows: With background replacement and no background image, the background of an NV12 frame is black, as on Apple.

Fixes#

• IFrameFactory::createNV12 with makeCopy read past the end of the UV plane and could crash.
• Windows: tsvb.dll is unloaded by FreeLibrary; it used to stay loaded with idle threads until the process exited.
• Windows x86: Smart Zoom works again; enableSmartZoom failed with engineInitializationError.
• Windows and CPU backend: The processed frame keeps the rotation of the input frame.
• Apple: Fixed a socket leak on each authorization request.
• Color Filter and Color Grading: wrong colors for a few frames after changing the LUT or switching from CPU to GPU backend.
• IFrame::lock() on the background image during process() could get a cleared data pointer.
• IFrameFactory::loadImage and loadImageWithData return nullptr for a null argument instead of crashing.
• Swift: await factory.auth(customerID:) never returned when authorization failed before the web request.
• Obj-C: Fixed memory leaks in newPipeline, failed authorization and enableReplaceBackground:.
• Obj-C: colorCorrectionPower returns the actual value.
• Obj-C: newFrameWithFormat: with NV12 returns nil instead of reading invalid memory; pass NV12 as CVPixelBuffer.

Sample#

The sample is reworked: it uses Qt QML now, with an updated design.

• Camera capture is reworked on native APIs: Media Foundation on Windows, AVFoundation on macOS.
• The SDK usage is reworked to demonstrate a more efficient path: camera frames go to the SDK and to the screen without conversions and copies on CPU (NV12 on Windows, CVPixelBuffer on macOS).
• Clearer error messages.