Video Effects SDK

Overview

Add real-time AI video enhancement to your application in a few hours and make the video meeting experience more effective and comfortable. This reference describes the C++ API of the SDK: see Features for what it can do and Usage details to get started.

Obtaining Effects SDK License

To receive a license, fill in the contact form.

Technical Details

• SDK available for macOS 11.0 and newer, on Intel and Apple silicon.
• SDK available for Windows 10/11 x64 platform.
• x86 applications running on x64 Windows are supported too.
• SDK for Linux is available on request.
• ML inference runs on the CPU, and on Apple platforms also on the GPU or the Apple Neural Engine through Core ML.

Features

• Virtual backgrounds - implemented
• Background blur - implemented
• Beautification/Touch up my appearance - implemented
• Auto framing/Smart Zoom - implemented
• Color correction, automatic and manual - implemented
• Color grading - implemented
• Color filtering with a lookup table - implemented
• Low light adjustment - implemented
• Sharpening - implemented
• Noise suppression - implemented
• Portrait lighting - implemented

Usage details

The entry point of the SDK is the instance of tsvb::ISDKFactory. Using an ISDKFactory instance you will be able to prepare frames for processing and configure the pipeline of processing (enable transparency, blur, replace background etc.).

How to obtain a tsvb::ISDKFactory instance

• On macOS link libtsvb.dylib and call tsvb::createSDKFactory(). Loading it at run time gains nothing there: the library is never unloaded, see Library lifetime.
• On Windows either link tsvb.lib, the import library of tsvb.dll, and call tsvb::createSDKFactory(), or load tsvb.dll at run time as below. Load it at run time when the application has to unload the library or to start without it.
• To load the SDK library at run time, tsvb.dll or libtsvb.so, use LoadLibrary() or dlopen. Get the address of createSDKFactory() function from it, cast it to tsvb::pfnCreateSDKFactory type and call it to instantiate a tsvb::ISDKFactory object.
• Authorize the instance: call tsvb::ISDKFactory::auth() or tsvb::ISDKFactory::authWithKey().

::tsvb::ISDKFactory* createFactory()
{
    HMODULE hModule = ::LoadLibraryA("tsvb");
    auto createSDKFactory = reinterpret_cast<::tsvb::pfnCreateSDKFactory>(
        GetProcAddress(hModule, "createSDKFactory")
    );
    return createSDKFactory();
}

On macOS, linked with libtsvb.dylib, and on Windows, linked with tsvb.lib:

::tsvb::ISDKFactory* sdkFactory = ::tsvb::createSDKFactory();

See sdk_library_handler_win.cpp and sdk_library_handler_unix.cpp in the sample for the complete code.

Library lifetime

On Windows and Linux a library loaded at run time is unloaded once its last handle is freed with FreeLibrary() or dlclose(). Free it only after every object obtained from it is released. Linked with tsvb.lib, tsvb.dll stays loaded until the process exits.

On macOS dlclose() leaves libtsvb.dylib loaded: the system never unloads a library that contains Objective-C code, and the SDK uses it for Metal and Core ML. A later dlopen() in the same process returns the same, already initialized library. Memory, threads and models held by the SDK are freed when its objects are released, not when the library is closed.

Memory management

All objects created by the SDK implement tsvb::IRelease. The IRelease interface provides a tsvb::IRelease::release() method which releases the memory that was allocated for the object. The release() method should be called explicitly when such an object must be destroyed.

::tsvb::ISDKFactory* sdkFactory = createFactory();
//some code
sdkFactory->release();

Don’t use operator delete for the SDK objects.

To have std::unique_ptr or std::shared_ptr release them, see Smart pointers.

Library Usage

Preparation:

• Authorize the tsvb::ISDKFactory instance with tsvb::ISDKFactory::auth() or tsvb::ISDKFactory::authWithKey().
• Create an instance of tsvb::IFrameFactory.
• Create an instance of tsvb::IPipeline.
• Enable background blur using tsvb::IPipeline::enableBlurBackground() or background replacement using tsvb::IPipeline::enableReplaceBackground().
• With background replacement enabled the background is transparent; to show an image there, pass it to tsvb::IReplacementController::setBackgroundImage().

Frame processing:

• Put your frame to tsvb::IFrame using tsvb::IFrameFactory::createBGRA() or tsvb::IFrameFactory::createNV12().
• Process it through tsvb::IPipeline::process().

On Apple platforms you can use CVPixelBuffer for better performance. See tsvb::IAppleCompatFrameFactory.

Use separate tsvb::IPipeline instances per video stream.

tsvb::IPipeline and tsvb::IReplacementController are thread-safe: process frames on the video thread and turn effects on and off or change their settings from the GUI thread (or any other), with no lock of your own. tsvb::IPipeline::process() may be called from any thread too, but not while another tsvb::IPipeline::process() call is running: pass the frames of a stream one after another, in their order.

bool initialize(const char* customerID, const char* backgroundPath)
{
    ::tsvb::ISDKFactory* factory = createFactory();
    ::tsvb::IAuthResult* authResult = factory->auth(customerID, nullptr, nullptr);
    const ::tsvb::AuthStatus status = authResult->status();
    authResult->release();
    if (::tsvb::AuthStatus::active != status) {
        // Authorization failed: report the status, the SDK can't be used.
        factory->release();
        return false;
    }
 
    frameFactory = factory->createFrameFactory();
    pipeline = factory->createPipeline();
    factory->release();
 
    pipeline->enableNoiseSuppression();
    pipeline->enableReplaceBackground(&replacementController);
 
    ::tsvb::IFrame* background = frameFactory->loadImage(backgroundPath);
    if (nullptr != background) {
        replacementController->setBackgroundImage(background);
        background->release();
    }
    return true;
}
 
// For each frame of the stream:
::tsvb::IFrame* processFrame(void* bgra, unsigned int bytesPerLine, unsigned int width, unsigned int height)
{
    ::tsvb::IFrame* input = frameFactory->createBGRA(bgra, bytesPerLine, width, height, false);
    ::tsvb::IFrame* output = pipeline->process(input, nullptr);
    input->release();
    return output;
}
 
void release()
{
    frameFactory->release();
    frameFactory = nullptr;
    replacementController->release();
    replacementController = nullptr;
    pipeline->release();
    pipeline = nullptr;
}

For more usage details see sample_controller.cpp, which sets the effects up, and video_filter.cpp, which processes the frames, in the sample.

Links

• Our website: effectssdk.ai
• To receive a license, fill in the contact form: effectssdk.ai/contacts
• Check out other projects: github.com/EffectsSDK