Table of Contents

Class VideoEmbeddingBlock

Namespace
VisioForge.Core.MediaBlocks.AI
Assembly
VisioForge.Core.AI.dll

A media block that encodes sampled video frames into CLIP image embeddings for semantic video search. Implements the VisioForge.Core.MediaBlocks.MediaBlock. Implements the VisioForge.Core.MediaBlocks.IMediaBlockInternals.

public class VideoEmbeddingBlock : MediaBlock, IMediaBlockInternals, IVideoProcessingBlock, IMediaBlock, IDisposable

Inheritance

MediaBlock

Implements

IMediaBlockInternals
IVideoProcessingBlock
IMediaBlock

Inherited Members

MediaBlock._isBuilt
MediaBlock._pipeline
MediaBlock._pipelineCtx
MediaBlock.GetPipelineContext()
MediaBlock.SetPipelineContext(BlockPipelineContext)
MediaBlock.SetPipeline(MediaBlocksPipeline)
MediaBlock.Context
MediaBlock.Name
MediaBlock.IsBuilt
MediaBlock.Owner
MediaBlock.Type
MediaBlock.ID
MediaBlock.Input
MediaBlock.Inputs
MediaBlock.Output
MediaBlock.Outputs
MediaBlock.HasInputs
MediaBlock.HasOutputs
MediaBlock.Build()
MediaBlock.CreateElements()
MediaBlock.AddElementsToPipeline()
MediaBlock.RemoveElementsFromPipeline()
MediaBlock.DeepCopy(string)
MediaBlock.Reset()
MediaBlock.ToYAMLBlock()
MediaBlock.ClearPads()
MediaBlock.Dispose(bool)
MediaBlock.Dispose()

Remarks

Taps RGBA frames via an internal sample grabber and, no more often than VisioForge.Core.Types.X.AI.VideoEmbeddingSettings.SampleInterval, encodes each sampled frame into an L2-normalized CLIP embedding on a background worker (thread name "ClipWorker"), so the streaming thread is never blocked. Each embedding is raised via VisioForge.Core.MediaBlocks.AI.VideoEmbeddingBlock.OnFrameEmbedding and, when VisioForge.Core.Types.X.AI.VideoEmbeddingSettings.Index is set, added to that index under VisioForge.Core.Types.X.AI.VideoEmbeddingSettings.SourceTag. The passthrough leaves the video unchanged. Use VisioForge.Core.MediaBlocks.AI.VideoEmbeddingBlock.EncodeText(System.String) (valid after the block is built) to turn a natural-language query into an embedding in the same space for searching the index.

Constructors

VideoEmbeddingBlock(VideoEmbeddingSettings)

Initializes a new instance of the VisioForge.Core.MediaBlocks.AI.VideoEmbeddingBlock class.

public VideoEmbeddingBlock(VideoEmbeddingSettings settings)

Parameters

settings VideoEmbeddingSettings

The video embedding settings. Must specify a valid CLIP vision model path.

Exceptions

ArgumentNullException

Thrown when settings is null.

Properties

ActiveProvider

Gets the execution provider the CLIP sessions actually engaged. Valid after the block is built; reports VisioForge.Core.Types.X.AI.OnnxExecutionProvider.CPU otherwise.

public OnnxExecutionProvider ActiveProvider { get; }

Property Value

OnnxExecutionProvider

Dimension

Gets the CLIP embedding dimension, read from the model output metadata. Zero before the block is built.

public int Dimension { get; }

Property Value

int

DroppedFrameCount

Gets the number of frames the analysis had to discard. Stays 0 under the sample-latest design: the video always passes through untouched and the worker embeds the freshest frame once free, so nothing is dropped.

public long DroppedFrameCount { get; }

Property Value

long

Input

Gets the primary input pad.

public override MediaBlockPad Input { get; }

Property Value

MediaBlockPad

Inputs

Gets the array of all input pads.

public override MediaBlockPad[] Inputs { get; }

Property Value

MediaBlockPad[]

LastInferenceTimeMs

Gets the wall-clock time, in milliseconds, the most recent frame embedding took.

public float LastInferenceTimeMs { get; }

Property Value

float

Output

Gets the primary output pad.

public override MediaBlockPad Output { get; }

Property Value

MediaBlockPad

Outputs

Gets the array of all output pads.

public override MediaBlockPad[] Outputs { get; }

Property Value

MediaBlockPad[]

Type

Gets the type of the media block.

public override MediaBlockType Type { get; }

Property Value

MediaBlockType

Methods

Build()

Constructs the internal GStreamer elements and pads for this block.

public override bool Build()

Returns

bool

true if successful, false otherwise.

CleanUp()

Cleans up internal resources, specifically the GStreamer element.

public void CleanUp()

Dispose(bool)

Releases unmanaged and - optionally - managed resources.

protected override void Dispose(bool disposing)

Parameters

disposing bool

true to release both managed and unmanaged resources; false to release only unmanaged resources.

EncodeText(string)

Encodes a natural-language query into an L2-normalized CLIP text embedding in the same space as the frame embeddings, for searching a VisioForge.Core.Types.X.AI.FrameEmbeddingIndex.

public float[] EncodeText(string query)

Parameters

query string

The query text.

Returns

float[]

The L2-normalized query embedding.

Exceptions

InvalidOperationException

Thrown when the block has not been built yet.

GetCore()

Gets the core GStreamer element wrapped by this block.

public BaseElement GetCore()

Returns

BaseElement

The VisioForge.Core.GStreamer.Base.BaseElement wrapper, or null if not built yet.

GetElement()

Gets the GStreamer element instance.

public Element GetElement()

Returns

Element

The Gst.Element, or null if not built yet.

IMediaBlockInternals.SetContext(MediaBlocksPipeline)

Sets the pipeline context for this block.

void IMediaBlockInternals.SetContext(MediaBlocksPipeline pipeline)

Parameters

pipeline MediaBlocksPipeline

The pipeline.

OnFrameEmbedding

Event raised for each sampled frame with its L2-normalized CLIP embedding.

public event EventHandler<FrameEmbeddingEventArgs> OnFrameEmbedding

Event Type

EventHandler<FrameEmbeddingEventArgs>

See Also

MediaBlock
IMediaBlockInternals