Table of Contents

Class SpeakerDiarizationBlock

Namespace
VisioForge.Core.MediaBlocks.AI
Assembly
VisioForge.Core.AI.dll

A speaker-diarization ("who spoke when") media block. It taps the audio stream, resamples it to mono 16 kHz, and on a background worker runs a pyannote segmentation ONNX model over overlapping 10 s windows and turns every speech turn it finds into a voiceprint with a WeSpeaker/3D-Speaker embedding ONNX model. When the stream ends, it clusters the voiceprints into speakers and publishes the timeline. Implements the VisioForge.Core.MediaBlocks.MediaBlock. Implements the VisioForge.Core.MediaBlocks.IMediaBlockInternals.

public class SpeakerDiarizationBlock : MediaBlock, IMediaBlockInternals, IMediaBlockStoppable, IAudioProcessingBlock, IMediaBlock, IDisposable

Inheritance

MediaBlock

Implements

IMediaBlockInternals
IMediaBlockStoppable
IAudioProcessingBlock
IMediaBlock

Inherited Members

MediaBlock._isBuilt
MediaBlock._pipeline
MediaBlock._pipelineCtx
MediaBlock.GetPipelineContext()
MediaBlock.SetPipelineContext(BlockPipelineContext)
MediaBlock.SetPipeline(MediaBlocksPipeline)
MediaBlock.Context
MediaBlock.Name
MediaBlock.IsBuilt
MediaBlock.Owner
MediaBlock.Type
MediaBlock.ID
MediaBlock.Input
MediaBlock.Inputs
MediaBlock.Output
MediaBlock.Outputs
MediaBlock.HasInputs
MediaBlock.HasOutputs
MediaBlock.Build()
MediaBlock.CreateElements()
MediaBlock.AddElementsToPipeline()
MediaBlock.RemoveElementsFromPipeline()
MediaBlock.DeepCopy(string)
MediaBlock.Reset()
MediaBlock.ToYAMLBlock()
MediaBlock.ClearPads()
MediaBlock.Dispose(bool)
MediaBlock.Dispose()

Remarks

THE RESULT ARRIVES AT END-OF-STREAM, NOT DURING IT. This is not a design shortcut, it is what the problem allows. The segmentation model numbers speakers only WITHIN one 10 s window — window 5's "speaker 1" and window 6's "speaker 1" are unrelated — and the only thing that can tie a voice heard at minute 1 to the same voice at minute 40 is clustering every voiceprint of the recording together. So VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.OnSpeakerSegment fires for every turn once, in order, after the stream has ended. The block suits files and finite streams; an endless live source never reaches the moment when an answer exists.

The algorithm is a port of the reference sherpa-onnx offline diarization pipeline, which uses these exact two models. Its central idea, and the reason a window-at-a-time pipeline cannot match it: the windows OVERLAP heavily (10 s advancing 1 s, so every instant of audio is seen by about ten of them), and rather than trusting whichever window a moment fell into, all windows are laid on one global frame grid and every frame is decided by a VOTE among the windows covering it. A window that mislabels a moment is outvoted by its neighbours.

The block is audio-only and passes audio through unchanged (the tap forces a 32-bit float sample format but leaves rate/channels alone). Analysis runs on a dedicated worker thread reading an UNBOUNDED queue of the resampled PCM, so audio is never paced to the models and never discarded: a source that outruns the worker (a file decoded at full speed by an unsynced renderer) only grows the backlog — 64 KB per buffered second at 16 kHz mono. The answer to a machine permanently slower than its source is faster hardware or a GPU execution provider, never dropped audio. At end-of-stream the streaming thread waits for the worker to drain the backlog before teardown.

VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.OnSpeakerSegment is raised on the worker thread — marshal to the UI thread before touching UI. The whole timeline is also available from VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.GetTimeline. Check VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.IsTimelineComplete before trusting it: a pipeline stopped before end-of-stream never reaches the clustering step, so it has no timeline to give. Speaker ids identify people only within one run; they are not stable across runs.

Constructors

SpeakerDiarizationBlock(SpeakerDiarizationSettings)

Initializes a new instance of the VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock class.

public SpeakerDiarizationBlock(SpeakerDiarizationSettings settings)

Parameters

settings SpeakerDiarizationSettings

The diarization settings. Must specify valid segmentation and embedding model paths.

Exceptions

ArgumentNullException

Thrown when settings is null.

Properties

ActiveProvider

Gets the execution provider the segmentation session engaged. Valid from the moment the block is built, and it STAYS valid after the stream ends and the engines are torn down — which is when a caller normally logs it.

public OnnxExecutionProvider ActiveProvider { get; }

Property Value

OnnxExecutionProvider

Input

Gets the input pad.

public override MediaBlockPad Input { get; }

Property Value

MediaBlockPad

Inputs

Gets the array of all input pads.

public override MediaBlockPad[] Inputs { get; }

Property Value

MediaBlockPad[]

IsTimelineComplete

Gets a value indicating whether the run actually produced an answer: the stream reached end-of-stream, the worker drained its backlog, and the speakers were clustered and published.

Check this before reading VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.GetTimeline. When it is false — the pipeline was stopped before end-of-stream, or the worker faulted — the timeline is EMPTY, and that emptiness means "we never got to work out who spoke", NOT "nobody spoke". The two are not distinguishable from the timeline alone. 

public bool IsTimelineComplete { get; }

Property Value

bool

Output

Gets the output pad.

public override MediaBlockPad Output { get; }

Property Value

MediaBlockPad

Outputs

Gets the array of all output pads.

public override MediaBlockPad[] Outputs { get; }

Property Value

MediaBlockPad[]

SpeakerCount

Gets the number of distinct speakers found. It is 0 until the timeline is published at end-of-stream — the count is a RESULT of clustering every voiceprint of the recording, so there is no meaningful running value to report while the audio is still arriving.

public int SpeakerCount { get; }

Property Value

int

Type

Gets the type of the media block.

public override MediaBlockType Type { get; }

Property Value

MediaBlockType

UnattributedTurns

Gets the number of speech turns the embedding model failed on. Their audio WAS analysed, but they could not be voiceprinted, so they carry no speaker identity and are absent from the timeline. A timeline is fully trustworthy when VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.IsTimelineComplete is true AND this is 0.

public long UnattributedTurns { get; }

Property Value

long

Methods

Build()

Constructs the internal GStreamer elements and pads, initializes the two ONNX engines, and starts the analysis worker.

public override bool Build()

Returns

bool

true if successful, false otherwise.

CleanUp()

Cleans up internal resources when the pipeline stops. Build() recreates them on the next start.

public void CleanUp()

Dispose(bool)

Releases unmanaged and - optionally - managed resources.

protected override void Dispose(bool disposing)

Parameters

disposing bool

true to release both managed and unmanaged resources.

GetCore()

Gets the core GStreamer element wrapped by this block.

public BaseElement GetCore()

Returns

BaseElement

The VisioForge.Core.GStreamer.Base.BaseElement wrapper, or null if not built yet.

GetElement()

Gets the GStreamer element instance.

public Element GetElement()

Returns

Element

The Gst.Element, or null if not built yet.

GetTimeline()

Returns the diarized timeline — every speaker turn, ordered by start time.

It is EMPTY until the stream ends: the speakers are worked out in one pass over the whole recording, so there is nothing to return before that. It is also empty when the run never got there (a stop before end-of-stream, or a worker fault) — VisioForge.Core.MediaBlocks.AI.SpeakerDiarizationBlock.IsTimelineComplete is what tells the two apart from a recording in which nobody spoke. 

public IReadOnlyList<SpeakerSegment> GetTimeline()

Returns

IReadOnlyList<SpeakerSegment>

The list of speaker turns.

IsAvailable()

Determines whether this media block is available. Correct NuGet SDK redist should be included in your project.

public static bool IsAvailable()

Returns

bool

true if available; otherwise, false.

RequestStop()

Early stop signal from the pipeline: stops the worker so the streaming thread can leave the pad-probe.

public void RequestStop()

IMediaBlockInternals.SetContext(MediaBlocksPipeline)

Sets the pipeline context for this block.

void IMediaBlockInternals.SetContext(MediaBlocksPipeline pipeline)

Parameters

pipeline MediaBlocksPipeline

The pipeline.

OnSpeakerSegment

Occurs once per speaker turn, in start-time order, AFTER the stream has ended and the speakers have been worked out. Raised on the background worker thread — marshal to the UI thread before touching UI.

Nothing is raised while audio is still flowing, and that is not a limitation of this implementation: until the recording is over there is no such thing as "speaker 2" to report (see the class remarks). 

public event EventHandler<SpeakerSegmentEventArgs> OnSpeakerSegment

Event Type

EventHandler<SpeakerSegmentEventArgs>

See Also

MediaBlock
IMediaBlockInternals