Table of Contents

Class SpeakerDiarizationSettings

Namespace
VisioForge.Core.Types.X.AI
Assembly
VisioForge.Core.dll

Settings for the speaker-diarization block ("who spoke when"): a pyannote segmentation model finds who is talking inside each 10 s window of audio, a WeSpeaker/3D-Speaker embedding model turns every such turn into a voiceprint, and the voiceprints are clustered into speakers once the recording has ended.

public class SpeakerDiarizationSettings

Inheritance

Inherited Members

Remarks

Diarization is OFFLINE. A speaker's identity cannot be settled while the audio is still arriving: the segmentation model numbers speakers only WITHIN one window (window 5's "speaker 1" and window 6's "speaker 1" are unrelated), and the only thing that ties a voice at minute 1 to the same voice at minute 40 is clustering every voiceprint of the recording together. So the block analyses continuously as audio flows, but publishes its timeline at END-OF-STREAM. It suits files and finite streams; an endless live source never reaches the point where an answer exists.

Two ONNX models are required, both supplied by the application (they are never shipped in the SDK NuGet packages):

  • A pyannote segmentation-3.0 model (VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.SegmentationModelPath) — MIT-licensed, exported by the sherpa-onnx project. It takes a fixed 10 s window of mono 16 kHz audio and returns per-frame powerset speaker activity for up to three local speakers.
  • A speaker-embedding model (VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.EmbeddingModelPath) — for example the Apache-2.0 WeSpeaker English voxceleb_resnet34_LM model. It consumes 80-bin kaldi fbank features and returns a fixed-length speaker embedding.

These settings do not derive from VisioForge.Core.Types.X.AI.OnnxInferenceSettings: those settings describe video-frame preprocessing (letterbox resize, NCHW), which does not apply to an audio model pair.

Constructors

SpeakerDiarizationSettings()

Initializes a new instance of the VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings class.

public SpeakerDiarizationSettings()

SpeakerDiarizationSettings(string, string)

Initializes a new instance of the VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings class with the two model paths.

public SpeakerDiarizationSettings(string segmentationModelPath, string embeddingModelPath)

Parameters

segmentationModelPath string

The absolute path to the pyannote segmentation ONNX model.

embeddingModelPath string

The absolute path to the speaker-embedding ONNX model.

Fields

ERes2NetV2MultilingualModelUrl

The download URL for the multilingual 3D-Speaker ERes2NetV2 sv_zh-cn_16k-common embedding model (Apache-2.0). Use this for non-English or mixed-language content.

public const string ERes2NetV2MultilingualModelUrl = "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/3dspeaker_speech_eres2netv2_sv_zh-cn_16k-common.onnx"

Field Value

string

WeSpeakerEnglishModelUrl

The download URL for the default WeSpeaker English voxceleb_resnet34_LM embedding model (Apache-2.0).

public const string WeSpeakerEnglishModelUrl = "https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/wespeaker_en_voxceleb_resnet34_LM.onnx"

Field Value

string

Properties

ClusterThreshold

Gets or sets the cosine-DISTANCE threshold at which two voiceprints stop being the same person. Defaults to 0.5.

Note the direction: this is a DISTANCE (0 = identical voices, 2 = opposite), not a similarity. A SMALLER value is stricter and yields MORE speakers; a larger one merges more readily. Ignored when VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.NumSpeakers is positive. 

public float ClusterThreshold { get; set; }

Property Value

float

DeviceId

Gets or sets the hardware device id used when a GPU provider is selected. Defaults to 0.

public int DeviceId { get; set; }

Property Value

int

EmbeddingModel

Gets or sets which speaker-embedding model the settings are configured for. This is a declarative hint used to pick sensible defaults — the ONNX file actually loaded is always VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.EmbeddingModelPath, and the fbank feature front-end configures itself (sample scaling, feature normalization, feature dimension) from that file's ONNX metadata regardless of this value. Use VisioForge.Core.Types.X.AI.SpeakerEmbeddingModel.ERes2NetV2Multilingual for non-English content. Defaults to VisioForge.Core.Types.X.AI.SpeakerEmbeddingModel.WeSpeakerEnglish.

public SpeakerEmbeddingModel EmbeddingModel { get; set; }

Property Value

SpeakerEmbeddingModel

EmbeddingModelPath

Gets or sets the absolute path to the speaker-embedding ONNX model file (for example the WeSpeaker voxceleb_resnet34_LM model). Required.

public string EmbeddingModelPath { get; set; }

Property Value

string

EmbeddingModelUrl

Gets or sets the URL an application may use to download the speaker-embedding model. Informational only. Defaults to the sherpa-onnx WeSpeaker English ResNet34 release; see VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.ERes2NetV2MultilingualModelUrl for the multilingual model.

public string EmbeddingModelUrl { get; set; }

Property Value

string

MinDurationOff

Gets or sets the longest pause, in seconds, that still counts as the SAME turn. Two consecutive turns of one speaker separated by less than this are joined, so a speaker pausing for breath mid-sentence stays one turn rather than becoming two. Defaults to 0.5 s.

public double MinDurationOff { get; set; }

Property Value

double

MinDurationOn

Gets or sets the shortest turn, in seconds, that is reported. Anything shorter is dropped as noise — without this the timeline fills with fragments of a neighbour's breath. Defaults to 0.3 s.

public double MinDurationOn { get; set; }

Property Value

double

NumSpeakers

Gets or sets the exact number of speakers in the recording, when it is known. A positive value pins the answer to that many speakers and VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.ClusterThreshold is then ignored. Defaults to -1, meaning "work it out from the voices" — which is the right setting whenever the count is not certain, and the only honest one for arbitrary content.

public int NumSpeakers { get; set; }

Property Value

int

Provider

Gets or sets the execution provider used by both ONNX sessions. Defaults to VisioForge.Core.Types.X.AI.OnnxExecutionProvider.Auto, which uses the fastest hardware provider present in the loaded ONNX Runtime native build and falls back to the CPU when none is available.

public OnnxExecutionProvider Provider { get; set; }

Property Value

OnnxExecutionProvider

SegmentationModelPath

Gets or sets the absolute path to the pyannote segmentation-3.0 ONNX model file. Required.

public string SegmentationModelPath { get; set; }

Property Value

string

SegmentationModelUrl

Gets or sets the URL an application may use to download the segmentation model. Informational only — the SDK never downloads anything on its own; the file actually loaded is VisioForge.Core.Types.X.AI.SpeakerDiarizationSettings.SegmentationModelPath. Defaults to the sherpa-onnx pyannote release.

public string SegmentationModelUrl { get; set; }

Property Value

string