Table of Contents

Class AudioEventDetectorBlock

Namespace
VisioForge.Core.MediaBlocks.AI
Assembly
VisioForge.Core.AI.dll

An audio-event-detection media block: it taps the audio stream, resamples it to 16 kHz mono, and runs a YAMNet (AudioSet, 521-class) ONNX classifier over short windows to recognize real-world sounds — siren, dog bark, glass break, alarm, music, speech and hundreds more — raising VisioForge.Core.MediaBlocks.AI.AudioEventDetectorBlock.OnAudioEvent as sounds are detected. Audio passes through unchanged.

public class AudioEventDetectorBlock : MediaBlock, IMediaBlockInternals, IMediaBlockStoppable, IAudioProcessingBlock, IMediaBlock, IDisposable

Inheritance

MediaBlock

Implements

IMediaBlockInternals
IMediaBlockStoppable
IAudioProcessingBlock
IMediaBlock

Inherited Members

MediaBlock._isBuilt
MediaBlock._pipeline
MediaBlock._pipelineCtx
MediaBlock.GetPipelineContext()
MediaBlock.SetPipelineContext(BlockPipelineContext)
MediaBlock.SetPipeline(MediaBlocksPipeline)
MediaBlock.Context
MediaBlock.Name
MediaBlock.IsBuilt
MediaBlock.Owner
MediaBlock.Type
MediaBlock.ID
MediaBlock.Input
MediaBlock.Inputs
MediaBlock.Output
MediaBlock.Outputs
MediaBlock.HasInputs
MediaBlock.HasOutputs
MediaBlock.Build()
MediaBlock.CreateElements()
MediaBlock.AddElementsToPipeline()
MediaBlock.RemoveElementsFromPipeline()
MediaBlock.DeepCopy(string)
MediaBlock.Reset()
MediaBlock.ToYAMLBlock()
MediaBlock.ClearPads()
MediaBlock.Dispose(bool)
MediaBlock.Dispose()

Remarks

Unlike the speech-to-text block (which runs inference synchronously and paces the source), this block never blocks the streaming thread: the tap resamples each buffer and appends it to an UNBOUNDED internal sample queue, and a single background worker classifies VisioForge.Core.Types.X.AI.AudioEventDetectorSettings.WindowSeconds windows off that queue. The queue never discards, so a source that outruns the worker (a live source on a slow provider, or an offline file decoded at full speed by an unsynced renderer) only grows the backlog — at 16 kHz mono that costs 64 KB per buffered second — and the answer to a machine that is permanently slower than the source is faster hardware (or a GPU execution provider), never dropped audio. The per-class scores are smoothed and a per-class Schmitt trigger (enter at VisioForge.Core.Types.X.AI.AudioEventDetectorSettings.Threshold, leave at that times VisioForge.Core.Types.X.AI.AudioEventDetectorSettings.ReleaseRatio) collapses one continuous sound into a single event with a start/end time. VisioForge.Core.MediaBlocks.AI.AudioEventDetectorBlock.OnAudioEvent (and the optional VisioForge.Core.MediaBlocks.AI.AudioEventDetectorBlock.OnScores) are raised on the worker thread — marshal to the UI thread before touching UI. At end-of-stream the trailing window and any still-active events are flushed before teardown. Stopping the pipeline early is not the same thing: a stop abandons whatever backlog the worker has not reached yet, closes out the events detected so far and returns promptly, rather than making the caller wait out an analysis it just cancelled.

Constructors

AudioEventDetectorBlock(AudioEventDetectorSettings)

Initializes a new instance of the VisioForge.Core.MediaBlocks.AI.AudioEventDetectorBlock class.

public AudioEventDetectorBlock(AudioEventDetectorSettings settings)

Parameters

settings AudioEventDetectorSettings

The audio-event settings. Must specify a valid YAMNet ONNX model path.

Exceptions

ArgumentNullException

Thrown when settings is null.

Properties

ActiveProvider

Gets the execution provider the ONNX session engaged. The value is latched when VisioForge.Core.MediaBlocks.AI.AudioEventDetectorBlock.Build creates the session and stays valid after the stream ends and the engine is torn down, which is when a caller normally logs it. It reads VisioForge.Core.Types.X.AI.OnnxExecutionProvider.CPU before the first successful build, and a build that fails before the session is created resets it to VisioForge.Core.Types.X.AI.OnnxExecutionProvider.CPU rather than leaving the previous run's provider in place.

public OnnxExecutionProvider ActiveProvider { get; }

Property Value

OnnxExecutionProvider

Input

Gets the primary input pad.

public override MediaBlockPad Input { get; }

Property Value

MediaBlockPad

Inputs

Gets the array of all input pads.

public override MediaBlockPad[] Inputs { get; }

Property Value

MediaBlockPad[]

Output

Gets the primary output pad.

public override MediaBlockPad Output { get; }

Property Value

MediaBlockPad

Outputs

Gets the array of all output pads.

public override MediaBlockPad[] Outputs { get; }

Property Value

MediaBlockPad[]

Type

Gets the type of the media block.

public override MediaBlockType Type { get; }

Property Value

MediaBlockType

Methods

Build()

Constructs the internal GStreamer elements and pads and initializes the YAMNet engine and worker.

public override bool Build()

Returns

bool

true if successful, false otherwise.

CleanUp()

Cleans up internal resources when the pipeline stops. Build() recreates them on the next start.

public void CleanUp()

Dispose(bool)

Releases unmanaged and - optionally - managed resources.

protected override void Dispose(bool disposing)

Parameters

disposing bool

true to release both managed and unmanaged resources.

GetCore()

Gets the core GStreamer element wrapped by this block.

public BaseElement GetCore()

Returns

BaseElement

The VisioForge.Core.GStreamer.Base.BaseElement wrapper, or null if not built yet.

GetElement()

Gets the GStreamer element instance.

public Element GetElement()

Returns

Element

The Gst.Element, or null if not built yet.

IsAvailable()

Determines whether this media block is available. Correct NuGet SDK redist should be included in your project.

public static bool IsAvailable()

Returns

bool

true if available; otherwise, false.

RequestStop()

Early stop signal from the pipeline: stops the worker so the streaming thread and teardown can proceed. Non-blocking. The remaining backlog is abandoned un-classified — a stop means stop now; a stream that ends by itself drains its backlog instead.

public void RequestStop()

IMediaBlockInternals.SetContext(MediaBlocksPipeline)

Sets the pipeline context for this block.

void IMediaBlockInternals.SetContext(MediaBlocksPipeline pipeline)

Parameters

pipeline MediaBlocksPipeline

The pipeline.

OnAudioEvent

Occurs when a sound has been detected (one event per continuous detection, raised when the sound ends or at end-of-stream). Raised on the background inference worker thread.

public event EventHandler<AudioEventArgs> OnAudioEvent

Event Type

EventHandler<AudioEventArgs>

OnScores

Occurs for every processed window with the top-K class scores when VisioForge.Core.Types.X.AI.AudioEventDetectorSettings.EnableScoresEvent is true. Raised on the worker thread.

public event EventHandler<AudioScoresEventArgs> OnScores

Event Type

EventHandler<AudioScoresEventArgs>

See Also

MediaBlock
IMediaBlockInternals