Fooling Around

// Oprogramowanie Roman Ryltsov

Microsoft’s MPEG-DASH client implementation inaccurately handles SSR flavor of streaming media

Posted on by Roman

Recent additions to MPEG-DASH specification (ISO/IEC 23009-1 5th ed., AMD3; also SCTE 214-6) offer new ways to implement low-latency low-delay playback: using so called Segment Sequence Representations (SSR).

MPGE-DASH manifests advertise this flavor of data by including a respective EssentialProperty.

Before Shaka Player Demo added support for this technology, they ignored the respective adaptation set as “unknown” and “unsupported”, which seems to be the right way to address something you do not understand.

Here is the Microsoft’s approach.

They do support MPEG-DASH playback in, for example, their XAML MediaElement (below is the screenshot from PlayReady sample):

They apparently support adaptation set switching urn:mpeg:dash:adaptation-set-switching:2016 as they mix downloads from regular and SSR-enabled adaptation sets.

However they do not support segment sequence representations urn:mpeg:dash:ssr:2023 themselves as they attempt to download content without replacing $SubNumber$ placeholder, and, hence, they ignore the EssentialProperty markup: the adaptation set is marked as something new they do not understand but they take chances to play it back.

GPU shader Sobel filter and video capture with Media Foundation Capture Engine API

Posted on by Roman

Back to some experiments…

The current video capture API in Windows is Media Foundation Capture Engine API (AKA IMFCaptureEngine and mfcaptureengine.h). Media Foundation is layered: you can work at lower level with video capture Media Sources, but if you don’t want to go into details you have the Capture Engine.

The application continues a good old CaptureEngineVideoCapture sample from Windows Classic Samples, and adds an effect in form factor of Media Foundation Transform to the engine.

One of the unobvious aspects is that video stream goes with data backed in video memory by default. Hence, the effect would preferrably handle the stream on associated Direct3D 11 device.

The effect here is a HLSL equivalent of Webcam edge glow shadertoy, and processes the video from webcamera on GPU, via shader implementing Sobel operator. The data does not leave video memory and continues its travel along capture engine to preview visualization.

The shader code can be looked up at in the executable resources, it is compiled on runtime.

Capture-20240413-a887e2ae105d1623b8707cdc971916c5d8a58aebDownload