Fooling Around

// Oprogramowanie Roman Ryltsov

CleanPoint markup fun with a fragmented MP4 file and Media Foundation MPEG-4 Source

Posted on by Roman

MPEG-4 Media Foundation Source stubbornly keeps marking a second video sample with a MFSampleExtension_CleanPoint flag even though nothing suggests that the video frame is an IDR frame.

The actual video frame is a P frame both in terms of MP4 box formatting and contained NAL units (the video is in fact an “infinite GOP” flavor of recording where all frames are P frames except the very first IDR one).

The problem is specific to fragmented MP4 files (and maybe even a subset of those), however is pretty much consistent and shows up with both H.264 and H.265/HEVC video.

Use of ICodecAPI interface with a video encoder managed by Media Foundation Sink Writer instance

Posted on by Roman

A bump of StackOverflow post about Media Foundation design flaw related to video encoding.

Set attributes via ICodecAPI for a H.264 IMFSinkWriter Encoder

I am trying to tweak the attributes of the H.264 encoder created via ActivateObject() by retrieving the ICodecAPI interface to it. Although I do not get errors, my settings are not taken into account. […]

Media Foundation’s Sink Writer is a simplified API with a encoder configuration question slipped away. The fundamental problem here is that you don’t own the encoder MFT and you are accessing it over the writer’s head, then the behavior of encoders around changing settings after everything is set up depends on implementation, which is in encoder’s case a vendor specific implementation and might vary across hardware.

Your more reliable option is to manage encoding MFT directly and supply Sink Writer with already encoded video.

Your potential trick to make things work with less of effort is to retrieve IMFTransform of the encoder as well and clear and then set back the input/output media types after you finished with ICodecAPI update. Nudging the media types, you suggest that encoder re-configures the internals and it would do this already having your fine tunings. Note that this, generally speaking, might have side issues.

Continue reading →

Source code to fit 80 columns of text

Posted on by Roman

LLVM Coding Standards – Source Code Width:

Write your code to fit within 80 columns of text. This helps those of us who like to print out code and look at your code in an xterm without resizing it.
The longer answer is that there must be some limit to the width of the code in order to reasonably allow developers to have multiple files side-by-side in windows on a modest display. If you are going to pick a width limit, it is somewhat arbitrary but you might as well pick something standard. Going with 90 columns (for example) instead of 80 columns wouldn’t add any significant value and would be detrimental to printing out code. Also many other projects have standardized on 80 columns, so some people have already configured their editors for it (vs something else, like 90 columns).
This is one of many contentious issues in coding standards, but it is not up for debate.

Is there any more stupid rule than to wrap around source code lines just because someone would possibly look at code in an xterm?

So source is consuming less than 25% width of a quote ordinary monitor wasting all this space on the right. Same time, the source code lines are objectively long and are massively wrapped around.

Wrapping destroys readability of code.

Re-wrapping source code has an obvious negative effect on change tracking.

I, for once, want to see as much of source code as possible momentarily because it helps to have a picture of what is going on. Information at the end of lines is less important so it is not a big deal even if it goes beyond the right visible margin, but it’s important to have as many LINES of code as possible – I would even prefer to skip blank lines and utilize IDE’s capabilities to collapse comments, functions, regions and scopes. For this reason some developers even rotate monitors into portrait mode – to see more of source code at a time.

Fitting 80 columns and having it even not up for debate is a clearly genius move to keep devs productive. Through continuous irritation.

Video compression in AVerMedia Live Gamer Ultra GC553

Posted on by Roman

“The next generation of game capture is here.” The device addresses needs of real time capture of video signal: offering a pass-through HDMI connection the box provides a video capture sink with USB 3.1 Type C interface and makes the video signal available to video capture applications via standard DirectShow and Media Foundation APIs.

I was interested whether the device implements video compression, H.264 and/or H.265/HEVC in hardware. The technical specifications include:

• Max Pass-Through Resolutions:2160p60 HDR /1440p144 / 1080p240
• Max Record Resolutions:2160p30 / 1440p60 / 1080p120 / 1080p60 HDR
• Supported Resolutions (Video input):2160p, 1440p, 1080p, 1080i, 720p, 576p, 480p
• Record Format: MPEG 4 (H.264+AAC) or (H.265+AAC)*

…

Notes:
*H.265 Compression and HDR are supported by RECentral

So there is a direct mention of video compression, and given the state of the technology and the price of the box it makes sense to have it there. Logitech C930e camera has been offering H.264 video compression onboard for years.

So is it there in the Ultra thing? NO, IT IS NOT. Pathetic…

One could guess this of course from a study of FAQ section in the part of third party software configuration. The software is clearly expected to use external compression capabilities. However popular software is also known to not use the latest stuff, so there was a little chance that hardware codec is still there. I think it would fair to include that right there into technical specification that the product does not offer any encoding capabilities.

The good thing is that the box offers 10-bit video capture up to 2560×1440@30 – there is not so much of inexpensive hardware capable to do such job.

The specification mentions high rate 1920×1080@120 mode but I don’t see it in the effectively advertised capabilities.

Also, video capture capabilities in Media Foundation API suggest that it is possible to capture into video memory bypassing system memory mapping/copy. Even though it is irrelevant to most of the applications, some newer ones including those leveraging UWP video capture API could take advantage (such as, for example, video capture apps running on low power consumption devices).

Media Foundation on Raspberry Pi 3 Model B+

Posted on by Roman

The interesting part with live WebM Media Foundation media source I mentioned in the previous post is that the whole thing works great on… Raspberry Pi 3 Model B+ running Windows 10 IoT Core (RaspberryPi 3B+ Technical Preview Build 17661).

Windows 10 IoT has quite the same Media Foundation infrastructure as in other Universal Windows Platform environments (Desktop, Xbox, HoloLens) including the core API, primitives, support in XAML MediaElement (MediaPlayerElement). There is no DirectX support on Raspberry Pi 3 Model B+ and video delivery fails, however this is a sort of known/expected problem with the Technical Preview build. Audio playback is okay.

The picture above is taken on C# UWP application (that’s ARM platform) running a MediaPlayerElement control taking live audio signal from network using a Windows.Networking.Sockets.MessageWebSocket connection.

A custom (the platform does not have a capable primitive out of the box) WebM live media source forwards the signal to media element for low latency audio playback. The codec is Opus and, yes, stock Media Foundation audio decoder MFT decodes the signal just fine.

Parsing live WebM stream is not so easy

Posted on by Roman

A magic transition from E_BUFFER_NOT_FULL to E_FILE_FORMAT_INVALID in depths of libwebm…

Media.dll!mkvparser::Block::Parse(const mkvparser::Cluster * pCluster) Line 7630    C++
Media.dll!mkvparser::BlockGroup::Parse() Line 7579  C++
Media.dll!mkvparser::Cluster::CreateBlockGroup(__int64 start_offset, __int64 size, __int64 discard_padding) Line 7253   C++
Media.dll!mkvparser::Cluster::CreateBlock(__int64 id, __int64 pos, __int64 size, __int64 discard_padding) Line 7154 C++
Media.dll!mkvparser::Cluster::ParseBlockGroup(__int64 payload_size, __int64 & pos, long & len) Line 6724    C++
Media.dll!mkvparser::Cluster::Parse(__int64 & pos, long & len) Line 6381    C++
Media.dll!mkvparser::Cluster::GetNext(const mkvparser::BlockEntry * pCurr, const mkvparser::BlockEntry * & pNext) Line 7369 C++
Media.dll!WebmLiveMediaSource::HandleData(std::vector,std::allocator > > & AsyncResultVector) Line 757    C++
Media.dll!WebmLiveMediaSource::ReadInvoke(AsyncCallbackT * AsyncCallback, IMFAsyncResult * AsyncResult) Line 1176    C++
Media.dll!AsyncCallbackT::Invoke(IMFAsyncResult * AsyncResult) Line 2344 C++
RTWorkQ.dll!CSerialWorkQueue::QueueItem::ExecuteWorkItem()  Unknown
RTWorkQ.dll!CSerialWorkQueue::QueueItem::OnWorkItemAsyncCallback::Invoke()  Unknown
RTWorkQ.dll!ThreadPoolWorkCallback()    Unknown
ntdll.dll!TppWorkpExecuteCallback() Unknown
ntdll.dll!TppWorkerThread() Unknown
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown

The problem here is that the stream is live and is available in increments in ultra low latency consumption mode. The libwebm library code structure does not suggest it is sufficiently robust for such processing (or maybe it’s good and it’s just one bug? who can tell) even though it apparently have multiple code paths added for live signal. There is no problem to parse a complete file, of course, and then even a retry from the same point succeeds once new data is appended.

It looks like a reasonable workaround here is to check whether we are close to the edge of the stream and temporarily ignore errors like this.

Library parsing performance/efficiency is also a bit questionable. The library is not capable to process incremental reads via mkvparser::IMkvReader. Instead it keeps steppping back all over the parsing process and the live signal source has to keep a bit of processed data because it can be requested once again…

Apparently UWP as a platform has code capable to process this type of data reliably as MediaElement has built-in support for the format in Media Source Extensions (MSE) mode. However, this implementation is limited for internal consumers.