Comment on Video Capture Issues with Windows 10 Anniversary Update

There is a comment from MSFT’s Mike M on MSDN Forums on recent issue with compressed video capture. I am pulling it out completely as a quote below:

I’d like to start off by providing you guys a little more context on the behavior you’re encountering.

One of the main reasons that Windows is decoding MJPEG for your applications is because of performance. With the Anniversary Update to Windows 10, it is now possible for multiple applications to access the camera in ways that weren’t possible before. It was important for us to enable concurrent camera access, so Windows Hello, Microsoft Hololens and other products and features could reliably assume that the camera would be available at any given time, regardless of what other applications may be accessing it. One of the reasons this led to the MJPEG decoding is because we wanted to prevent multiple applications from decoding the same stream at the same time, which would be a duplicated effort and thus an unnecessary performance hit. This can be even more noticeable or perhaps trigger error cases on in-market devices with a hardware decoder which may be limited on how many decodes can take place simultaneously. We wanted to prevent applications from unknowingly degrading the user experience due to a platform change.

The reasoning for H.264 being decoded can get a little more complicated (and I’m just learning the details myself as I talk to other members of the team), but the basics revolve around how H.264 allows for encoding parameters to be changed on the camera directly, and how in a situation where multiple applications are making use of this control path, they could interfere with each other. Regarding Roman’s concerns about Lync: both Lync and Skype are partner teams, and we stay in touch throughout the development process, so the camera functionality in those applications will continue to work.

So yes, MJPEG and H.264 being decoded / filtered out is the result of a set of features we needed to implement, and this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year.  We worked with partners to make sure their applications continued to function throughout this change, but we have done a poor job communicating this change out to you guys. We dropped the ball on that front, so I’d like to offer my apologies to you all. We’re working on getting better documentation out, to help answer any questions you may have. Of course, you can always reach out to us via these forums for specific issues, as we monitor them regularly, or file feedback using the Feedback Hub. We’re constantly collecting feedback on this and other issues, so we can better understand the impact on our application developers and customers. If you’re having issues adapting your application code to the NV12 / YUY2 media types, we’d like to support you through the changes you may need to make. To get you started, please refer to the documentation links in my previous post. If there are reasons why working with this format isn’t feasible for your project, please let me know, and I’ll present them to the rest of the team, to try and find the best solution for your case.

Dacuda and Stephan B, I’m curious about your specific situations, since you report that this change is breaking functionality for your customers. Are your customers using custom camera hardware? Is the set of supported cameras restricted by your applications? How do your applications deal with devices like the Surface Pro 4, Surface Book, or Dell Venue Pro, which wouldn’t offer the media types your applications are relying on?
I’d like to wrap up this wall of text by letting you know that your feedback here and through other channels is greatly appreciated and something that’s on our radar. We’re trying to look into what other options we can offer you to be able to improve on this for your (and our) customers, so stay tuned! I invite you to please subscribe to this thread (use the “Alert me” link at the top), and I’ll keep you guys updated on what we find. Thanks!

Basically, it’s bad news for those who consume compressed video from capture devices – the breaking change is intentional. Something is offered in exchange and I hope someone will present the platform changes in a friendly readable document. In particular, Microsoft seems to be adding VP8/9 video decoder and encoder in this new platform version (more later on that perhaps).

Utility Clearance: Matrox DSX SDK Capabilities

The tool gathers and formats capture capabilities of hardware accessible via Matrox DSX SDK:

Matrox DSX SDK is a feature-rich development toolkit that enables Matrox’s industry-leading DSX hardware capabilities. OEMs use this hardware and software to build complete solutions that ensure 24/7 on-air reliability in a broadcast environment. Matrox DSX SDK customers have access to unlimited premium support from experienced applications engineers, personalized training courses, design review programs, and feature-customization to meet their specific needs.

Specifically, I used it a lot (X.AVCio, MXO2 series and other) to identify inputs, their properties, genlock, auto-detection of video signal.


DeckLinkCapabilities: A Printout of Capabilities of Blackmagic Design/DeckLink Hardware

The tool provides a user- (well, actually a developer-) friendly printout of capabilities accessible via Blackmagic Design DeckLink SDK for DeckLink series of hardware. This covers features of DeckLink and Intensity series of hardware for video/audio capture, accessible via vendor’s SDK. The data is printed out in Markdown format, easy to read on its own and even nicer on Markdown Pad.

Alax.Info DeckLinkCapabilities

Alax.Info DeckLinkCapabilities Output on MarkdownPad

The hardware is good, and the SDK is designed nicely as well, however the product range is wide and capabilities vary. So do driver and SDK versions, and the tool is handy to quick check the information out. One might want to use SDK for many reasons, including the following ideas I am sharing off the top of my head:

  • wanting to leverage the full feature set of the hardware
  • operate at minimal overhead
  • user a simpler API compared to generic media APIs
  • being unsatisfied with DirectShow interface provided by Blackmagic Design


CaptureClock: Utility to Check Video/Audio Capture Rates

Someone discovered the utility while browsing my public repository (the app prompts to post data back to the website, and the anonymous user accepted the offer and posted the report from this unpublished application), so I have to drop a few lines about the tool.

The idea is basically straightforward: live capture involves attaching time stamps to media samples, and there is a chance that the time stamps slide away causing unwanted effects on captured clip. The application captures video and audio simultaneously and tracks media sample time stamps, and compares them against system clock as well. Having it simply run for a few minutes one can see how the capture is doing and if any of the timings drift away. Being stopped it puts report onto clipboard and optionally posts it back to me online (no actually specific intent about this data, however if you want to share data for a device that does drift away, you are to only click once to send me the details).

CaptureClock operation

The output is on clipboard in tab-separated values (TSV) format:

Computer Name   PSI
Windows Version 6.1.7601 Service Pack 1
Video Device    Conexant's BtPCI Capture    @device:pnp:\\?\pci#ven_109e&dev_036e&subsys_18511851&rev_02#4&39c3dd91&0&08f0#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global
Audio Device    Stereo Mix (Realtek High Defini @device:cm:{33D9A762-90C8-11D0-BD43-00A0C911CE86}\Stereo Mix (Realtek High Defini

System Time Video Sample Count  Video Sample Time   Relative Video Sample Time  Audio Sample Count  Audio Sample Time   Relative Audio Sample Time
30439   907 30381   -57 304 30291   -147

Or you might prefer pasting it onto Excel:

CapptureClock Output on Excel

By the way, this is also an easy way to ensure devices are operational and check effective video frame rate.

Download links:

Memory Allocators with Blackmagic DeckLink video capture hardware

With all respect to Blackmagic Design products, there are bugs so painfully hard to identify. Capture board driver has specific requirements as for buffers allocated for captured video frames. The parameters of memory allocator in used are going in ALLOCATOR_PROPERTIES structure.

typedef struct _AllocatorProperties {
  long cBuffers; // 30
  long cbBuffer;
  long cbAlign; // 16
  long cbPrefix;

The board is going to use 30 buffers (for a second of video specific capture with NTSC video) and the buffers have to have specific alignment. This is reasonable and goes well if the capture filter, backed in turn by WDM driver, runs with all defaults set and driver’s defaults apply.

As a part of buffer negotiation however, the capture filter queries its peer if there are other suggestions as for buffer allocation. There might be some, why not? Most of the filters don’t have any preferences and thus do fine, because default implementation on the input pin is E_NOTIMPL:

// what requirements do we have of the allocator - override if you want
// to support other people's allocators but need a specific alignment
// or prefix.
CBaseInputPin::GetAllocatorRequirements(__out ALLOCATOR_PROPERTIES*pProps)
    return E_NOTIMPL;

The problem is that that if there are any, esp. those for different alignment, the capture filter accepts them, prefers them to the ones he would use otherwise. Then when streaming starts he finds itself checking the properties of the allocator and as the check fails, the filter does not do any streaming without indicating any runtime error.

That’s it – you hit Start button and there is nothing but blackness, without any single video frame streamed through. And the reason is just downstream peer saying “well, I’d say two buffers on the allocator would be good for me – but since I am not responsible for final choice you can do whatever you think is okay”.

As the capture filter is responsible to choose the allocator and set it up, and alignment is so important for it, then why wouldn’t it negotiate and apply the values that would work? If it was unable to end with with negotiating anything meaningful, why would not it indicate a streaming error? It should have definitely done that.

IP Video Source: Pure JPEG URLs and Software Version

This does not update the software with new features, but there are a few simple things worth mentioning explicitly.

The first is that virtual DirectShow camera device can be set up with both M-JPEG and JPEG URLs. That is, IP cameras which do not implement M-JPEG, or implement it in a buggy way (there is a *huge* deal of such out there) can still be set up to send video as individual video frames/images as long as they implement JPEG snapshots. This is taking place often at a lower frame rate, but still works.

The driver will automatically detect type of URL (by response from the device) and will choose best access method for the given URL.

Second is that if you are looking for IP Video Source software version, such as to check against available updates, it is here on the UI (right click the caption):

IP Video Source: 64-bit version, resolution flexibility, Adobe FMLE

The IP Video Source update provides several improvements to the driver:

  • copy/paste feature to backup, restore, or synchronize installed devices between 32-bit and 64-bit versions
  • 64-bit version and .MSI
  • dynamic video resizing (via Video Resizer DSP)
  • Adobe FMLE compatibility

Updates in greater detail follow.

Device Copy/Paste Feature

The video device management window is providing Copy and Paste buttons, which let user transfer device information, including name and settings, through clipboard for various purposes:

  • save data in order to be able to restore devices later
  • restore devices from saved list, or re-create from a list saved on another machine
  • duplicate a device
  • synchronize devices between 32-bit and 64-bit versions

The device data is a text, one line per device, lines in comma-separated values (CSV) format.