Some of high resolution HDMI video capture hardware

Recent submissions of DirectShow and Media Foundation capture capabilities brought references to new interesting hardware.

1. Magewell USB Capture HDMI Gen 2 AKA XI100DUSB-HDMI — USB\VID_2935&PID_0001

Reports ability to capture up to 1920×1080 video at 60 frames per second in YUY2 and RGB pixel formats.

2. Inogeni 4K HDMI to USB 3.0 Converter AKA 2A34-INOGENI 4K2USB3 — USB\VID_2997&PID_0004

Goes beyond that and promises resolutions up to 4096×2160@24, 2560×1440@60

Also, not only with Windows 10, but it seems that it can work well with older OSes like Windows 7 as well.

3. Magewell Pro Capture HDMI 4K — PCI\VEN_1CD7&DEV_0012

A PCI version of HDMI capture board with a rich offering of media types and 2K resolutions

4. Some simpler noname HDMICap USB dongle for up to 1920×1080 YUV and RGB capture — USB\VID_EB1A&PID_7595

(I did not have the actual hardware, I just see reported specifications)

Applicability of Virtual DirectShow Sources

Virtual DirectShow  sources have been a long time synonym of software-only camera implementation exposed to applications along with physical cameras in a way that applications consume the sources without making a difference whether the camera is real or virtual. Vivek’s template was a starting point for many:

Capture Source Filter filter (version 0.1) 86 KB zipped, includes binaries.  A sample source filter that emulates a video capture device contributed by Vivek (rep movsd from the public newsgroups).  Thanks Vivek!  TMH has not tested this filter yet.  Ask questions about this on microsoft.public.win32.programmer.directx.video.

With API changes over years, the sample and the concept is still understood as the method of adding a virtual camera, however new scenarios exist where the concept no longer works. Typical problems:

  1. 64-bit applications cannot consume virtual 32-bit virtual sources
  2. Virtual sources are no visible and accessible to applications consuming video using Media Foundation API

The diagram below explains the applicability of virtual cameras:

Applicability of Virtual DirectShow Sources

Important is that virtual sources can only be consumed by the DirectShow-based applications of the same bitness.

If source developer needs to synchronize virtual source throughout multiple applications (e.g. video is synthesized by another application and needs to be deliverable to multiple clients), he needs to add interprocess synchronization on the backyard of virtual source.

If developer needs to support both 32- and 64-bit apps, he needs both variants of virtual sources registered, and possibly synchronization of the kind of the paragraph above.

The only virtual device which is visible to all video capture applications if implemented by kernel level driver (implementations are rare but exist).

See also:

Small correction for DirectShow BaseClasses CTransInPlaceFilter::Copy

False assertion failure in DirectShow BaseClasses transip.cpp, in CTransInPlaceFilter::Copy:

ASSERT(lDestSize >= lSourceSize && lDestSize >= lDataLength);
[...]
CopyMemory((PVOID) pDestBuffer, (PVOID) pSourceBuffer, lDataLength);

The code asserts on “lDestSize >= lSourceSize” which is true in most cases, but it does not have or need to be true. The code below copies just lDataLength bytes and assertion on buffer sizes is, generally, excessive. Valid scenarios do exist with this assertion popping up for no reason.

KB3176938’s Frame Server update visually

  1. M-JPEG and H.264 media types are available again (good)
  2. Nevertheless connected, H.264 video is not processed correctly; new bug or old one? Not clear. Even though it sort of works, in DirectShow it looks broken in another new way (this and not just this), perhaps a collateral damage and maybe never ever fixed…
  3. There is no camera sharing between the applications even though it was the justification for the changes in first place. For now Frame Server is just useless overhead, which adds bad stuff, is polished a bit to do not so much harm, and maybe turns to be good some time later.
    • for the record, the camera works in Skype when it is not consumed elsewhere concurrently

BTW the hack that bypasses FrameServer survived the update and remains in good standing.

DirectShowCaptureCapabilities and MediaFoundationCaptureCapabilities: API version of EnableFrameServerMode state

Both tools now include exact version of the API and also include an export or registry key related to frame server.

Capture Capabilities: API Version and State

mfcore.dll version of 10.0.14393.105 corresponds to Cumulative Update for Windows 10 Version 1607: August 31, 2016 also known as KB3176938 with DirectShow and Media Foundation improvement for Windows 10 Anniversary Update that restores availability of compressed media types.

See:

Enumeration of DirectShow Capture Capabilities (Video and Audio)

Media Foundation Video/Audio Capture Capabilities

Anniversary Webcam Saga: It’s clear who’s guilty, now what to do? (Updated)

As new and new people discover the Windows 10 Anniversary Update breaking changes (expectedly running mad), let’s reiterate the possible solutions:

  1. You don’t like the idea that video sharing service adds latency, and adds man-in-the-middle access/spying over a video feed.
    See #6 below.
  2. You are consuming raw video from camera using one of the uncompressed modes within USB 2.0 bandwidth.
    You are likely to be not affected by the changes.
  3. You are consuming raw video from camera but resolution/rate combination makes it unable to capture raw video, so you captured M-JPEG instead and decoded that, via DirectShow API.
    It is no longer possible, but you can use Media Foundation API instead. Or someone will develop a wrapper that re-exposes Media Foundation captured video via DirectShow.
  4. Same as #3 above but via Media Foundation API.
    You have the option to consume already decoded video, new subsystem will automatically capture M-JPEG and decode into NV12.
  5. You take advantage of compressed format of video captured (DirectShow or Media Foundation) so that you don’t need compress it for storage or network transmission purposes.
    Compressed captured video is no longer available, see #6 below.
  6. You take advantage of H.264 video capture offered by UVC 1.5 device, including fine tuning of hardware H.264 compression.
    Just as in #1 and #5 above, you are in trouble. Windows Camera Frame Server no longer offers access to such video feed. You need a hack (yes, it’s confirmed to be possible) that restores original behavior of video capture hardware.

These and other reasons related to the fact that applications no longer talk to real capture device, but rather a Frame Server Client that proxies a web camera, will possibly require that video capture applications are updated in order to work well in new version of the operating system.

It is unclear if and how Microsoft and Media Foundation team will respond to customer pain voices. First, it looked as a bug and one could expect a response and fix. But with the information from Windows Camera Team it looks completely different. No, they did not accidentally break it up – it was a planned change. Then they connected new behavior with new Microsoft Products – new products rely on new behavior. Then they did a few nasty things, not just one: added proxy service, killed UVC related compression control over the device, reduced range of operation modes for DirectShow they look for ways to deprecate, conceptually removed compressed video capture modes. I think there is no way back – Windows Camera Frame Server is new reality. The best to expect is that some of the mentioned problems are relaxed by offering greater flexibility by the platform. Maybe they will add some sort of exclusive modes for video capture or “professional” hardware which offers more through the API. In any event these changes are unlikely to appear soon, as they will go through the full cycle of development and take months to get delivered. Public pressure might force that to appear rather earlier than later, but I don’t think it is what is going to happen.

16 Aug update: Windows Camera Team reported that they see customer pain and will do something to ease it shortly. As I see it, they will address scenarios #3, #4, #5 above, for MJPG video, to allow compressed formats pass Frame Server so that users could consume them from their applications and Frame Server would be able to release frames not just after decoder, but also before the shared decoder. Also as use of H.264 is limited, it might be not included into hotfix at all, or will be included much later being given more serious consideration (which might end up as dropped support in DirectShow and something new introduced for Media Foundation).

19 Aug update: Someone took time to locate a registry value. User WithinRafael on MSDN Forums:

Try opening up

HKLM\SOFTWARE\Microsoft\Windows Media Foundation\Platform (32- and 64-bit OS)
HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows Media Foundation\Platform (64-bit OS)

and add a DWORD value with name EnableFrameServerMode. Set its value to 0 and try again.

Put a sticky note on your monitor to revisit this if/when Microsoft issues a fix.

Also:

Untitled

Video Capture in Windows 10 Anniversary Update Again: MJPG is still here but hidden by new Frame Server thing

The problem with video capture issues looks more or less clear.

As explained by Mike M here, the breaking changes in Windows 10 Anniversary Update are caused by intentional redesign of the platform that enable shared access to video capture devices.

… it is now possible for multiple applications to access the camera in ways that weren’t possible before. It was important for us to enable concurrent camera access, so Windows Hello, Microsoft Hololens and other products and features could reliably assume that the camera would be available at any given time, regardless of what other applications may be accessing it.

Originally video capture application were highly performance sensitive due to insufficient horsepower of computers overall, and sharing of video capture sessions between the applications was not on agenda. Then Microsoft hibernated for over a decade and did not do updates to the platform to follow software and hardware trends. Those needing camera in 2+ applications had to use third party camera splitting software. Time has come to include video sharing to the platform and… that washed away support for compressed video formats. If camera is shared, who is going to decode video into presentable format? Guys at Microsoft decided that they will, that is it’s now “decode then share between applications” scenario.

When an application runs video capture session, Windows 10 Anniversary Update now runs the actual session in a service process. A new Windows Camera Frame Server service is responsible to acquire video, decode and distribute it.

Windows Camera Frame Server Service

Applications access FrameServer service with the help of FSClient.dll connecting to shared service which runs actual session.

I am not sure how sharing works exactly, but I was unable to start two TopoEdit instances doing video capture from the same camera. Presumably, default behavior is still imitating exclusive use of hardware and possibly priority clients (like mentioned Windows Hello) have new ways to take over video capture device on demand, or we will see new functionality with respective SDK/documentation update.

Applications now – as it is assumed from the description – get the only option to communicate to FrameServer service and not the video capture source directly. On that way, formats like MJPG and H264 are lost.

As recent comments indicate this to be a well planned and scheduled scenario, it looks unlikely that things are going to change. It was decided that there is no exclusive mode video capture, just shared. Developers are to wait for possibly changed attitude and something similar to WASAPI exclusive low-latency mode for those specific application which need it.

So yes, MJPEG and H.264 being decoded / filtered out is the result of a set of features we needed to implement, and this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year. We worked with partners to make sure their applications continued to function throughout this change, but we have done a poor job communicating this change out to you guys. We dropped the ball on that front, so I’d like to offer my apologies to you all.

A small relief is that they restructured the platform and not dropped the support for MJPG and H264 in first place. Okay, there is no formal access to compressed streams using standard API but a stab into doing it undocumented way shows that all the gear remains in place.

A small proof-of-concept DirectShow video source filter that talks to Logitech C930e camera bypassing newly introduced stuff is confirming that streams like 1920×1080@30 MJPG are still supported by the camera and are operational. That is, it is still possible to stream MJPG and H264 from USB web cameras, specifically at modes exceeding standard USB 2.0 bandwidth limit for raw video, and eliminating software compression:

DirectShowLogitechC930eVideoSource

Logitech C930e Live

This of course takes again exclusively control over the camera and prevents from sharing video feed as the update intended. However, the video itself is where is was.

Logitech C930 Running 1920x1080@30 MJPG

There is no public source and/or details on this filter because it’s sensitive to undocumented behavior of Media Foundation platform. Just as a demo, the DLLs are there: Win32, x64. (limited to Logitech Webcam C930e’s highest MJPG mode but basically the method could work for any MJPG camera, and C930e’s H264 too).

That is, if your application is broken by Windows 10 Anniversary Update because you simply assumed availability of specific modes, then there is a chance that update of the application to make it compatible to new platform design with FrameServer service could fix it. If you intentionally consumed compressed video for quality, rate and performance reasons then you’re in trouble and there is no really a solution from Microsoft is expected soon. Perhaps the best would be to not upgrade to Anniversary Update.