Video Capture in Windows 10 Anniversary Update Again: MJPG is still here but hidden by new Frame Server thing

The problem with video capture issues looks more or less clear.

As explained by Mike M here, the breaking changes in Windows 10 Anniversary Update are caused by intentional redesign of the platform that enable shared access to video capture devices.

… it is now possible for multiple applications to access the camera in ways that weren’t possible before. It was important for us to enable concurrent camera access, so Windows Hello, Microsoft Hololens and other products and features could reliably assume that the camera would be available at any given time, regardless of what other applications may be accessing it.

Originally video capture application were highly performance sensitive due to insufficient horsepower of computers overall, and sharing of video capture sessions between the applications was not on agenda. Then Microsoft hibernated for over a decade and did not do updates to the platform to follow software and hardware trends. Those needing camera in 2+ applications had to use third party camera splitting software. Time has come to include video sharing to the platform and… that washed away support for compressed video formats. If camera is shared, who is going to decode video into presentable format? Guys at Microsoft decided that they will, that is it’s now “decode then share between applications” scenario.

When an application runs video capture session, Windows 10 Anniversary Update now runs the actual session in a service process. A new Windows Camera Frame Server service is responsible to acquire video, decode and distribute it.

Windows Camera Frame Server Service

Applications access FrameServer service with the help of FSClient.dll connecting to shared service which runs actual session.

I am not sure how sharing works exactly, but I was unable to start two TopoEdit instances doing video capture from the same camera. Presumably, default behavior is still imitating exclusive use of hardware and possibly priority clients (like mentioned Windows Hello) have new ways to take over video capture device on demand, or we will see new functionality with respective SDK/documentation update.

Applications now – as it is assumed from the description – get the only option to communicate to FrameServer service and not the video capture source directly. On that way, formats like MJPG and H264 are lost.

As recent comments indicate this to be a well planned and scheduled scenario, it looks unlikely that things are going to change. It was decided that there is no exclusive mode video capture, just shared. Developers are to wait for possibly changed attitude and something similar to WASAPI exclusive low-latency mode for those specific application which need it.

So yes, MJPEG and H.264 being decoded / filtered out is the result of a set of features we needed to implement, and this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year. We worked with partners to make sure their applications continued to function throughout this change, but we have done a poor job communicating this change out to you guys. We dropped the ball on that front, so I’d like to offer my apologies to you all.

A small relief is that they restructured the platform and not dropped the support for MJPG and H264 in first place. Okay, there is no formal access to compressed streams using standard API but a stab into doing it undocumented way shows that all the gear remains in place.

A small proof-of-concept DirectShow video source filter that talks to Logitech C930e camera bypassing newly introduced stuff is confirming that streams like 1920×1080@30 MJPG are still supported by the camera and are operational. That is, it is still possible to stream MJPG and H264 from USB web cameras, specifically at modes exceeding standard USB 2.0 bandwidth limit for raw video, and eliminating software compression:

DirectShowLogitechC930eVideoSource

Logitech C930e Live

This of course takes again exclusively control over the camera and prevents from sharing video feed as the update intended. However, the video itself is where is was.

Logitech C930 Running 1920x1080@30 MJPG

There is no public source and/or details on this filter because it’s sensitive to undocumented behavior of Media Foundation platform. Just as a demo, the DLLs are there: Win32, x64. (limited to Logitech Webcam C930e’s highest MJPG mode but basically the method could work for any MJPG camera, and C930e’s H264 too).

That is, if your application is broken by Windows 10 Anniversary Update because you simply assumed availability of specific modes, then there is a chance that update of the application to make it compatible to new platform design with FrameServer service could fix it. If you intentionally consumed compressed video for quality, rate and performance reasons then you’re in trouble and there is no really a solution from Microsoft is expected soon. Perhaps the best would be to not upgrade to Anniversary Update.

Logitech C930e camera and Media Foundation

Logitech’s C930e camera is the first one to be compliant with UVC 1.5 specification:

First 1080p HD webcam to support H.264 with Scalable Video Coding and UVC 1.5 encoding technology. […] The result is a smoother video stream in applications like Skype for Business and Microsoft® Lync® 2013.

More marketing information there at Logitech. More interesting is what the new capabilities look from API side programmatically. Additionally to well known Motion JPEG (FourCC MJPG) and YUY2 video, the camera delivers H.264 (FourCC H264) video.

Logitech C930e Webcam

Lync (Skype for Business) is presumably modified to accept that and it communicates to the camera using Media Foundation API.

The camera’s H.264 capabilities are accessible using both APIs, DirectShow and Media Foundation, and there is apparently a mess with driver versions and operating system versions as well. The best results are achieved with stock driver from Microsoft (without installing Logitech driver, this information is in good standing: “The only way I was able to get that stream under Windows 8.x was by NOT USING LOGITECH DRIVERS. This is a UVC 1.5 compatible camera and it will be configured automatically by the OS. With that driver (from Microsoft), use pin 1 (not 0) and you will get a ton of H264 formats.”).

A printout of DirectShow capabilities using DirectShowCaptureCapabilities is available here (note KS_H264VIDEOINFO structure). This time it is about what it looks when one’s doing Media Foundation.

As a Media Source, exposed are a few attributes and a great deal of media types (216 + 476), greater amount compared to DirectShow as it seems:

    • MF_DEVSOURCE_ATTRIBUTE_MEDIA_TYPE: 76 69 64 73 00 00 10 00 80 00 00 AA 00 38 9B 71 59 55 59 32 00 00 10 00 80 00 00 AA 00 38 9B 71
    • MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_SYMBOLIC_LINK: \\?\usb#vid_046d&pid_0843&mi_00#6&2314864d&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global (Type `VT_LPWSTR`)
    • MF_DEVSOURCE_ATTRIBUTE_FRIENDLY_NAME: Logitech Webcam C930e (Type `VT_LPWSTR`)
    • MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_CATEGORY: KSCATEGORY_VIDEO_CAMERA
    • MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE: MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_GUID
    • MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_HW_SOURCE: 4 (Type `VT_UI4`)
  • Characteristics: MFMEDIASOURCE_IS_LIVE | MFMEDIASOURCE_CAN_PAUSE
  • Stream 0: Default Selected, Identifier 0x0, Major Type MFMediaType_Video, 216 Media Types
  • Stream 1: Identifier 0x1, Major Type MFMediaType_Video, 476 Media Types

The H.264 formats are marked with subtypes of MFVideoFormat_H264 and MFVideoFormat_H264_ES. A raw print out is downloadable:

Specifically, it is interesting what are the attributes there since with Media Foundation it is tricky thing to find out quickly. The keys/identifiers are listed below.

Common

  • MF_MT_ALL_SAMPLES_INDEPENDENT
  • MF_MT_AM_FORMAT_TYPE
  • MF_MT_AVG_BITRATE
  • MF_MT_FIXED_SIZE_SAMPLES
  • MF_MT_FRAME_RATE
  • MF_MT_FRAME_RATE_RANGE_MAX
  • MF_MT_FRAME_RATE_RANGE_MIN
  • MF_MT_FRAME_SIZE
  • MF_MT_INTERLACE_MODE
  • MF_MT_MAJOR_TYPE
  • MF_MT_PIXEL_ASPECT_RATIO
  • MF_MT_SUBTYPE

MFVideoFormat_H264, MFVideoFormat_H264_ES

  • MF_MT_COMPRESSED
  • MF_MT_H264_CAPABILITIES
  • MF_MT_H264_MAX_CODEC_CONFIG_DELAY
  • MF_MT_H264_MAX_MB_PER_SEC
  • MF_MT_H264_RESOLUTION_SCALING
  • MF_MT_H264_SIMULCAST_SUPPORT
  • MF_MT_H264_SUPPORTED_RATE_CONTROL_MODES
  • MF_MT_H264_SUPPORTED_SLICE_MODES
  • MF_MT_H264_SUPPORTED_SYNC_FRAME_TYPES
  • MF_MT_H264_SUPPORTED_USAGES
  • MF_MT_H264_SVC_CAPABILITIES
  • MF_MT_VIDEO_LEVEL
  • MF_MT_VIDEO_PROFILE

MFVideoFormat_MJPG

  • MF_MT_SAMPLE_SIZE
  • MF_MT_VIDEO_CHROMA_SITING
  • MF_MT_VIDEO_LIGHTING
  • MF_MT_VIDEO_NOMINAL_RANGE
  • MF_MT_VIDEO_PRIMARIES
  • MF_MT_YUV_MATRIX

MFVideoFormat_YUY2

  • MF_MT_DEFAULT_STRIDE
  • MF_MT_SAMPLE_SIZE
  • MF_MT_VIDEO_CHROMA_SITING
  • MF_MT_VIDEO_LIGHTING
  • MF_MT_VIDEO_NOMINAL_RANGE
  • MF_MT_VIDEO_PRIMARIES
  • MF_MT_YUV_MATRIX