Number of streams served by IMFSourceReader interface

It looks confusing that IMFSourceReader interface does not offer a dedicated method to find out the number of streams behind it. There is a IMFMediaSource instance behind the reader, and its streams are available through IMFMediaSource::CreatePresentationDescriptor method and IMFPresentationDescriptor::GetStreamDescriptorCount method call.

I am under impression that source reader’s method just has to be there even though I am not seeing it looking at the list of methods. Okay, there are other methods, namely IMFSourceReader::GetStreamSelection method, which takes either ordinal stream index or an alias as the first argument, then returns MF_E_INVALIDSTREAMNUMBER if you run out of streams. However the problem is that this is associated with an internal exception, and I consider exceptions as exceptional conditions the code should not normally hit. I would expect to have a legal exception-free way to find out the number of streams. I am using debugger that breaks on exception or at least pollutes output log for no reason, I use other tools that intercept and log exceptions as something that needs attention – getting number of streams is nowhere near there.

Internal MF_E_INVALIDSTREAMNUMBER Exception

Even though it is not a real drawback of the API since it is still possible to get the data and the API acts as documented, I still think someone overlooked this and API like this should have have a normal method or argument to request number of streams explicitly. Or I am just not seeing it even though I am trying thoroughly.

Logitech camera video freezes in Skype after Windows 10 Anniversary Update

Windows 10 Anniversary Update broke Skype video conferencing in classic Skype desktop application for many users.

Video from Logitech cameras is freezing in Skype

There is a number of pieces of software running together to power video conferencing and standing in front of the end user application people don’t see whose fault is the broken video. There is Microsoft, Logitech, Skype (oh wait, Skype was acquired by Microsoft, so some think that that somehow changed the internals and the core of the existing at the time of the acquisition application).

This really confuses me. The Logitech cameras got a skype certificate. But then MS decides to make a change that renders those devices uselss? […]

The reality is that even though Anniversary Update does an extremely extravagant change in the API used by many applications, that it changed behavior of the applications, and that it constrained the capabilities of such nice cameras as Logitech’s C920, C930e and other. The reality is that video capture using these Logitech cameras is still well functioning in updated Windows. The update made highest modes unavailable, destroyed ability to capture M-JPEG and H.264 video,but the cameras still work in other simpler raw video modes smoothly.

So what’s the heck the problem with this videoconferencing? Skype’s implementation for video capture in desktop application has been terrible for years. There was noone there to ask questions why they support some devices and not others, what was the problem in their inability to work with certain cameras while other application work with them well etc. Instead, they hid the problem by offering certified cameras. This is leading to nowhere in long term, and this case of broken videoconferencing is the respective example. The video capture was not done right in first place and relied on things it should not have relied on.

Another question is that videoconferencing is one of the key Skype features. Insider builds of new Windows were available over months.

… this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year. We worked with partners to make sure their applications continued to function throughout this change…

Somehow it appeared that Skype’s guys were unable to respond timely to the coming update deployment, and ended up with a tornado of customer complaints. Who are the partners whom Microsoft worked with to make sure that the change is not fatal for their applications and hardware? Logitech and Skype don’t seem to be on that list.

German users discuss it here:

Wobei das auch ein perfektes Beispiel ist, wie ignorant Entwickler und User gegenüber den angebotenen Möglichkeiten von Microsoft sind.

Mir geht es nicht um den technischen Aspekt, der ist in der Tat diskutabel und auch wenn Microsoft da noble Absichten hat (gleichzeitiger Zugriff mehrerer Apps auf die Kamera etc. – steht ja alles im MS Foren Link), ist das nen breaking Change der viele eiskalt erwischt hat und wohl übers Ziel hinaus geschossen ist.

Nur da muss man sich fragen warum. Microsoft gibt zwar zu, dass die Änderung besser dokumentiert sein könnte, aber sie ist seit Januar! in den Insider Builds live. Im MS Forum beschweren sich User, dass tausende ihrer Kunden von einen Tag auf den anderen Probleme mit ihren Produkten haben aufgrund dieser Änderung und ich frag mich nur – was hat die Firma das halbe Jahr gemacht seit die Änderung zum testen verfügbar ist? So wichtig kann denen ihr Kundenstamm ja nicht sein… Jeder kann kostenfrei Insider Builds beziehen und Entwickler sollten die Möglichkeit nutzen ihre Produkte frühzeitig zu testen. Ich verstehe diese Ignoranz nicht – warten bis ein Update kommt und hoffen es hat sich nichts geändert. Früher war das leider die Regel und es gab bei jedem großen OS Release immer ne ganze Weile bis Softwarehersteller Inkompatibilitäten behoben haben, aber das sollte gerade durch das Insider Programm sehr zurückgegangen sein. Stattdessen warten Hersteller immer noch ab und betreiben lieber Flickschusterei statt pro aktiv Ihre Software im Vorhinein anzupassen, so dass ihre Kunden auch beim Start der neuen OS Version gleich ein funktionierendes System haben.

Im Forum sieht man auch wie redebereit Microsoft ist und wie kurzfristig auf das Userfeedback eingegangen wird und dementsprechend zügig auch ein wenig ihrerseits von den Einschränkungen des System zurückgenommen werden (MJPEG wird im ersten Schritt wieder zugelassen). Aber die Diskussion ist nen halbes Jahr zu spät, nicht seitens Microsoft sondern auf der Seite der entsprechenden Entwickler die nicht die angebotenen Möglichkeiten genutzt haben, frühzeitig die Änderungen zu testen.

OK, Skype is acquisition, what about Skype for Business (previously known as Lync)?

… Optimized for Skype for Business, the C930e Webcam supports H.264 with Scalable Video Coding and UVC 1.5 encoding to minimize its dependence on computer and network resources.

I suppose Lync team is also not on the mentioned list of partners. With some knowledge of Lync’s internals the Anniversary Update is likely to block the optimization cited above. As a video capture source, Logitech C930e cameras no longer offer H.264 video and Skype for Business is unlikely to be able to utilize it – the camera is likely to operate in fallback mode, just as other cameras. Perhaps someone from Lync should have mentioned that in January or earlier. Or maybe they did it, who knows.

Long story short, supposed Skype update will fix freezing soon (also good news they are delivering it not using Windows Update, hence fix will come faster), and then Media Foundation team will provide solution that restores H.264 optimizations later in a few months.

Anniversary Webcam Saga: It’s clear who’s guilty, now what to do? (Updated)

As new and new people discover the Windows 10 Anniversary Update breaking changes (expectedly running mad), let’s reiterate the possible solutions:

  1. You don’t like the idea that video sharing service adds latency, and adds man-in-the-middle access/spying over a video feed.
    See #6 below.
  2. You are consuming raw video from camera using one of the uncompressed modes within USB 2.0 bandwidth.
    You are likely to be not affected by the changes.
  3. You are consuming raw video from camera but resolution/rate combination makes it unable to capture raw video, so you captured M-JPEG instead and decoded that, via DirectShow API.
    It is no longer possible, but you can use Media Foundation API instead. Or someone will develop a wrapper that re-exposes Media Foundation captured video via DirectShow.
  4. Same as #3 above but via Media Foundation API.
    You have the option to consume already decoded video, new subsystem will automatically capture M-JPEG and decode into NV12.
  5. You take advantage of compressed format of video captured (DirectShow or Media Foundation) so that you don’t need compress it for storage or network transmission purposes.
    Compressed captured video is no longer available, see #6 below.
  6. You take advantage of H.264 video capture offered by UVC 1.5 device, including fine tuning of hardware H.264 compression.
    Just as in #1 and #5 above, you are in trouble. Windows Camera Frame Server no longer offers access to such video feed. You need a hack (yes, it’s confirmed to be possible) that restores original behavior of video capture hardware.

These and other reasons related to the fact that applications no longer talk to real capture device, but rather a Frame Server Client that proxies a web camera, will possibly require that video capture applications are updated in order to work well in new version of the operating system.

It is unclear if and how Microsoft and Media Foundation team will respond to customer pain voices. First, it looked as a bug and one could expect a response and fix. But with the information from Windows Camera Team it looks completely different. No, they did not accidentally break it up – it was a planned change. Then they connected new behavior with new Microsoft Products – new products rely on new behavior. Then they did a few nasty things, not just one: added proxy service, killed UVC related compression control over the device, reduced range of operation modes for DirectShow they look for ways to deprecate, conceptually removed compressed video capture modes. I think there is no way back – Windows Camera Frame Server is new reality. The best to expect is that some of the mentioned problems are relaxed by offering greater flexibility by the platform. Maybe they will add some sort of exclusive modes for video capture or “professional” hardware which offers more through the API. In any event these changes are unlikely to appear soon, as they will go through the full cycle of development and take months to get delivered. Public pressure might force that to appear rather earlier than later, but I don’t think it is what is going to happen.

16 Aug update: Windows Camera Team reported that they see customer pain and will do something to ease it shortly. As I see it, they will address scenarios #3, #4, #5 above, for MJPG video, to allow compressed formats pass Frame Server so that users could consume them from their applications and Frame Server would be able to release frames not just after decoder, but also before the shared decoder. Also as use of H.264 is limited, it might be not included into hotfix at all, or will be included much later being given more serious consideration (which might end up as dropped support in DirectShow and something new introduced for Media Foundation).

19 Aug update: Someone took time to locate a registry value. User WithinRafael on MSDN Forums:

Try opening up

HKLM\SOFTWARE\Microsoft\Windows Media Foundation\Platform (32- and 64-bit OS)
HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows Media Foundation\Platform (64-bit OS)

and add a DWORD value with name EnableFrameServerMode. Set its value to 0 and try again.

Put a sticky note on your monitor to revisit this if/when Microsoft issues a fix.

Also:

Untitled

Video Capture in Windows 10 Anniversary Update Again: MJPG is still here but hidden by new Frame Server thing

The problem with video capture issues looks more or less clear.

As explained by Mike M here, the breaking changes in Windows 10 Anniversary Update are caused by intentional redesign of the platform that enable shared access to video capture devices.

… it is now possible for multiple applications to access the camera in ways that weren’t possible before. It was important for us to enable concurrent camera access, so Windows Hello, Microsoft Hololens and other products and features could reliably assume that the camera would be available at any given time, regardless of what other applications may be accessing it.

Originally video capture application were highly performance sensitive due to insufficient horsepower of computers overall, and sharing of video capture sessions between the applications was not on agenda. Then Microsoft hibernated for over a decade and did not do updates to the platform to follow software and hardware trends. Those needing camera in 2+ applications had to use third party camera splitting software. Time has come to include video sharing to the platform and… that washed away support for compressed video formats. If camera is shared, who is going to decode video into presentable format? Guys at Microsoft decided that they will, that is it’s now “decode then share between applications” scenario.

When an application runs video capture session, Windows 10 Anniversary Update now runs the actual session in a service process. A new Windows Camera Frame Server service is responsible to acquire video, decode and distribute it.

Windows Camera Frame Server Service

Applications access FrameServer service with the help of FSClient.dll connecting to shared service which runs actual session.

I am not sure how sharing works exactly, but I was unable to start two TopoEdit instances doing video capture from the same camera. Presumably, default behavior is still imitating exclusive use of hardware and possibly priority clients (like mentioned Windows Hello) have new ways to take over video capture device on demand, or we will see new functionality with respective SDK/documentation update.

Applications now – as it is assumed from the description – get the only option to communicate to FrameServer service and not the video capture source directly. On that way, formats like MJPG and H264 are lost.

As recent comments indicate this to be a well planned and scheduled scenario, it looks unlikely that things are going to change. It was decided that there is no exclusive mode video capture, just shared. Developers are to wait for possibly changed attitude and something similar to WASAPI exclusive low-latency mode for those specific application which need it.

So yes, MJPEG and H.264 being decoded / filtered out is the result of a set of features we needed to implement, and this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year. We worked with partners to make sure their applications continued to function throughout this change, but we have done a poor job communicating this change out to you guys. We dropped the ball on that front, so I’d like to offer my apologies to you all.

A small relief is that they restructured the platform and not dropped the support for MJPG and H264 in first place. Okay, there is no formal access to compressed streams using standard API but a stab into doing it undocumented way shows that all the gear remains in place.

A small proof-of-concept DirectShow video source filter that talks to Logitech C930e camera bypassing newly introduced stuff is confirming that streams like 1920×1080@30 MJPG are still supported by the camera and are operational. That is, it is still possible to stream MJPG and H264 from USB web cameras, specifically at modes exceeding standard USB 2.0 bandwidth limit for raw video, and eliminating software compression:

DirectShowLogitechC930eVideoSource

Logitech C930e Live

This of course takes again exclusively control over the camera and prevents from sharing video feed as the update intended. However, the video itself is where is was.

Logitech C930 Running 1920x1080@30 MJPG

There is no public source and/or details on this filter because it’s sensitive to undocumented behavior of Media Foundation platform. Just as a demo, the DLLs are there: Win32, x64. (limited to Logitech Webcam C930e’s highest MJPG mode but basically the method could work for any MJPG camera, and C930e’s H264 too).

That is, if your application is broken by Windows 10 Anniversary Update because you simply assumed availability of specific modes, then there is a chance that update of the application to make it compatible to new platform design with FrameServer service could fix it. If you intentionally consumed compressed video for quality, rate and performance reasons then you’re in trouble and there is no really a solution from Microsoft is expected soon. Perhaps the best would be to not upgrade to Anniversary Update.

Comment on Video Capture Issues with Windows 10 Anniversary Update

There is a comment from MSFT’s Mike M on MSDN Forums on recent issue with compressed video capture. I am pulling it out completely as a quote below:

I’d like to start off by providing you guys a little more context on the behavior you’re encountering.

One of the main reasons that Windows is decoding MJPEG for your applications is because of performance. With the Anniversary Update to Windows 10, it is now possible for multiple applications to access the camera in ways that weren’t possible before. It was important for us to enable concurrent camera access, so Windows Hello, Microsoft Hololens and other products and features could reliably assume that the camera would be available at any given time, regardless of what other applications may be accessing it. One of the reasons this led to the MJPEG decoding is because we wanted to prevent multiple applications from decoding the same stream at the same time, which would be a duplicated effort and thus an unnecessary performance hit. This can be even more noticeable or perhaps trigger error cases on in-market devices with a hardware decoder which may be limited on how many decodes can take place simultaneously. We wanted to prevent applications from unknowingly degrading the user experience due to a platform change.

The reasoning for H.264 being decoded can get a little more complicated (and I’m just learning the details myself as I talk to other members of the team), but the basics revolve around how H.264 allows for encoding parameters to be changed on the camera directly, and how in a situation where multiple applications are making use of this control path, they could interfere with each other. Regarding Roman’s concerns about Lync: both Lync and Skype are partner teams, and we stay in touch throughout the development process, so the camera functionality in those applications will continue to work.

So yes, MJPEG and H.264 being decoded / filtered out is the result of a set of features we needed to implement, and this behavior was planned, designed, tested, and flighted out to our partners and Windows Insiders around the end of January of this year.  We worked with partners to make sure their applications continued to function throughout this change, but we have done a poor job communicating this change out to you guys. We dropped the ball on that front, so I’d like to offer my apologies to you all. We’re working on getting better documentation out, to help answer any questions you may have. Of course, you can always reach out to us via these forums for specific issues, as we monitor them regularly, or file feedback using the Feedback Hub. We’re constantly collecting feedback on this and other issues, so we can better understand the impact on our application developers and customers. If you’re having issues adapting your application code to the NV12 / YUY2 media types, we’d like to support you through the changes you may need to make. To get you started, please refer to the documentation links in my previous post. If there are reasons why working with this format isn’t feasible for your project, please let me know, and I’ll present them to the rest of the team, to try and find the best solution for your case.

Dacuda and Stephan B, I’m curious about your specific situations, since you report that this change is breaking functionality for your customers. Are your customers using custom camera hardware? Is the set of supported cameras restricted by your applications? How do your applications deal with devices like the Surface Pro 4, Surface Book, or Dell Venue Pro, which wouldn’t offer the media types your applications are relying on?
I’d like to wrap up this wall of text by letting you know that your feedback here and through other channels is greatly appreciated and something that’s on our radar. We’re trying to look into what other options we can offer you to be able to improve on this for your (and our) customers, so stay tuned! I invite you to please subscribe to this thread (use the “Alert me” link at the top), and I’ll keep you guys updated on what we find. Thanks!

Basically, it’s bad news for those who consume compressed video from capture devices – the breaking change is intentional. Something is offered in exchange and I hope someone will present the platform changes in a friendly readable document. In particular, Microsoft seems to be adding VP8/9 video decoder and encoder in this new platform version (more later on that perhaps).

Understanding Your DirectShow Filter Graph

Many questions in DirectShow development are caused by lack of developer’s understanding what topology his code effectively built. Intelligent Connect and RenderXxx methods help adding and connecting filters and in the end a developer does not have a faintest idea what the pipeline looks like.

DirectShow API provides methods to enumerate filters, pins, connection and obtained detailed information about the filter graph. The API is well-documented. Then Windows SDK is shipped with GraphEdit which helps building graphs interactively. Ability to publish a graph on ROT and review it from GraphEdit is nothing but powerful. And then we have GraphStudioNext which makes everything even more convenient.

This does not seem sufficient and clear as many new questions and misunderstanding show that developers have false assumptions on graphs their applications use.

DirectShowSpy goes one step further with debugging options. With DirectShowSpy one can embed reviewing UI right into the developed application and either generate detailed textual description of filters, connections, media types as well as pass filter graph to GraphEdit/GraphStudioNext for interactive review with visualized topology. No excuses left any longer for misunderstanding built topologies.

Steps below explain in detail how to visualize your application DirectShow filter graph and generate a textual report on graph details.

1. For starters, one needs to intall DirectShowSpy in target system. Standard installation is mentioned in original post.

  • It is necessary that DirectShowSpy of correct/matching bitness is installed. 32-bit applications use 32-bit DirectShowSpy and 64-bit applications – 64-bit DirectShowSpy. .NET applications built as “Any CPU” are effectively either 32 or 64 bit processes and respectively need a matching spy as well.
  • To cut long story short, simply download DirectShow*.* from Toolbox and use DirectShowSpy-Win32-reg-ui.bat or DirectShowSpy-x64-reg-ui.bat to pop up registration UI. You need local administrator privileges for the registration step (or spy is usable through COM otherwise but it’s beyond scope of this post).

2. DirectShowSpy’s FilterGraphHelper object (already mentioned earlier) offers DoPropertyFrameModal method to pop up diagnostic UI. The helper needs prior initialization with either graph, filter or pin interface. C++ code snippet:

#import "libid:B9EC374B-834B-4DA9-BFB5-C1872CE736FF" raw_interfaces_only // AlaxInfoDirectShowSpy
// ...
CComPtr<IFilterGraph2> pFilterGraph;
// ...
CComPtr<AlaxInfoDirectShowSpy::IFilterGraphHelper> pFilterGraphHelper;
ATLENSURE_SUCCEEDED(pFilterGraphHelper.CoCreateInstance(__uuidof(AlaxInfoDirectShowSpy::FilterGraphHelper)));
ATLENSURE_SUCCEEDED(pFilterGraphHelper->put_FilterGraph(pFilterGraph));
ATLENSURE_SUCCEEDED(pFilterGraphHelper->DoPropertyFrameModal(NULL));

C# code snippet:

IFilterGraph2 graph = new FilterGraph() as IFilterGraph2;
// ...
FilterGraphHelper helper = new FilterGraphHelper();
helper.FilterGraph = graph;
helper.DoPropertyFrameModal(0);

Downloadable sample projects (FilterGraphHelperDialog for C# and FilterGraphHelperDialog2 for C++) are available in Subversion repository or Trac.

3. DoPropertyFrameModal methods opens a window (it’s argument is parent window handle, optional) with details about the graph, including copyable diagnostic text, filters and their property pages all gathered in single window.

FilterGraphHelper.DoPropertyFrameModal UI

NOTE: With root tree element “Filters” selected, the right-side pane contains the text that provides filter graph description (see image above)!

4. Additionally, it is possible to launch GraphEdit/GraphStudioNext with a hotkey and open – through ROT – the graph visually.

FilterGraphHelper.DoPropertyFrameModal UI (Actions)

Remote Graph in GraphStudioNext

This requires that Windows SDK proppage.dll is available. It is normally registered with Windows SDK, and otherwise can be copied from SDK into target system and COM-registered using regsvr32. Or copied into the folder of DirectShowSpy in which case DirectShowSpy-Win32-reg-ui.bat (see item 1 above) file will see it and offer additional property page for registration.

5. When no longer needed, DirectShowSpy can be removed from system using the batch file mentioned above in item 1.

Whatever debugging you do with DirectShow filter graph, you need a complete understanding what filter graph you deal with. If you want to provide additional information to certain DirectShow related question, a copy/pasted diagnostic information needs to be attached to such question so that others understand what you are dealing with exactly.

Video Capture Issues with Windows 10 Anniversary Update

Windows 10 Anniversary Update brought a breaking change that removed, in many cases, hardware compressed video formats from video capture APIs, even though the devices themselves are known to have respective capabilities.

The magic of Logitech C930e camera is also lost since it is no longer available for video capture in hardware compressed H.264.

Presumably an issue in middle layer, such as Kernel Streaming, it is reflected by missing support for some of video device formats, esp. Motion JPEG (FourCC MJPG) and H.264 (FourCC H264). The depth of the issue also results in effect that capabilities are equally missing from both DirectShow and Media Foundation APIs.

Unless someone wants to develop a driver talking to hardware directly bypassing Kernel Streaming subsystem (such as, for example, “DeckLink Video Capture” for Blackmagic Design devices leveraging SDK and not the complementary WDM driver), it seems that the only solution is to wait for the fix from Microsoft. Rather sooner than later it will be here taking into consideration the scope of the problem.

Rather, try to stop automatic updates if video capture is important for you.

See also:

UPDATE: Mike from MSFT commented in this thread:

Hey guys, Mike from the Camera team here. We saw this concern pop up for the first time a couple of days ago, and we’d like to understand exactly what difficulties you are all facing. There are two items that I would like the discussion to focus on.

Firstly, the media types supported on a given machine vary depending on the capabilities of the camera device you’re using. There is no guarantee that a media type such as MJPEG will be supported on all cameras (for example, the Surface Pro 4 / Surface Book cameras do not support it). This means that, by taking a hard dependency on it being available, you’re limiting the portability of your application to the set of cameras that do offer that media type. What applications should do instead, is query the available formats (see: IMFSourceReader::GetNativeMediaType for MediaFoundation, IAMStreamConfig::GetStreamCaps or the IPin::EnumMediaTypes article for DirectShow), and make a selection based on the one that best suits your needs or capabilities.

Secondly, we are expecting that almost all clients using MJPEG frames will be decoding the stream as one of the first steps in their pipeline. In order for this to be done in the most efficient way, with the smallest impact to overall system performance, we want to offload that piece of the process to be taken care of by the platform. This means your application should be able to consume the uncompressed frames, which will have been decoded to NV12 for MediaFoundation applications, and YUY2 for DirectShow applications.

If there are any other issues that you’d like to bring to our attention, please do so. We’d love to understand how we can best help you through this transition, and we’ll be considering your feedback for future planning. Thanks!