According to CSI: Crime Scene Investigation, Season 06 Episode 13 (17:04), investigators are using snake scope camera with DirectShow AMCAP tool to present the captured image:

It is really great news that Windows 7 offers quite some goodies for MPEG-4 support right out of the box, but some things appear to be made not so accurate.

The component responsible for video decoding is Microsoft DTV-DVD Video Decoder (also known as Microsoft MPEG-2 Video Decoder), available with DirectShow interface. It offers video decoding capabilities for video resolutions up to 1920×1088 (Full HD), including DXVA version 2 mode. The decoder is a “preferred decoder” and is given a first chance to be taken when it comes for Intelligent Connect to render a video stream.

Technology went ahead though, stepped behind 1920×1088 in particular. Microsoft DTV-DVD Video Decoder DirectShow filter still remains the first take in H.264 video decoding, and what however is taking place – the decoder accepts the format (media type) which is clearly cannot decode, neither according to specification, nor in real life. As soon as pin connection succeeds, DirectShow is not trying to look for another decoder, which can truly decode the feed. Result: failure to decode video.

The problem takes place in a quite surprising way. Given 5 megapixel video (2560×1592, H.264, Baseline profile, from RTSP-enabled IP camera), the decoder advertises 720×480 NV12 output media type. While consuming input data, the decoder outputs nothing and keeps video window black.

It does not issue any filter graph events, no. The filter is persistent and works hard in assumption that a new piece of knowledge will come from Mars and provide new capabilities to start decoding video, or filter’s input will give up and provide a new SPS to lower resolution down to something less complex.

Well then, so is this advanced technology yet to be discovered by science? It appears that no. CoreAVC Video Decoder, applied to the same H.264 video decodes it well and nice:

Update 2011-09-26: According to Microsoft classification, this is a bug on a released product. That is, if anyone is experiencing severe issues as a result of this behavior, one might need to contact Microsoft Support directly in search of a solution.

Actually, as resolutions over Full HD (1920×1080) are not so much common at the moment, the issue side effects are not yet likely. Still moving on with time, I think the issues are likely to come up, without being severe though.

Crime

An application which builds a DirectShow graph unexpectedly started failing with VFW_E_NOT_CONNECTED (0×80040209) error code.

Scene

The problem takes place during DirectShow graph building, yet in stopped state. Specific call which appeared to be giving out the error in first place appears to be EVR input pin’s IPin::ConnectionMediaType, and the problem is also specific to Enhanced video Renderer (Windows 7, but not necessarily only this version).

Investigation

The problem does not appear to be persistent. On the contrary, it is taking place for just a few milliseconds after pin connection. After the problem is gone, it does not seem to ever come up again unless the filter graph is built again from the beginning.

EVR pin connection is always reporting success, so the following error code stating VFW_E_NOT_CONNECTED “The operation cannot be performed because the pins are not connected.” goes against documented behavior, and is thus a bug.

Depending on time between pin connection and media type polling, the call can reach EVR:

• before it starts showing the problem – stage A
• at the time the call fails – stage B
• after the failure time interval, when the call is successful from then on – stage C

Thus, the problem is limited to specific use cases:

• the application should care about media type on EVR input
• unexpected failure takes place when the call reaches in stage B
• also found: the clipping window for the EVR has to belong to a non-primary monitor

If an application keep polling for media type in a loop, the result may be about the following:

UINT nStageA = 0, nStageB = 0, nStageC = 0;
// [...]
for(; ; )
{
    AM_MEDIA_TYPE MediaType;
    ZeroMemory(&MediaType, sizeof MediaType);
    const HRESULT nConnectionMediaTypeResult = pInputPin->ConnectionMediaType(&MediaType);
    if(SUCCEEDED(nConnectionMediaTypeResult))
    {
        if(nStageB)
        {
            nStageC++;
            break;
        } else
            nStageA++;
    } else
    {
        ATLASSERT(nConnectionMediaTypeResult == VFW_E_NOT_CONNECTED);
        nStageB++;
    }
    CoTaskMemFree(MediaType.pbFormat);
}
// [...]
CString sMessage;
sMessage.Format(_T("Bingo!\r\n\r\n") _T("nStageA %d, nStageB %d - 0x%08x, nStageC %d\n"), nStageA, nStageB, nResult, nStageC);
AtlMessageBox(m_hWnd, (LPCTSTR) sMessage, _T("Result"), MB_ICONERROR);

Workaround

An obvious straightforward workaround is to follow EVR connection with a wait for Stage B to pass, or timeout – whichever takes place first.

Also, vote for the bug on Microsoft Connect.

More Details

Video renderer filter are notorious for re-agreeing media type and being fretful as for memory allocators and media types (for a good reason though!). So it makes sense to suggest that the problem takes place when the filter is doing something related, such as it starts background activity immediately after connection in order to discover upstream peer capabilities.

In order to possibly get details on this, it is possible to raise an exception as soon as Stage B is detected and take a look at thread states using a debugger. Indeed, on of the background threads is engaged in EVR reconnection activity:

Yes it does the reconnection, but nevertheless it is expected to do the things undercover and transparently, it still allows a failure on the outer API.

     evr.dll!GetSourceRectFromMediaType() + 0x37 bytes    
     evr.dll!CEVRInputPin::CheckMediaType() + 0x81 bytes    
     evr.dll!CBasePin::ReceiveConnection() + 0x61 bytes    
     evr.dll!CEVRInputPin::ReceiveConnection() + 0x1fc2d bytes    
     quartz.dll!CBasePin::AttemptConnection() - 0x21 bytes    
     quartz.dll!CBasePin::TryMediaTypes() + 0x60 bytes    
     quartz.dll!CBasePin::AgreeMediaType() + 0x54 bytes    
     quartz.dll!CBasePin::Connect() + 0x46 bytes    
     quartz.dll!CFilterGraph::ConnectDirectInternal() + 0x83 bytes    
     quartz.dll!CFilterGraph::ConnectRecursively() + 0x2c bytes    
     quartz.dll!CFilterGraph::ConnectInternal() + 0xde bytes    
     quartz.dll!CFilterGraph::Connect() + 0x17 bytes    
     quartz.dll!CFGControl::WorkerDisplayChanged() + 0xf1 bytes    
     quartz.dll!CFGControl::CGraphWindow::OnReceiveMessage() + 0x2e2a bytes    
>    quartz.dll!WndProc() + 0x3e bytes    
     user32.dll!_InternalCallWinProc@20() + 0x23 bytes    
     user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes    
     user32.dll!_DispatchMessageWorker@8() + 0xed bytes    
     user32.dll!_DispatchMessageW@4() + 0xf bytes    
     quartz.dll!ObjectThread() + 0x65 bytes

A test Visual C++ .NET 2010 application is available from SVN. The code requires a media file, and refers to 352×288 I420.avi, which is included into ZIP file attached to MS Connect Feedback.

A customer complained on video playback problems which had symptoms of software compatibility issue. At certain operation modes software froze leaving the only option to kill process and restart.

I appeared that the system has a third party DLL installed, a new hero name’s – picapmk4.dll. The DLL registers itself as Video for Windows codec and is clearly indicating capabilities it does not really support.

The DLL itself has no indication of its source, no valid version info. Instead it has GPL v2 embedded, XviD logo and about box resource template showing up as “PMK4 Video Codec”.

Looks like crapware, which is hostile to DirectShow environment, with possibly GPL compliance issues.

Two widely popular applications, which use DirectShow API appeared to be locked in a combat: Google video chat is installing a plugin which registers artificial video capture sources “Google Camera Adapter 0″ and “Google Camera Adapter 1″.

As the application does not announce any integration capabilities and is only using video for internal purposes for in-browser video, the approach with system-wide device registration is, well, questionable.

No need to mention, that devices are not operational: they are giving ERROR_DLL_INIT_FAILED (Error 0x8007045A) as soon as you try to instantiate a filter in a DirectShow video capture enabled application.

In-browser Google Video Chat itself does work and shows video from “real” DirectShow devices, including such artificial as created by IP Video Source tool. Google developers prefer still to take data off the device and as soon as possible leave DirectShow domain, as their graph looks like:

Video Capture Source -> Smart Tee Filter -> Sample Grabber Filter -> Null Renderer Filter

While Google plugin developers might need some interprocess synchronization, as the plugin runs in a child “plugin container” process (plugin-container.exe), introduction of virtual video capture devices is not necessary, but is a plain garbage for the operating system and DirectShow environment.

For some reason, Google developers decided to introduce a special thing: they mirror captured video left-to right when it comes to showing picture capture locally. Well, quite possibly it starts a new era in video conferencing: one will not see a direct copy of video sent remotely, but instead he will be given a mirrored thing.

The curious part is, however, the effect the broken devices introduce to another popular software – Skype.

As Skype is capable of video capturing from a DirectShow device, for video-enabled conferencing, it enumerates the available devices and definitely gets in touch with the aliens from Google planet, which are already known to be broken junk.

The outcome appears to be subject to chances, there are two possible outcomes and only one is such that is more or less good for a user:

• either, Skype shows other available devices and ignores Google adapters (while basically any reasonable DirectShow video capture device would still list those devices – after all they are registered as existing!)
• or, Skype stumbles upon a broken thing, stops enumeration and show no video capture devices available at all! Since Skype does enumeration only once on startup, if it fails to discover devices a user needs to close Skype completely and start it again for another try with video hardware.

It appears that both applications have severe issues.

One of the popular questions asked in relation to DirectShow filters, and also a popular task is to modify video stream while in playback. There are various reasons one might need to do it, including keying video to replace color/range, or apply text or time code on top of video, including while re-compressing footage, adjust brightness or contrast.

DirectShow BaseClasses/SDK include samples (in most cases EzRGB24 sample is the best to start for a video effect, also demonstrates use of private interface on a filter) to quick-start with the task without getting too much into detail and once this part is done, next step is to integrate filter into playback, connect it with other filters.

As a result of DircetShowflexibility, there are ways to do things not so good, while still being under impression of keeping right track.

File playback is one of the basic tasks with DirectShow. To play a file, one creates a filter graph using powerful helpers provided by Filter Graph Manager object. It might be a actually a single call IGraphBuilder::RenderFile which takes all the complexity of finding matching filters, connecting them together, dealing with splitters and renderers, video and audio. A single call resolves the problems in a convenient way – easy.

Still a simple thing of inserting your own video transformation filters breaks the simplicity. One needs to build the graph partilly, insert the effect and complete building, or build the thing and break in with a new filter. How to find insertion point? Will the other filters like intrusion? Different file types and formats.

There is an easy and elegant solution to pre-add your own effect filter into graph and start rendering a file from there. Sounds reasonable and sometimes works. The problem is however that it does not work always, and you never know when it lets you down. The graph might be build and the effect filter is never taken and is left orphaned aside of playback pipeline.

Reliable graph building assumes you are in control over building steps and allow only the level of flexibility required to connect and build parts – and this is where Intelligent Connect is still a powerful helper. With an effect, the parts are “before the effect” and “after the effect”. RenderFile is no longer an option, and one has to dive deeper into graph building API.

First of all, the building starts with the file itself: unlike RenderFile, IGraphBuilder::AddSourceFilter method adds just the first filter for a given file/format. It stops there and lets caller continue building as required. At this point, it is the right time to manually add effect filter with IFilterGraph::AddFilter (IGraphBuilder is inherited from IFilterGraph and exposes the same method).

Having both ends in the graph for the “before the effect” part, intelligent connect can be used to connect and add filters required to make the connection. For an arbitrary file format, the task may be not trivial: depending on format and installed software components, the chain may look rather different. First, some filters combine stream splitting capability with immediate access to file (or another type of resource), others rely on joint operation of Async File Source filter with a format-dependent splitter filter. Some expose elementary stream pins immediately, some provide transport stream pin.

There may be a few approaches as for addressing pin connection task (see also General Graph-Building Techniques on MSDN). Straightforwardly, one might want to call IGraphBuilder::Connect and take advantage of intelligent connect. Before this can be done, however it takes caller to select a pin of the obtained source filter to start from. There might be a few pins, including those exposing video formats, non-video formats and pre-split formats where video is behind depacketizing (demultiplexing). Considering variety of formats and support, it might make sense to make a first attempt finding a video pin (by enumerating pins and their media types, looking and AM_MEDIA_TYPE::majortype and comparing to MEDIATYPE_Video) and, if not found, taking a first output pin of any type, or going through ping trying to connect first one which succeeds in connection.

An alternate approach is to take advantage of a helper object: Capture Graph Builder. While originally it is intended to help capture graph building, it contains useful methods for general building and connecting pins. It does not own a graph itself: it is designed to operate on top of existing graph, provided by its owner. So one need to provide its existing graph and call helper methods for easy graph building. One of the methods is ICaptureGraphBuilder2::RenderStream, which connects pins of given filters. Unlike API discussed earlier, it takes filter interfaces on input and will be responsible for finding proper pins itself, which might be a good idea if you don’t want to bother yourself doing it. To specify the requested task, it takes media type argument, which in this case might be video or, if fails, any type provided that video media type will still be anyway checked on input of effect filter.

Once the part “before the effect” is done, the other part may be completed as simple as calling IGraphBuilder::Render on the output pin of the effect. This will correspond to final step of original RenderFile execution.

A tiny Visual Studio 2010 C++ project illustrates discussed techniques and it available at SVN repository: RenderStreamTest01:

• for a given media file in line 109 the project will start graph building
• a suitable replacement for a video effect filter will be a Sample Grabber filter initialized with a video type (24-bit RGB, but the line 137 can be commented out)
• switch in line 147 switches between base Connect approach and Capture Graph Builder helper
• while message box is on the screen and also showing building status, the graph can be looked at using Graph Edit or similar tool, provided that DirectShow Spy is installed; alternatively you might want to put the graph onto ROT manually

The project also illustrates a solution for recent problems referencing Sample Grabber with new SDK. Sample Grabber was obsoleted and removed from Window SDK definition file (qedit.h). In order to resolved the problem without using an older version of SDK, the definitions might be imported from type the corresponding library and (apart from used as such) copied into the source code directly, as in lines 13-60.

See also on graph building:

• Basic DirectShow Tasks
• Intelligent Connect
• General Graph-Building Techniques
• Connect Filters
• Using Capture Graph Builder

This quick update for DirectShow Filter Graph Spy adds tracing for (deprecated) IAMFilterData interface, and also fixes problem with Media Player Classic – Home Cinema, when the player crashes in External Filters windows.

Partial Visual C++ .NET 2008 source code is available from SVN, release binary included (Win32, x64); installation instructions are in another post.

DirectShow Filter Graph Spy was updated to add new functionality: spying over another DirectShow object – Filter Mapper. Filter Mapper object is used for filter registration purposes and also internally by DirectShow’s Intelligent Connect. IFilterMapper2 interface traces provide detailed description on steps DirectShow takes in order to connect and/or render filter pins. For example when trying to render MEDIATYPE_Audio pin, Filter Mapper suggested “RDP DShow Redirection Filter” and, then, “Default DirectSound Device”:

FilterMapperSpy.h(186): CFilterMapperSpy::EnumMatchingFilters: nFlags 0x0, bExactMatch 0, nMinimalMerit 0x00200001, bInputNeeded 1, nInputTypeCount 1, pInputPinCategory NULL, bRender 1, bOutputNeeded 0, nOutputTypeCount 0, pOutputPinCategory NULL
FilterMapperSpy.h(191): CFilterMapperSpy::EnumMatchingFilters: nInputTypeIndex 0, MajorType {73647561-0000-0010-8000-00AA00389B71}, Subtype {00000000-0000-0000-0000-000000000000}
FilterMapperSpy.h(212): CFilterMapperSpy::EnumMatchingFilters: pMoniker @device:sw:{083863F1-70DE-11D0-BD40-00A0C911CE86}\{AB9D6472-752F-43F6-B29E-61207BDA8E06}
FilterMapperSpy.h(220): CFilterMapperSpy::EnumMatchingFilters: sFriendlyName "RDP DShow Redirection Filter", sDescription "", sDevicePath ""
FilterMapperSpy.h(212): CFilterMapperSpy::EnumMatchingFilters: pMoniker @device:cm:{E0F158E1-CB04-11D0-BD4E-00A0C911CE86}\Default DirectSound Device
FilterMapperSpy.h(220): CFilterMapperSpy::EnumMatchingFilters: sFriendlyName "Default DirectSound Device", sDescription "", sDevicePath ""

Important is that DirectShow’s internals are using the mapper through COM instantiation and this provides a safe method to override default behavior process wide in order to, for example, extend Intelligent Connect functionality onto custom, unregistered globally, private filters.

Also, the binary is renamed from FilterGraphSpy.dll to DirectShowSpy.dll.

Partial Visual C++ .NET 2008 source code is available from SVN, release binary included (Win32, x64); installation instructions are in another post.

The question is being asked from time to time. Everyone knows what is full screen video. Video renderers implement full screen capability since long ago through their IVideoWindow::put_FullScreenMode property, and even Filter Graph Manager exposes its own IVideoWindow interface to forward calls to filter’s implementation of IVideoWindow interface.

However, for Video Mixing Renderers, version 7 and 9, the preferred and recommended mode is windowless, where no IVideoWindow interface is available.

Note The IVMRWindowlessControl or IVMRWindowlessControl9 interface is now preferred over IVideoWindow. For more information, see Using Windowless Mode.

So in order to implement full screen mode it takes the application to attach Video Mixing Renderer filter to a private frameless window, to its entire client area and expand the window to entire monitor area.

The sample project FullScreenWindowlessVmrSample01 is illustrating this technique.

Main application dialog owns a special window to host renderer, implemented by CRendererDialog class.

class CRendererDialog :
    public CDialogImpl<CRendererDialog>
{
public:

    enum { IDD = IDD_RENDERER };

BEGIN_MSG_MAP_EX(CRendererDialog)
    MSG_WM_INITDIALOG(OnInitDialog)
    MSG_WM_DESTROY(OnDestroy)
    MSG_WM_ERASEBKGND(OnEraseBkgnd)
    MSG_WM_PAINT(OnPaint)
    MSG_WM_DISPLAYCHANGE(OnDisplayChange)
    MSG_WM_KEYDOWN(OnKeyDown)
    COMMAND_ID_HANDLER_EX(IDCANCEL, OnCancel)
END_MSG_MAP()

private:
    CMainDialog& m_Owner;
//...

The dialog template does not have a caption, it has no frame and has its window rectangle equal to client area. As a result, spanning the Video Mixing Renderer to the entire client area and proper positioning of the dialog window is showing video on the entire monitor.

ATLVERIFY(m_RendererDialog.Create(m_hWnd));
ATLVERIFY(m_RendererDialog.MoveWindow(MonitorData.m_Position));
m_RendererDialog.ShowWindow(SW_SHOWNORMAL);
// ...
ATLENSURE_SUCCEEDED(pVmrWindowlessControl->SetVideoClippingWindow(m_RendererDialog));
ATLENSURE_SUCCEEDED(pVmrWindowlessControl->SetVideoPosition(NULL, m_RendererDialog.GetVideoPosition()));

Visual C++ .NET 2008 source code is available from SVN, release binary included. The project is using ATL and WTL libraries.

From a conversation:

Q: I am using Web camera with DirectShow. Camera has only capture pin, so I am looking for the best way to switch resolution on the pin while graph is running. I would like to keep preview with smaller image size and when i would like to capture an image i would like to switch to full resolution. I have to use sample grabber callback since i need access to single image to process them. Is there a way to do that without stopping and starting a graph? This works but very slow between captures.

Is it possible to just reconnect the sample grabber only with new media changes and then resume the graph? Currently i just kill the graph and start it again, but it takes few seconds to do so, so I am looking the way to reduce that time.

There is no way to switch resolution on the running graph. There is a technique to dynamically start/stop individual filters and re-negotiate resolutions (media types), but it does not work for the majority of filters. Additionally to that Sample Grabber Filter cannot change resolution too, as it passes data through.

If you only have to use switched resolution for preview, you can use Geraint’s GMFBridge Toolkit to join two graphs, and a filter that changes resolution. Combining all that you will have a running capture graph that [also] renders video to a bridge sink. In the other graph you have another bridge sink that receives video from first graph and then you resize video to the resolution of interest already in the second graph. You can stop and reconfigure only second graph to update resolution and have first graph running and capturing. This is the best you can do, or just stop your single graph and change resolution this simple way.

So a solution, which is used by many, and I can recommend it too, is to use GMFBridge bridging. Additionally, you can find questions and answers on it on MSDN DirectShow Development Forum (search for “bridge” there). With a certain effort you can duplicate this with your own code but this is more or less ready to use solution and, again, the key advantage you have that you have two graphs which you can top independently.

Would it be possible to use smart tee as a splitter on the capture pin, and then use two sample grabbers on capture and preview, one with smaller resolution and the other one with higher one?

With a Smart Tee Filter you will still have 1 graph, so no individual resolution changes without stopping the graph. Additionally to that, Smart Tee Filter will deliver same frame on its output pins, so they will have to have one resolution and no resizing takes place inside.

Also, is it possible to run 2 graphs in the same time? Same device, 2 graphs, 2 sample grabbers.

Most likely no, for only one reason: you won’t be able to have two running filters for the same device, as source filter will exclusively lock the device. So capture filter will be a single filter. You can use Infinite Tee Pin Filter to split stream between 2+ processing lines. And you can use the same bridge to pass data into another graph for further processing.