As a part of American Kestrel Partnership project, Peregrine Fund put live cameras online to broadcast interior and exterior views of American Kestrel nest boxes in their campus in Boise, Idaho.

One of the nestboxes is not only populated, four eggs have been laid recently and hatching is expected soon. The live picture is made possible by putting IP cameras and connecting them with to Ustream.tv broadcasting service through Adobe FMLE software through my IP Video Source DirectShow driver for IP cameras (the IP camera video feed is streamed 24/7 through a virtual video device picked up by FMLE and re-encoded into uploadable stream forwarded to Ustream online service for further distribution).

For more information, see:

• American Kestrel Website
• News Release: The Peregrine Fund launches new project, website to conserve American Kestrels
• NPR news story about the project
• American Kestrel Partnership Project
• American Kestrel on Wikipedia
• The Peregrine Fund on Ustream.tv

Watch the pet on live cameras below:

North Nestbox – Outside

http://ustre.am/Evbs

Live stream videos at Ustream

North Nestbox – Inside

http://ustre.am/J6jD

Live broadcasting by Ustream

South Nestbox – Outside

http://ustre.am/JjP2

Streaming Live by Ustream

South Nestbox – Inside

http://ustre.am/JjP7

Streaming Live by Ustream

Direct3D based DirectShow video renderers – Video Mixing Renderer 9 and Enhanced Video Renderer – have been notoriously known for consuming resources in a way that you can run at most X simultaneously. There has been no comment published on the topic and questions (e.g. like this: How many VMR 9 can a PC support concurrently) remain unanswered for a long time. Video Mixing Renderer 7 was a good alternative for some time in past until it was cut down to be unable to support hardware scaling (thanks Microsoft!).

The trendy way to render video nowadays is using Enhanced Video Renderer, a Media Foundation subsystem with an interface into DirectShow to take over state of the art video rendering capabilities. So, how many EVRs one can run simultaneously? Chances are that it is less than one could suppose.

The interesting part is that there is no obvious evidence on type of resource running out, which causing next EVR instance to fail to run. And not even run, the failure seem to be coming up at an earlier stage of just connecting pins in stopped state. The failure might be accompanied with errors like E_INVALIDARG, ERROR_FILE_NOT_FOUND, E_UNEXPECTED. The actual limit appear to be loosely correlating to parameters of video output, such as resolution and bitness.

Desperately waiting for clarification, I am sharing the tool to estimate the limit:

• multiple EVR instances at once, hosted by multiple windows, which can be distributed across multiple monitors
• choices of resolutions and formats
• a double click on an individual renderer pops up property page set displaying effective frame rate
• 32- and 64-bit versions

A binary [Win32, x64] and partial Visual C++ .NET 2010 partial source code are available from SVN.

Unfortunately, AMAP_3D_TARGET appears to be useless. Internally, surface allocation comes up with the following capabilities for the surface: DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM | DDSCAPS_OFFSCREENPLAIN | DDSCAPS_3DDEVICE. Sadly, DirectDraw responds (might respond?) with E_NOTIMPL.

While one can advance one step further by removing DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM, another problem is hit that you are no longer able to create a surface with is both FourCC enabled and in the same time has 3D rendering capabilities. Remove your YUV four character code (such as NV12, YV12, YUY2 you would normally have as codec’s output) or fail miserably with DDERR_INVALIDPIXELFORMAT.

Back to the original problem, the AMAP_3D_TARGET flag failing to work makes it impossible to allocate 3D-enabled DirectDraw surface with DirectShow Video Mixing Renderer. While it might sound as deprecated technology, it is yet a supported Windows SDK component, the most efficient video rendering component, the least hardware sensitive video rendering component, and – the most immediately important – the only way an SDK I am working with can deliver its video overlay.

So, there is no easy way to request a 3D enabled surface through a custom allocator-presenter, with a hook in the IVMRSurfaceAllocator::AllocateSurface updating the allocation flags. It does not make much sense to replace the whole allocator-presenter either: while an earlier DirectShow/Windows SDK provides a decent base for custom allocator-presenter, it is going to eventually hit the same problems mentioned above. And you cannot blit from 3D surface into a plain offscreen surface either without getting E_FAIL.

What appears to still be possible is allocating an addition 3D enabled surface, such as within IVMRImagePresenter::StartPresenting and using it as a replacement surface within IVMRImagePresenter::PresentImage. Having a custom allocator-presenter receive video frame, you can blit the picture into 3D-enabled surface, do your thing and pass the updated VMRPRESENTATIONINFO to default allocator-presenter so that it presents the additional update surface, not the original one.

According to CSI: Crime Scene Investigation, Season 06 Episode 13 (17:04), investigators are using snake scope camera with DirectShow AMCAP tool to present the captured image:

It is really great news that Windows 7 offers quite some goodies for MPEG-4 support right out of the box, but some things appear to be made not so accurate.

The component responsible for video decoding is Microsoft DTV-DVD Video Decoder (also known as Microsoft MPEG-2 Video Decoder), available with DirectShow interface. It offers video decoding capabilities for video resolutions up to 1920×1088 (Full HD), including DXVA version 2 mode. The decoder is a “preferred decoder” and is given a first chance to be taken when it comes for Intelligent Connect to render a video stream.

Technology went ahead though, stepped behind 1920×1088 in particular. Microsoft DTV-DVD Video Decoder DirectShow filter still remains the first take in H.264 video decoding, and what however is taking place – the decoder accepts the format (media type) which is clearly cannot decode, neither according to specification, nor in real life. As soon as pin connection succeeds, DirectShow is not trying to look for another decoder, which can truly decode the feed. Result: failure to decode video.

The problem takes place in a quite surprising way. Given 5 megapixel video (2560×1592, H.264, Baseline profile, from RTSP-enabled IP camera), the decoder advertises 720×480 NV12 output media type. While consuming input data, the decoder outputs nothing and keeps video window black.

It does not issue any filter graph events, no. The filter is persistent and works hard in assumption that a new piece of knowledge will come from Mars and provide new capabilities to start decoding video, or filter’s input will give up and provide a new SPS to lower resolution down to something less complex.

Well then, so is this advanced technology yet to be discovered by science? It appears that no. CoreAVC Video Decoder, applied to the same H.264 video decodes it well and nice:

Update 2011-09-26: According to Microsoft classification, this is a bug on a released product. That is, if anyone is experiencing severe issues as a result of this behavior, one might need to contact Microsoft Support directly in search of a solution.

Actually, as resolutions over Full HD (1920×1080) are not so much common at the moment, the issue side effects are not yet likely. Still moving on with time, I think the issues are likely to come up, without being severe though.

Crime

An application which builds a DirectShow graph unexpectedly started failing with VFW_E_NOT_CONNECTED (0×80040209) error code.

Scene

The problem takes place during DirectShow graph building, yet in stopped state. Specific call which appeared to be giving out the error in first place appears to be EVR input pin’s IPin::ConnectionMediaType, and the problem is also specific to Enhanced video Renderer (Windows 7, but not necessarily only this version).

Investigation

The problem does not appear to be persistent. On the contrary, it is taking place for just a few milliseconds after pin connection. After the problem is gone, it does not seem to ever come up again unless the filter graph is built again from the beginning.

EVR pin connection is always reporting success, so the following error code stating VFW_E_NOT_CONNECTED “The operation cannot be performed because the pins are not connected.” goes against documented behavior, and is thus a bug.

Depending on time between pin connection and media type polling, the call can reach EVR:

• before it starts showing the problem – stage A
• at the time the call fails – stage B
• after the failure time interval, when the call is successful from then on – stage C

Thus, the problem is limited to specific use cases:

• the application should care about media type on EVR input
• unexpected failure takes place when the call reaches in stage B
• also found: the clipping window for the EVR has to belong to a non-primary monitor

If an application keep polling for media type in a loop, the result may be about the following:

UINT nStageA = 0, nStageB = 0, nStageC = 0;
// [...]
for(; ; )
{
    AM_MEDIA_TYPE MediaType;
    ZeroMemory(&MediaType, sizeof MediaType);
    const HRESULT nConnectionMediaTypeResult = pInputPin->ConnectionMediaType(&MediaType);
    if(SUCCEEDED(nConnectionMediaTypeResult))
    {
        if(nStageB)
        {
            nStageC++;
            break;
        } else
            nStageA++;
    } else
    {
        ATLASSERT(nConnectionMediaTypeResult == VFW_E_NOT_CONNECTED);
        nStageB++;
    }
    CoTaskMemFree(MediaType.pbFormat);
}
// [...]
CString sMessage;
sMessage.Format(_T("Bingo!\r\n\r\n") _T("nStageA %d, nStageB %d - 0x%08x, nStageC %d\n"), nStageA, nStageB, nResult, nStageC);
AtlMessageBox(m_hWnd, (LPCTSTR) sMessage, _T("Result"), MB_ICONERROR);

Workaround

An obvious straightforward workaround is to follow EVR connection with a wait for Stage B to pass, or timeout – whichever takes place first.

Also, vote for the bug on Microsoft Connect.

More Details

Video renderer filter are notorious for re-agreeing media type and being fretful as for memory allocators and media types (for a good reason though!). So it makes sense to suggest that the problem takes place when the filter is doing something related, such as it starts background activity immediately after connection in order to discover upstream peer capabilities.

In order to possibly get details on this, it is possible to raise an exception as soon as Stage B is detected and take a look at thread states using a debugger. Indeed, on of the background threads is engaged in EVR reconnection activity:

Yes it does the reconnection, but nevertheless it is expected to do the things undercover and transparently, it still allows a failure on the outer API.

     evr.dll!GetSourceRectFromMediaType() + 0x37 bytes    
     evr.dll!CEVRInputPin::CheckMediaType() + 0x81 bytes    
     evr.dll!CBasePin::ReceiveConnection() + 0x61 bytes    
     evr.dll!CEVRInputPin::ReceiveConnection() + 0x1fc2d bytes    
     quartz.dll!CBasePin::AttemptConnection() - 0x21 bytes    
     quartz.dll!CBasePin::TryMediaTypes() + 0x60 bytes    
     quartz.dll!CBasePin::AgreeMediaType() + 0x54 bytes    
     quartz.dll!CBasePin::Connect() + 0x46 bytes    
     quartz.dll!CFilterGraph::ConnectDirectInternal() + 0x83 bytes    
     quartz.dll!CFilterGraph::ConnectRecursively() + 0x2c bytes    
     quartz.dll!CFilterGraph::ConnectInternal() + 0xde bytes    
     quartz.dll!CFilterGraph::Connect() + 0x17 bytes    
     quartz.dll!CFGControl::WorkerDisplayChanged() + 0xf1 bytes    
     quartz.dll!CFGControl::CGraphWindow::OnReceiveMessage() + 0x2e2a bytes    
>    quartz.dll!WndProc() + 0x3e bytes    
     user32.dll!_InternalCallWinProc@20() + 0x23 bytes    
     user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes    
     user32.dll!_DispatchMessageWorker@8() + 0xed bytes    
     user32.dll!_DispatchMessageW@4() + 0xf bytes    
     quartz.dll!ObjectThread() + 0x65 bytes

A test Visual C++ .NET 2010 application is available from SVN. The code requires a media file, and refers to 352×288 I420.avi, which is included into ZIP file attached to MS Connect Feedback.

A customer complained on video playback problems which had symptoms of software compatibility issue. At certain operation modes software froze leaving the only option to kill process and restart.

I appeared that the system has a third party DLL installed, a new hero name’s – picapmk4.dll. The DLL registers itself as Video for Windows codec and is clearly indicating capabilities it does not really support.

The DLL itself has no indication of its source, no valid version info. Instead it has GPL v2 embedded, XviD logo and about box resource template showing up as “PMK4 Video Codec”.

Looks like crapware, which is hostile to DirectShow environment, with possibly GPL compliance issues.

Two widely popular applications, which use DirectShow API appeared to be locked in a combat: Google video chat is installing a plugin which registers artificial video capture sources “Google Camera Adapter 0″ and “Google Camera Adapter 1″.

As the application does not announce any integration capabilities and is only using video for internal purposes for in-browser video, the approach with system-wide device registration is, well, questionable.

No need to mention, that devices are not operational: they are giving ERROR_DLL_INIT_FAILED (Error 0x8007045A) as soon as you try to instantiate a filter in a DirectShow video capture enabled application.

In-browser Google Video Chat itself does work and shows video from “real” DirectShow devices, including such artificial as created by IP Video Source tool. Google developers prefer still to take data off the device and as soon as possible leave DirectShow domain, as their graph looks like:

Video Capture Source -> Smart Tee Filter -> Sample Grabber Filter -> Null Renderer Filter

While Google plugin developers might need some interprocess synchronization, as the plugin runs in a child “plugin container” process (plugin-container.exe), introduction of virtual video capture devices is not necessary, but is a plain garbage for the operating system and DirectShow environment.

For some reason, Google developers decided to introduce a special thing: they mirror captured video left-to right when it comes to showing picture capture locally. Well, quite possibly it starts a new era in video conferencing: one will not see a direct copy of video sent remotely, but instead he will be given a mirrored thing.

The curious part is, however, the effect the broken devices introduce to another popular software – Skype.

As Skype is capable of video capturing from a DirectShow device, for video-enabled conferencing, it enumerates the available devices and definitely gets in touch with the aliens from Google planet, which are already known to be broken junk.

The outcome appears to be subject to chances, there are two possible outcomes and only one is such that is more or less good for a user:

• either, Skype shows other available devices and ignores Google adapters (while basically any reasonable DirectShow video capture device would still list those devices – after all they are registered as existing!)
• or, Skype stumbles upon a broken thing, stops enumeration and show no video capture devices available at all! Since Skype does enumeration only once on startup, if it fails to discover devices a user needs to close Skype completely and start it again for another try with video hardware.

It appears that both applications have severe issues.

One of the popular questions asked in relation to DirectShow filters, and also a popular task is to modify video stream while in playback. There are various reasons one might need to do it, including keying video to replace color/range, or apply text or time code on top of video, including while re-compressing footage, adjust brightness or contrast.

DirectShow BaseClasses/SDK include samples (in most cases EzRGB24 sample is the best to start for a video effect, also demonstrates use of private interface on a filter) to quick-start with the task without getting too much into detail and once this part is done, next step is to integrate filter into playback, connect it with other filters.

As a result of DircetShowflexibility, there are ways to do things not so good, while still being under impression of keeping right track.

File playback is one of the basic tasks with DirectShow. To play a file, one creates a filter graph using powerful helpers provided by Filter Graph Manager object. It might be a actually a single call IGraphBuilder::RenderFile which takes all the complexity of finding matching filters, connecting them together, dealing with splitters and renderers, video and audio. A single call resolves the problems in a convenient way – easy.

Still a simple thing of inserting your own video transformation filters breaks the simplicity. One needs to build the graph partilly, insert the effect and complete building, or build the thing and break in with a new filter. How to find insertion point? Will the other filters like intrusion? Different file types and formats.

There is an easy and elegant solution to pre-add your own effect filter into graph and start rendering a file from there. Sounds reasonable and sometimes works. The problem is however that it does not work always, and you never know when it lets you down. The graph might be build and the effect filter is never taken and is left orphaned aside of playback pipeline.

Reliable graph building assumes you are in control over building steps and allow only the level of flexibility required to connect and build parts – and this is where Intelligent Connect is still a powerful helper. With an effect, the parts are “before the effect” and “after the effect”. RenderFile is no longer an option, and one has to dive deeper into graph building API.

First of all, the building starts with the file itself: unlike RenderFile, IGraphBuilder::AddSourceFilter method adds just the first filter for a given file/format. It stops there and lets caller continue building as required. At this point, it is the right time to manually add effect filter with IFilterGraph::AddFilter (IGraphBuilder is inherited from IFilterGraph and exposes the same method).

Having both ends in the graph for the “before the effect” part, intelligent connect can be used to connect and add filters required to make the connection. For an arbitrary file format, the task may be not trivial: depending on format and installed software components, the chain may look rather different. First, some filters combine stream splitting capability with immediate access to file (or another type of resource), others rely on joint operation of Async File Source filter with a format-dependent splitter filter. Some expose elementary stream pins immediately, some provide transport stream pin.

There may be a few approaches as for addressing pin connection task (see also General Graph-Building Techniques on MSDN). Straightforwardly, one might want to call IGraphBuilder::Connect and take advantage of intelligent connect. Before this can be done, however it takes caller to select a pin of the obtained source filter to start from. There might be a few pins, including those exposing video formats, non-video formats and pre-split formats where video is behind depacketizing (demultiplexing). Considering variety of formats and support, it might make sense to make a first attempt finding a video pin (by enumerating pins and their media types, looking and AM_MEDIA_TYPE::majortype and comparing to MEDIATYPE_Video) and, if not found, taking a first output pin of any type, or going through ping trying to connect first one which succeeds in connection.

An alternate approach is to take advantage of a helper object: Capture Graph Builder. While originally it is intended to help capture graph building, it contains useful methods for general building and connecting pins. It does not own a graph itself: it is designed to operate on top of existing graph, provided by its owner. So one need to provide its existing graph and call helper methods for easy graph building. One of the methods is ICaptureGraphBuilder2::RenderStream, which connects pins of given filters. Unlike API discussed earlier, it takes filter interfaces on input and will be responsible for finding proper pins itself, which might be a good idea if you don’t want to bother yourself doing it. To specify the requested task, it takes media type argument, which in this case might be video or, if fails, any type provided that video media type will still be anyway checked on input of effect filter.

Once the part “before the effect” is done, the other part may be completed as simple as calling IGraphBuilder::Render on the output pin of the effect. This will correspond to final step of original RenderFile execution.

A tiny Visual Studio 2010 C++ project illustrates discussed techniques and it available at SVN repository: RenderStreamTest01:

• for a given media file in line 109 the project will start graph building
• a suitable replacement for a video effect filter will be a Sample Grabber filter initialized with a video type (24-bit RGB, but the line 137 can be commented out)
• switch in line 147 switches between base Connect approach and Capture Graph Builder helper
• while message box is on the screen and also showing building status, the graph can be looked at using Graph Edit or similar tool, provided that DirectShow Spy is installed; alternatively you might want to put the graph onto ROT manually

The project also illustrates a solution for recent problems referencing Sample Grabber with new SDK. Sample Grabber was obsoleted and removed from Window SDK definition file (qedit.h). In order to resolved the problem without using an older version of SDK, the definitions might be imported from type the corresponding library and (apart from used as such) copied into the source code directly, as in lines 13-60.

See also on graph building:

• Basic DirectShow Tasks
• Intelligent Connect
• General Graph-Building Techniques
• Connect Filters
• Using Capture Graph Builder

This quick update for DirectShow Filter Graph Spy adds tracing for (deprecated) IAMFilterData interface, and also fixes problem with Media Player Classic – Home Cinema, when the player crashes in External Filters windows.

Partial Visual C++ .NET 2008 source code is available from SVN, release binary included (Win32, x64); installation instructions are in another post.