According to CSI: Crime Scene Investigation, Season 06 Episode 13 (17:04), investigators are using snake scope camera with DirectShow AMCAP tool to present the captured image:

It is really great news that Windows 7 offers quite some goodies for MPEG-4 support right out of the box, but some things appear to be made not so accurate.

The component responsible for video decoding is Microsoft DTV-DVD Video Decoder (also known as Microsoft MPEG-2 Video Decoder), available with DirectShow interface. It offers video decoding capabilities for video resolutions up to 1920×1088 (Full HD), including DXVA version 2 mode. The decoder is a “preferred decoder” and is given a first chance to be taken when it comes for Intelligent Connect to render a video stream.

Technology went ahead though, stepped behind 1920×1088 in particular. Microsoft DTV-DVD Video Decoder DirectShow filter still remains the first take in H.264 video decoding, and what however is taking place – the decoder accepts the format (media type) which is clearly cannot decode, neither according to specification, nor in real life. As soon as pin connection succeeds, DirectShow is not trying to look for another decoder, which can truly decode the feed. Result: failure to decode video.

The problem takes place in a quite surprising way. Given 5 megapixel video (2560×1592, H.264, Baseline profile, from RTSP-enabled IP camera), the decoder advertises 720×480 NV12 output media type. While consuming input data, the decoder outputs nothing and keeps video window black.

It does not issue any filter graph events, no. The filter is persistent and works hard in assumption that a new piece of knowledge will come from Mars and provide new capabilities to start decoding video, or filter’s input will give up and provide a new SPS to lower resolution down to something less complex.

Well then, so is this advanced technology yet to be discovered by science? It appears that no. CoreAVC Video Decoder, applied to the same H.264 video decodes it well and nice:

Update 2011-09-26: According to Microsoft classification, this is a bug on a released product. That is, if anyone is experiencing severe issues as a result of this behavior, one might need to contact Microsoft Support directly in search of a solution.

Actually, as resolutions over Full HD (1920×1080) are not so much common at the moment, the issue side effects are not yet likely. Still moving on with time, I think the issues are likely to come up, without being severe though.

DirectShow Spy is updated with a few new features:

• retroactive Intelligent Connect trace
• log output for IAMGraphBuilderCallback-related activity
• process name based black list to selectively exclude processes from spying

Intelligent Connect Trace

The utility received a capability to read back from its own log file (DirectShowSpy.log, located typically in C:\ProgramData directory) and reconstruct graph building sequence, including steps taken by DirectShow Intelligent Connect.

In order to activate the Intelligent Connect Trace property sheet, one needs to call exported function “DoGraphBuilderCallbackPropertySheetModal“, such as using runndll32 tool:

C:\DirectShowSpy>rundll32 DirectShowSpy.dll,DoGraphBuilderCallbackPropertySheetModal

The upper part of the property page displays recently created DirectShow fitler graphs, newest to older. For a selected graph, the lower part displays events in chronological order. The events include:

• Adding a filter, at IFilterGraph2::AddFilter method
• Removing a filter, at IFilterGraph2::RemoveFilter method
• Selecting a filter, at Intelligent Connect’s callback at IAMGraphBuilderCallback::SelectedFilter
• Creating a filter, at IAMGraphBuilderCallback::CreatedFilter

The latter two methods also show “Application Result” column and values, which are HRESULT values returned by IAMGraphBuilderCallback callback provided by the application. Typically, a failure HRESULT code indicates that the application rejected the filter.

The trace log is good to expose all DirectShow junk installed in the system. For example,

In an attempt to build graph to decode FourCC ‘LXH4′ video (which is a private alias for H.264 media type), Filter Graph Manager tried to apply ‘ffdshow Video Decoder’ filter, which is unable to process the feed, but is registered at an extremely high merit and violating the guidelines.

The tool shows that the file hosting the filter is “ffdshowBC.ax” and shows its location. The application however appears to be aware of this piece of trash, and returned E_FAIL (0×80004005) rejecting the filter.

How comes the garbage is hooking into place where it is not supposed to be, yet is sophisticated enough to reach such internals of operating system? This module is a clone/branch of well-known ffdshow software, which by default registers itself with abnormally high merit “with good intentions in mind”. A short-sighted manufacturer of cheap stuff IP camera duplicated the project, changed the GUIDs, packaged the pathetic crap into ActiveX control cabinet and started distributing it over clean and virgin workstations around the globe.

Going further with the the trace being checked, there are more things to enjoy:

Trying to decode the same H.264 video feed, the Filter Graph Manager mounting three demultiplexers made  by MainConcept in a row, for MP4, MPEG and MXF containers. The utility immediately indicates that this trash contribution was brought by MainConcept Showcase 8.7.0.28412. One might wanted to install it for a purpose, and it started killing other DirectShow software it shares workstation with.

The good news is that the method is friendly for production environment and only requires to register spy and collect log file. You don’t eve need to check the UI at the same computer, it is possible to transfer the log file into development environment and have spy read the log there.

The “Copy to Clipboard” link is a convenient option to transfer the findings into tab-separated values format, such as to paste into Microsoft Excel:

Process Black Listing

The utility received a capability to selectively not load into specific processes. DirectShowSpy is a developer tool and is not intended for production environment, nor it is guaranteed to be issue-free. So in case of specific conflict it is possible to exclude specific process from applying a spy using registry:

The values are under HKLM, SOFTWARE\[Wow6432Node\]Alax.Info\Utilities\DirectShowSpy key, and there are three values defined:

• CFilterMapperSpy Black List – defines exclusion for filter mapper spying
• CSpy Black List – defines exclusion for filter graph spy (effective for most applications)
• CNoThreadSpy Black List – defines exclusion for filter graphs created with CLSID_FilterGraphNoThread

The exclusion value is a comma-separated list of process names, exact case-insensitive match for executable file names without directory and extension.

It is recommended that filter mapper spy has at least as rich exclusion list as the filter graph spy entires (otherwise, it is possible that spy might raise unexpected exceptions).

Partial Visual C++ .NET 2008 source code is available from SVN, release binary included (Win32, x64); installation instructions are in another post.

A customer complained on video playback problems which had symptoms of software compatibility issue. At certain operation modes software froze leaving the only option to kill process and restart.

I appeared that the system has a third party DLL installed, a new hero name’s – picapmk4.dll. The DLL registers itself as Video for Windows codec and is clearly indicating capabilities it does not really support.

The DLL itself has no indication of its source, no valid version info. Instead it has GPL v2 embedded, XviD logo and about box resource template showing up as “PMK4 Video Codec”.

Looks like crapware, which is hostile to DirectShow environment, with possibly GPL compliance issues.

Two widely popular applications, which use DirectShow API appeared to be locked in a combat: Google video chat is installing a plugin which registers artificial video capture sources “Google Camera Adapter 0″ and “Google Camera Adapter 1″.

As the application does not announce any integration capabilities and is only using video for internal purposes for in-browser video, the approach with system-wide device registration is, well, questionable.

No need to mention, that devices are not operational: they are giving ERROR_DLL_INIT_FAILED (Error 0x8007045A) as soon as you try to instantiate a filter in a DirectShow video capture enabled application.

In-browser Google Video Chat itself does work and shows video from “real” DirectShow devices, including such artificial as created by IP Video Source tool. Google developers prefer still to take data off the device and as soon as possible leave DirectShow domain, as their graph looks like:

Video Capture Source -> Smart Tee Filter -> Sample Grabber Filter -> Null Renderer Filter

While Google plugin developers might need some interprocess synchronization, as the plugin runs in a child “plugin container” process (plugin-container.exe), introduction of virtual video capture devices is not necessary, but is a plain garbage for the operating system and DirectShow environment.

For some reason, Google developers decided to introduce a special thing: they mirror captured video left-to right when it comes to showing picture capture locally. Well, quite possibly it starts a new era in video conferencing: one will not see a direct copy of video sent remotely, but instead he will be given a mirrored thing.

The curious part is, however, the effect the broken devices introduce to another popular software – Skype.

As Skype is capable of video capturing from a DirectShow device, for video-enabled conferencing, it enumerates the available devices and definitely gets in touch with the aliens from Google planet, which are already known to be broken junk.

The outcome appears to be subject to chances, there are two possible outcomes and only one is such that is more or less good for a user:

• either, Skype shows other available devices and ignores Google adapters (while basically any reasonable DirectShow video capture device would still list those devices – after all they are registered as existing!)
• or, Skype stumbles upon a broken thing, stops enumeration and show no video capture devices available at all! Since Skype does enumeration only once on startup, if it fails to discover devices a user needs to close Skype completely and start it again for another try with video hardware.

It appears that both applications have severe issues.

This does not update the software with new features, but there are a few simple things worth mentioning explicitly.

The first is that virtual DirectShow camera device can be set up with both M-JPEG and JPEG URLs. That is, IP cameras which do not implement M-JPEG, or implement it in a buggy way (there is a *huge* deal of such out there) can still be set up to send video as individual video frames/images as long as they implement JPEG snapshots. This is taking place often at a lower frame rate, but still works.

The driver will automatically detect type of URL (by response from the device) and will choose best access method for the given URL.

Second is that if you are looking for IP Video Source software version, such as to check against available updates, it is here on the UI (right click the caption):

The IP Video Source update provides several improvements to the driver:

• copy/paste feature to backup, restore, or synchronize installed devices between 32-bit and 64-bit versions
• 64-bit version and .MSI
• dynamic video resizing (via Video Resizer DSP)
• Adobe FMLE compatibility

Updates in greater detail follow.

Device Copy/Paste Feature

The video device management window is providing Copy and Paste buttons, which let user transfer device information, including name and settings, through clipboard for various purposes:

• save data in order to be able to restore devices later
• restore devices from saved list, or re-create from a list saved on another machine
• duplicate a device
• synchronize devices between 32-bit and 64-bit versions

The device data is a text, one line per device, lines in comma-separated values (CSV) format.

64-Bit Version

Software is available in both 32-bit and 64-bit versions. While most of multimedia software is still 32-bit, sometimes 64-bit version is really helpful to, for example:

• interface to “Any CPU” .NET code which executes in 64-bit on a 64-bit operating system
• go beyond 32-bit code boundaries, especially in part of multiple devices and virtual address space limit

64-bit version duplicates functionality available in 32-bit version, though 32- and 64-bit modules run in completely separated environments.

Dynamic Video Resizing

The source DirectShow filter is capable of connecting with media types different from original/camera media type in resolution part for decompressed video. The filter is internally taking advantage of Video Resizer DSP, and resizes decompressed video to format of interest.

As Video Resizer DSP is available starting with Windows Vista, the functionality availability is subject to the same conditions.

The feature is absolutely required for Adobe Flash Media Live Encoder software, as it make certain assumptions as for supported resolutions, and completely ignores the information the filter makes available in standard way.

Adobe Flash Media Live Encoder Friendliness

The update make the device much closer to interoperation with Adobe product. The software works around a handful of bugs, including as ridiculous as neglecting to provide correct BI_RGB value in BITMAPINFOHEADER::biCompression structure, yet researchers are still to resolve the remained ones.

FMLE would start perfect and stream video being run under debugger, otherwise in a regular run chances are that Adobe application would internally respond with E_UNEXPECTED failure for unknown reason, stop any encoding without giving a single little notice on the GUI.

After all, the encoding runs perfectly through web based encoder, and the driver is providing a good way to stream to video distribution sites such as http://ustream.tv, http://livestream.com from an IP camera or video encoder.

Download Information

• Download (32-bit): AlaxInfoIpVideoSourceSetup.msi
• Download (64-bit): AlaxInfoIpVideoSourceSetup-x64.msi
• License: This software is free to use

This implements a DirectShow driver/wrapper over a HTTP based JPEG/M-JPEG streamed video, widely available with IP cameras. Once installed, it provides a Start Menu shortcut to manage video capture devices, where a user can add/remove devices. The devices are automatically registered with DriectShow and are available to applications.

The compatibility list includes:

• Windows SDK AmCap Sample (reference)
• VideoLan VLC
• Skype (see below)
• Google Talk Video Chat
• Luxriot (as rather an example as Luxriot has its own generic JPEG/M-JPEG device driver, however this still demonstrates compatibility and interoperability of applications)
• GraphEdit, GraphStudio and similar tools

For a created device it is required to configure:

1. JPEG or M-JPEG resource URL on the device, where the driver will access video feed or sequential snapshots
2. Image/video size on the device

Resource URLs vary from device to device and there is no standard identifier. This has to be looked up for specific model in device API reference or SDK. For JPEG snapshots it is possible to copy image shortcut from web browser and paste address into device settings.

Note that some devices, especially low end IP cameras deliver non-compliant feed which can be decoded either by manufacturer’s ActiveX control or model-specific implementation which this driver does not convert and is unlikely to ever cover. The driver will only work with hardware which honors industry standards.

For example, Stardot NetCam SC cameras make the video feed accessible using /nph-mjpeg.cgi resource, so the full URL will consist of protocol prefix “http://”, optional username and password (see screenshot above), IP address or host name e.g. “demo2.stardotcams.com”, resource identifier on the device “/nph-mjpeg.cgi”.

Hence, http://demo2.stardotcams.com/nph-mjpeg.cgi

It is necessary to provide video size of the footage in advance so that the driver could negotiate content media types without accessing the device. Future versions of the driver might implement auto-detection.

Skype Compatibility

To make the device work with Skype, it is necessary to:

1. Configure the device (using its web interface) to work at resolution 640×480
2. Add/configure device with Alax.Info IP video Source as described above, including setting of 640 by 480 pixel resolution in device configuration

Internals

Software implements and registers video input/capture devices, listed under DirectShow CLSID_VideoInputDeviceCategory category. The applications access devices from the list and treat as a regular device such as web camera.

Windows SDK AmCap sample will create the following DirectShow graph where the source filter implemented by this software:

Internally, software creates a worker filter graph in order to stream data from the device:

To decode video, software uses stock MJPEG Decompressor Filter provided with operating system.

Download Information

• Download (32-bit): AlaxInfoIpVideoSourceSetup.msi
• Download (64-bit): To Be Done/On Request
• License: This software is free to use

One of the popular questions asked in relation to DirectShow filters, and also a popular task is to modify video stream while in playback. There are various reasons one might need to do it, including keying video to replace color/range, or apply text or time code on top of video, including while re-compressing footage, adjust brightness or contrast.

DirectShow BaseClasses/SDK include samples (in most cases EzRGB24 sample is the best to start for a video effect, also demonstrates use of private interface on a filter) to quick-start with the task without getting too much into detail and once this part is done, next step is to integrate filter into playback, connect it with other filters.

As a result of DircetShowflexibility, there are ways to do things not so good, while still being under impression of keeping right track.

File playback is one of the basic tasks with DirectShow. To play a file, one creates a filter graph using powerful helpers provided by Filter Graph Manager object. It might be a actually a single call IGraphBuilder::RenderFile which takes all the complexity of finding matching filters, connecting them together, dealing with splitters and renderers, video and audio. A single call resolves the problems in a convenient way – easy.

Still a simple thing of inserting your own video transformation filters breaks the simplicity. One needs to build the graph partilly, insert the effect and complete building, or build the thing and break in with a new filter. How to find insertion point? Will the other filters like intrusion? Different file types and formats.

There is an easy and elegant solution to pre-add your own effect filter into graph and start rendering a file from there. Sounds reasonable and sometimes works. The problem is however that it does not work always, and you never know when it lets you down. The graph might be build and the effect filter is never taken and is left orphaned aside of playback pipeline.

Reliable graph building assumes you are in control over building steps and allow only the level of flexibility required to connect and build parts – and this is where Intelligent Connect is still a powerful helper. With an effect, the parts are “before the effect” and “after the effect”. RenderFile is no longer an option, and one has to dive deeper into graph building API.

First of all, the building starts with the file itself: unlike RenderFile, IGraphBuilder::AddSourceFilter method adds just the first filter for a given file/format. It stops there and lets caller continue building as required. At this point, it is the right time to manually add effect filter with IFilterGraph::AddFilter (IGraphBuilder is inherited from IFilterGraph and exposes the same method).

Having both ends in the graph for the “before the effect” part, intelligent connect can be used to connect and add filters required to make the connection. For an arbitrary file format, the task may be not trivial: depending on format and installed software components, the chain may look rather different. First, some filters combine stream splitting capability with immediate access to file (or another type of resource), others rely on joint operation of Async File Source filter with a format-dependent splitter filter. Some expose elementary stream pins immediately, some provide transport stream pin.

There may be a few approaches as for addressing pin connection task (see also General Graph-Building Techniques on MSDN). Straightforwardly, one might want to call IGraphBuilder::Connect and take advantage of intelligent connect. Before this can be done, however it takes caller to select a pin of the obtained source filter to start from. There might be a few pins, including those exposing video formats, non-video formats and pre-split formats where video is behind depacketizing (demultiplexing). Considering variety of formats and support, it might make sense to make a first attempt finding a video pin (by enumerating pins and their media types, looking and AM_MEDIA_TYPE::majortype and comparing to MEDIATYPE_Video) and, if not found, taking a first output pin of any type, or going through ping trying to connect first one which succeeds in connection.

An alternate approach is to take advantage of a helper object: Capture Graph Builder. While originally it is intended to help capture graph building, it contains useful methods for general building and connecting pins. It does not own a graph itself: it is designed to operate on top of existing graph, provided by its owner. So one need to provide its existing graph and call helper methods for easy graph building. One of the methods is ICaptureGraphBuilder2::RenderStream, which connects pins of given filters. Unlike API discussed earlier, it takes filter interfaces on input and will be responsible for finding proper pins itself, which might be a good idea if you don’t want to bother yourself doing it. To specify the requested task, it takes media type argument, which in this case might be video or, if fails, any type provided that video media type will still be anyway checked on input of effect filter.

Once the part “before the effect” is done, the other part may be completed as simple as calling IGraphBuilder::Render on the output pin of the effect. This will correspond to final step of original RenderFile execution.

A tiny Visual Studio 2010 C++ project illustrates discussed techniques and it available at SVN repository: RenderStreamTest01:

• for a given media file in line 109 the project will start graph building
• a suitable replacement for a video effect filter will be a Sample Grabber filter initialized with a video type (24-bit RGB, but the line 137 can be commented out)
• switch in line 147 switches between base Connect approach and Capture Graph Builder helper
• while message box is on the screen and also showing building status, the graph can be looked at using Graph Edit or similar tool, provided that DirectShow Spy is installed; alternatively you might want to put the graph onto ROT manually

The project also illustrates a solution for recent problems referencing Sample Grabber with new SDK. Sample Grabber was obsoleted and removed from Window SDK definition file (qedit.h). In order to resolved the problem without using an older version of SDK, the definitions might be imported from type the corresponding library and (apart from used as such) copied into the source code directly, as in lines 13-60.

See also on graph building:

• Basic DirectShow Tasks
• Intelligent Connect
• General Graph-Building Techniques
• Connect Filters
• Using Capture Graph Builder

A small update to the DirectShow Spy today: DirectShow Filter Graph Spy prints memory allocator properties as a part of graph topology trace on transition to running state. Why is that and what it is for? Filters normally agree on allocator properties (ALLOCATOR_PROPERTIES, obtained from IMemAllocator, obtained from IMemInputPin) themselves without interference from controlling application. Sometimes an undesired choice of buffers can cause sick runtime behavior, including but not limited to the following:

1. live audio capture buffers are too long, and introduce significant latency, including from live video capture stream taking place in parallel; controlling application might need to take advantage of IAMBufferNegotiation::SuggestAllocatorProperties and request shorter buffers.
2. a filter, such as for example DMO Wrapper Filter, may default to 1 buffer on allocator, which means that if a buffer reference is for some reason held for future reference (e.g. a video filter might be holding a reference to be able to re-push a video sample if an application is requesting video update), the entire streaming might lock.
3. some filters are requesting unreasonably many/large buffers and consume vast spaces of RAM, such as MainConcept MXF Demultiplexer requesting 200 buffers x 640 kilobytes each (128 MB in total out of sudden)
4. some filters are requesting unreasonably few/small buffers resulting in inability to pre-buffer data

In a chase for answers to questions “Where is my memory?”, “Why is it so choppy?”, “I would really appreciate a nice lipsync” and to troubleshoot the mentioned scenarios it is helpful to understand buffering configuration. DirectShow Filter Spy is here to deliver this information. Once the graph is put into running state, spy prints out topology data into log file (which is in most cases C:\ProgramData\DirectShowSpy.log):

Pin 2: Name "Input 01", Direction "Input", Peer "Tee 0x087A5AF0.Output2"
 Connection media type:
 majortype {73646976-0000-0010-8000-00AA00389B71}, subtype {31435641-0000-0010-8000-00AA00389B71}, pUnk 0x00000000
 bFixedSizeSamples 0, bTemporalCompression 0, lSampleSize 1
 formattype {E06D80E3-DB46-11CF-B4D1-00805F6CBBEA}, cbFormat 170, pbFormat 0x07c46fc0
pbFormat as MPEG2VIDEOINFO:
 rcSource { 0, 0, 0, 0 ), rcTarget { 0, 0, 0, 0 }
 dwBitRate 0, dwBitErrorRate 0, AvgTimePerFrame 0
 dwInterlaceFlags 0x0, dwCopyProtectFlags 0x0, dwPictAspectRatioX 16, dwPictAspectRatioY 9, dwControlFlags 0x0
 bmiHeader.biSize 40, bmiHeader.biWidth 1280, bmiHeader.biHeight 720, bmiHeader.biPlanes 1, bmiHeader.biBitCount 24, bmiHeader.biCompression avc1
 bmiHeader.biSizeImage 0, bmiHeader.biXPelsPerMeter 1, bmiHeader.biYPelsPerMeter 1, bmiHeader.biClrUsed 0, bmiHeader.biClrImportant 0
 dwStartTimeCode 0x00000000, cbSequenceHeader 38, dwProfile 100, dwLevel 31, dwFlags 0x4
 [0x0000] 00 1D 67 64 00 1F AC 24 88 05 00 5B BF F0 00 10
 [0x0010] 00 11 00 00 03 03 E8 00 00 E9 BA 0F 18 32 A0 00
 [0x0020] 05 68 EE 32 C8 B0
 Memory Allocator: 1 buffers, 1,024 bytes each (1,024 bytes total), align 1, prefix 0

Partial Visual C++ .NET 2008 source code is available from SVN, release binary included (Win32, x64); installation instructions are in another post.