Direct3D based DirectShow video renderers – Video Mixing Renderer 9 and Enhanced Video Renderer – have been notoriously known for consuming resources in a way that you can run at most X simultaneously. There has been no comment published on the topic and questions (e.g. like this: How many VMR 9 can a PC support concurrently) remain unanswered for a long time. Video Mixing Renderer 7 was a good alternative for some time in past until it was cut down to be unable to support hardware scaling (thanks Microsoft!).

The trendy way to render video nowadays is using Enhanced Video Renderer, a Media Foundation subsystem with an interface into DirectShow to take over state of the art video rendering capabilities. So, how many EVRs one can run simultaneously? Chances are that it is less than one could suppose.

The interesting part is that there is no obvious evidence on type of resource running out, which causing next EVR instance to fail to run. And not even run, the failure seem to be coming up at an earlier stage of just connecting pins in stopped state. The failure might be accompanied with errors like E_INVALIDARG, ERROR_FILE_NOT_FOUND, E_UNEXPECTED. The actual limit appear to be loosely correlating to parameters of video output, such as resolution and bitness.

Desperately waiting for clarification, I am sharing the tool to estimate the limit:

• multiple EVR instances at once, hosted by multiple windows, which can be distributed across multiple monitors
• choices of resolutions and formats
• a double click on an individual renderer pops up property page set displaying effective frame rate
• 32- and 64-bit versions

A binary [Win32, x64] and partial Visual C++ .NET 2010 partial source code are available from SVN.

An application appears to be not working in Windows XP in a weird way: rebar control appeared to fail showing up.

Where the application is expected to look much nicer with rebar control as a container for menu (implemented as command bar WTL control) and toolbar with buttons:

A WTL sample project generated by a Visual Studio wizard would never give such effect, and the bug was a combination of factors:

1. An application built with a newer version of Windows SDK, which includes support for features (Windows Vista+ Common Controls) that are more recent than production environment (Windows XP); the application targets to Windows Vista+ environment too (_WIN32_WINNT >= 0×0600)
2. Compatibility issues of Common Controls library
3. WTL version (7.5), which did not yet include a workaround for the problem

The problem, which caused the bug directly was the REBARBANDINFO structure and its use as an argument with Common Controls API. As MSDN shows, the structure was amended twice with additional fields.

One of the way to support multiple versions of the structure definition, and to resolve compatibility issues, is to embed structure size into structure payload. In fact, REBARBANDINFO::cbSize member is there exactly for this reason.

The application is normally filling cbSize with the maximal known structure size and fills the rest of the fields respectively. The API is expected to be checking cbSize member and be detecting API version compatibility scenarios:

1. cbSize holds exactly the value the API expects (that is, the maximal value known/defined to the API) – the simplest scenario where the API and the application are on the same page, both are using the same versions of the “protocol”/interface.
2. cbSize is smaller than API can support – the API sees that it is dealing with a sort of legacy application which cannot utilize all available features, and the API acts respectively supporting the older part of the protocol, and keeping defaults or “old look” for the rest of implementation. This addresses backward compatibility: the newer API works with apps designed for older version of the API
3. cbSize is greater then API can support – the API sees that the application is already aware of newer version API and is possibly requesting some of the missing features. The API might be ignoring the unsupported part in assumption that API evolution tool place keeping some compatibility in mind, and still do the best it can with the existing implementation. Or, the API might just fail to work.

The latter item #3 is the scenario here with rebar control. The application is using Windows Vista version of REBARBANDINFO structure and Windows XP implementation choses to completely fail.

While it does not seem to be directly a bug, this attitude is definitely not developer friendly: there is no reason for the control to not work in its best and default way. Having API acting this way, each developer using the API needs to take care of the situation explicitly: whenever Windows Vista enabled application needs to be able to run in Windows XP system, the code around REBARBANDINFO would look like this:

REBARBANDINFO BandInformation = { sizeof BandInformation, RBBIM_LPARAM };
#if _WIN32_WINNT >= 0x0600
if(GetOsVersion() < 0x00060000 || GetCommCtrlVersion() < 0x00060000) // pre-Vista, Common Controls pre-6.0
    BandInformation.cbSize = REBARBANDINFO_V6_SIZE;
#endif// _WIN32_WINNT >= 0x0600
const BOOL bGetBandInfoResult = Rebar.GetBandInfo(0, &BandInformation);

If the API was nicer to developers, the code would be plain and simple:

REBARBANDINFO BandInformation = { sizeof BandInformation, RBBIM_LPARAM };
const BOOL bGetBandInfoResult = Rebar.GetBandInfo(0, &BandInformation);

To address this problem, WTL 8.0 comes up with RunTimeHelper namespace and its SizeOf_REBARBANDINFO function. It takes care of details for the developer choosing the proper size of the structure on runtime. The code is being taken back to a simpler shape:

REBARBANDINFO BandInformation = { RunTimeHelper::SizeOf_REBARBANDINFO(), RBBIM_LPARAM };
const BOOL bGetBandInfoResult = Rebar.GetBandInfo(0, &BandInformation);

All in all:

• be aware of compatibility issues (same scenario exists with other SDK structures: LVGROUP, LVTILEINFO, MCHITTESTINFO, NONCLIENTMETRICS and other).
• use latest version of WTL to have things worked around for you where Microsoft developers were not kid enough to provide perfect API
• be aware and take advantage of WTL’s RunTimeHelper class

It is really great news that Windows 7 offers quite some goodies for MPEG-4 support right out of the box, but some things appear to be made not so accurate.

The component responsible for video decoding is Microsoft DTV-DVD Video Decoder (also known as Microsoft MPEG-2 Video Decoder), available with DirectShow interface. It offers video decoding capabilities for video resolutions up to 1920×1088 (Full HD), including DXVA version 2 mode. The decoder is a “preferred decoder” and is given a first chance to be taken when it comes for Intelligent Connect to render a video stream.

Technology went ahead though, stepped behind 1920×1088 in particular. Microsoft DTV-DVD Video Decoder DirectShow filter still remains the first take in H.264 video decoding, and what however is taking place – the decoder accepts the format (media type) which is clearly cannot decode, neither according to specification, nor in real life. As soon as pin connection succeeds, DirectShow is not trying to look for another decoder, which can truly decode the feed. Result: failure to decode video.

The problem takes place in a quite surprising way. Given 5 megapixel video (2560×1592, H.264, Baseline profile, from RTSP-enabled IP camera), the decoder advertises 720×480 NV12 output media type. While consuming input data, the decoder outputs nothing and keeps video window black.

It does not issue any filter graph events, no. The filter is persistent and works hard in assumption that a new piece of knowledge will come from Mars and provide new capabilities to start decoding video, or filter’s input will give up and provide a new SPS to lower resolution down to something less complex.

Well then, so is this advanced technology yet to be discovered by science? It appears that no. CoreAVC Video Decoder, applied to the same H.264 video decodes it well and nice:

Update 2011-09-26: According to Microsoft classification, this is a bug on a released product. That is, if anyone is experiencing severe issues as a result of this behavior, one might need to contact Microsoft Support directly in search of a solution.

Actually, as resolutions over Full HD (1920×1080) are not so much common at the moment, the issue side effects are not yet likely. Still moving on with time, I think the issues are likely to come up, without being severe though.

Crime

An application which builds a DirectShow graph unexpectedly started failing with VFW_E_NOT_CONNECTED (0×80040209) error code.

Scene

The problem takes place during DirectShow graph building, yet in stopped state. Specific call which appeared to be giving out the error in first place appears to be EVR input pin’s IPin::ConnectionMediaType, and the problem is also specific to Enhanced video Renderer (Windows 7, but not necessarily only this version).

Investigation

The problem does not appear to be persistent. On the contrary, it is taking place for just a few milliseconds after pin connection. After the problem is gone, it does not seem to ever come up again unless the filter graph is built again from the beginning.

EVR pin connection is always reporting success, so the following error code stating VFW_E_NOT_CONNECTED “The operation cannot be performed because the pins are not connected.” goes against documented behavior, and is thus a bug.

Depending on time between pin connection and media type polling, the call can reach EVR:

• before it starts showing the problem – stage A
• at the time the call fails – stage B
• after the failure time interval, when the call is successful from then on – stage C

Thus, the problem is limited to specific use cases:

• the application should care about media type on EVR input
• unexpected failure takes place when the call reaches in stage B
• also found: the clipping window for the EVR has to belong to a non-primary monitor

If an application keep polling for media type in a loop, the result may be about the following:

UINT nStageA = 0, nStageB = 0, nStageC = 0;
// [...]
for(; ; )
{
    AM_MEDIA_TYPE MediaType;
    ZeroMemory(&MediaType, sizeof MediaType);
    const HRESULT nConnectionMediaTypeResult = pInputPin->ConnectionMediaType(&MediaType);
    if(SUCCEEDED(nConnectionMediaTypeResult))
    {
        if(nStageB)
        {
            nStageC++;
            break;
        } else
            nStageA++;
    } else
    {
        ATLASSERT(nConnectionMediaTypeResult == VFW_E_NOT_CONNECTED);
        nStageB++;
    }
    CoTaskMemFree(MediaType.pbFormat);
}
// [...]
CString sMessage;
sMessage.Format(_T("Bingo!\r\n\r\n") _T("nStageA %d, nStageB %d - 0x%08x, nStageC %d\n"), nStageA, nStageB, nResult, nStageC);
AtlMessageBox(m_hWnd, (LPCTSTR) sMessage, _T("Result"), MB_ICONERROR);

Workaround

An obvious straightforward workaround is to follow EVR connection with a wait for Stage B to pass, or timeout – whichever takes place first.

Also, vote for the bug on Microsoft Connect.

More Details

Video renderer filter are notorious for re-agreeing media type and being fretful as for memory allocators and media types (for a good reason though!). So it makes sense to suggest that the problem takes place when the filter is doing something related, such as it starts background activity immediately after connection in order to discover upstream peer capabilities.

In order to possibly get details on this, it is possible to raise an exception as soon as Stage B is detected and take a look at thread states using a debugger. Indeed, on of the background threads is engaged in EVR reconnection activity:

Yes it does the reconnection, but nevertheless it is expected to do the things undercover and transparently, it still allows a failure on the outer API.

     evr.dll!GetSourceRectFromMediaType() + 0x37 bytes    
     evr.dll!CEVRInputPin::CheckMediaType() + 0x81 bytes    
     evr.dll!CBasePin::ReceiveConnection() + 0x61 bytes    
     evr.dll!CEVRInputPin::ReceiveConnection() + 0x1fc2d bytes    
     quartz.dll!CBasePin::AttemptConnection() - 0x21 bytes    
     quartz.dll!CBasePin::TryMediaTypes() + 0x60 bytes    
     quartz.dll!CBasePin::AgreeMediaType() + 0x54 bytes    
     quartz.dll!CBasePin::Connect() + 0x46 bytes    
     quartz.dll!CFilterGraph::ConnectDirectInternal() + 0x83 bytes    
     quartz.dll!CFilterGraph::ConnectRecursively() + 0x2c bytes    
     quartz.dll!CFilterGraph::ConnectInternal() + 0xde bytes    
     quartz.dll!CFilterGraph::Connect() + 0x17 bytes    
     quartz.dll!CFGControl::WorkerDisplayChanged() + 0xf1 bytes    
     quartz.dll!CFGControl::CGraphWindow::OnReceiveMessage() + 0x2e2a bytes    
>    quartz.dll!WndProc() + 0x3e bytes    
     user32.dll!_InternalCallWinProc@20() + 0x23 bytes    
     user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes    
     user32.dll!_DispatchMessageWorker@8() + 0xed bytes    
     user32.dll!_DispatchMessageW@4() + 0xf bytes    
     quartz.dll!ObjectThread() + 0x65 bytes

A test Visual C++ .NET 2010 application is available from SVN. The code requires a media file, and refers to 352×288 I420.avi, which is included into ZIP file attached to MS Connect Feedback.

A colleague pointed out that code snippet in previous post is misusing ATL’s ATLENSURE_SUCCEEDED macro making it [possibly] evaluate its argument twice in case of failure, that is evaluating into failure HRESULT code. As it is defined like this:

#define ATLENSURE_SUCCEEDED(hr) ATLENSURE_THROW(SUCCEEDED(hr), hr)

It does things in a straightforward way, for a code line

ATLENSURE_SUCCEEDED(pFilterGraph.CoCreateInstance(CLSID_FilterGraph));

It is doing “let’s CoCreateInstance the thing and if it fails, let’s CoCreateInstance it again to find out error code”. Disassembly shows this clearly:

This is exactly another spin of the story previously happened with HRESULT_FROM_WIN32 macro and possibly a number of others. With it being originally a macro, SDK offered an option to override the definition by pre-defining INLINE_HRESULT_FROM_WIN32. This way a user might be explicitly requesting a safer definition while still leaving legacy code live with macro. See more detailed story on this in Matthew’s blog.

A tricky thing is that with successful execution the problem does not come up. In case of failure, it depends on the functions called, some with just repeat the error code, some will return a different code on second run, some might create less desired and expected consequences. So you can find yourself having written quite some code before you even suspect a problem.

Having identified the issue, there are a few solutions.

1. First of all, the original ATLENSURE_SUCCEEDED macro can still be used, provided that you don’t put expressions as arguments.

This is going to do just fine:

const HRESULT nCoCreateInstanceResult = pFilterGraph.CoCreateInstance(CLSID_FilterGraph);
ATLENSURE_SUCCEEDED(nCoCreateInstanceResult);

2. Second straightforward way is to replace the original ATL definition in ATL code (boo, woodenly)

3. As ATL code is checking for the macros to be already defined, and skipping its own definition in such case, it is possible to inject a safer private definition before including ATL headers (which would typically need one to do the define in stdafx.h):

#define ATLENSURE_SUCCEEDED(x){ const HRESULT nResult =(x); ATLENSURE_THROW(SUCCEEDED(nResult), nResult); }

#include <atlbase.h>
#include <atlstr.h>

Pre-evaluating the argument into local variable is going to resolve the original multi-evaluation problem.

4. There might be a new inline function defined on top of the original macro, which will be used instead and which is free from the problem:

inline VOID ATLENSURE_INLINE_SUCCEEDED(HRESULT nResult)
{
    ATLENSURE_SUCCEEDED(nResult);
}

Either way, the correct code compiles into single argument evaluation and throws an exception with failure code immediately:

Also, vote for the suggestion on Microsoft Connect. The issue is marked as fixed in future version of Visual Studio.

If you dare to convert your C++ ATL Service project created with an earlier version of Visual Studio to version 2010, as I recently did, you might find yourself surprised with why the hell the bloody thing does not work anymore as a regular executable.

After passing compiler/linker and SDK update issues, which you possibly might have too, the started executable will stumble on ATL error/exception with CO_E_NOTINITIALIZED (0x800401F0 “CoInitialize has not been called.”). Luckily the error code is good enough for quickly locating the problem and the reason is that the one you trusted, that is ATL, introduced an small improvement which is good for running as service but it not initializing COM anymore if you run your .EXE in application mode.

The bug is on MS Connect since August 3, 2010 closed as if it is going to be fixed some time in future when the fix is propagated to end developers. If you are in a rush and would like to write some code before the event, here are the details.

Previously, COM was initialized right in module constructor, in CAtlExeModuleT. CAtlServiceModuleT class just inherited from there. Later on, someone smart decided that it was not so cool and moved initialization to a later point into CAtlExeModuleT::WinMain. Well, this makes sense as you might (a) end up not needing COM at all, or (b) you want to do some important things before even initializing COM.

Unfortunately, the fact that CAtlServiceModuleT is inherited and relies on base class was not paid too much attention. CAtlServiceModuleT is not getting COM initialization from constructor any longer, CAtlServiceModuleT::WinMain is overridden in full and does not receive initialization from new location either. So well, it does not receive it at all unless run as service, which code execution branch looks still heatlhy here and exhibits another issue later soon.

To resolve the problem, the fragment in CAtlServiceModuleT::Start needs the correction as shown below (within #pragma region):

        if (::StartServiceCtrlDispatcher(st) == 0)
                m_status.dwWin32ExitCode = GetLastError();
            return m_status.dwWin32ExitCode;
        }

        // local server - call Run() directly, rather than
        // from ServiceMain()
        #pragma region Run wrapped by InitializeCom/UninitializeCom
        // FIX: See http://connect.microsoft.com/VisualStudio/feedback/details/582774/catlservicemodulet-winmain-coinitialize-not-called-800401f0
#ifndef _ATL_NO_COM_SUPPORT
        HRESULT hr = E_FAIL;
        hr = T::InitializeCom();
        if (FAILED(hr))
        {
            // Ignore RPC_E_CHANGED_MODE if CLR is loaded. Error is due to CLR initializing
            // COM and InitializeCOM trying to initialize COM with different flags.
            if (hr != RPC_E_CHANGED_MODE || GetModuleHandle(_T("Mscoree.dll")) == NULL)
                return hr;
        } else
            m_bComInitialized = true;
        m_status.dwWin32ExitCode = pT->Run(nShowCmd);
        if (m_bComInitialized)
            T::UninitializeCom();
#else
        m_status.dwWin32ExitCode = pT->Run(nShowCmd);
#endif
        #pragma endregion 

        return m_status.dwWin32ExitCode;
    }

Going further from there, the introduced optimization also removed COM initialization from main process thread module Run function. Provided there earlier too through module constructor it is not longer there. So if you are doing something in application’s run when the application is set to run as service and is executed in application (where you might want to start application as a sort of a helper, or otherwise in specific mode), you need COM initialization there too.

It might be something like this:

class CFooModule :
    public CAtlServiceModuleT<CFooModule>
{
// [...]
    HRESULT Run(INT nShowCommand = SW_HIDE) throw()
    {
// [...]
            if(m_bServiceHelper)
            {
                // NOTE: Starting with Visual Studio 2010 service helper's Run receives no automatic COM initialization any longer
                nResult = InitializeCom();
                if(SUCCEEDED(nResult))
                {
                    nResult = __super::Run(nShowCommand);
                    UninitializeCom();
                }
            } else
                nResult = __super::Run(nShowCommand);
// [...]
    }
// [...]
};

Having done that we are really closer but there is still another breaking bug in ATL caused by Visual Studio 2010 changes. You start a service and its endlessly shown as being started.

It appears that ATL is failing to call SetServiceStatus(SERVICE_RUNNING) to indicated startup completion and service healthy state. Again, the bug is related to change in base class behavior. This time, the problem is caused by the fact that SetServiceStatus call was moved from CAtlServiceModuleT::Run into CAtlServiceModuleT::PreMessageLoop. However, it was moved into the section conditionally compiled with definition of _ATL_FREE_THREADED. If you are and old school guy with _ATL_APARTMENT_THREADED instead and is just converting and older project that worked before, you have a problem here again: you have to add missing call.

To fix this, the end of CAtlServiceModuleT::PreMessageLoop should be looking like this:

#else
        hr = CAtlExeModuleT<T>::PreMessageLoop(nShowCmd);

        #pragma region Missing SetServiceStatus
        if (m_bService)
        {
            LogEvent(_T("Service started/resumed"));
            SetServiceStatus(SERVICE_RUNNING);
        }
        #pragma endregion 
#endif // _ATL_FREE_THREADED

#endif    // _ATL_NO_COM_SUPPORT

        ATLASSERT(SUCCEEDED(hr));
        return hr;

Well, this is embarrassing.

A really long and annoying troubleshooting of a problem finally ended with a bug found in GetAdaptersInfo/GetAdaptersAddresses API.

It may unexpectedly fail under the following conditions:

• 32-bit application
• 64-bit operating system or /3GB feature enabled on 32-bit operating system
• hosting process is linked with /LARGEADDRESSAWARE flag or has otherwise set it in binary header
• over 2GB of RAM (in terms of Private Bytes/Virtual Size performance counters) consumed by the hosting process

The application might unexpectedly start getting ERROR_NO_DATA (232) or ERROR_NOACCESS (998) error codes instead of list of adapters. Supposedly, the responsible Microsoft component iplhlpapi.dll or one of the underlying components/APIs are incompatible with such environments (such as for example, treating 32-bit pointer with the most significant bit set as invalid or compares such pointers as signed values).

A test Visual C++ .NET 2010 application is available from SVN. More bug details on Microsoft Connect.

NOTE 1: You need to run “GetAdaptersAddressesTest /regserver” once to register the app before running it for the test.

NOTE 2: The problem does not seem to take place with 32-bit operating systems with /3GB tuning enabled.

It is as stupid as it sounds: you can add RBBS_NOGRIPPER flag to a rebar band to implement “lock controls” feature, but as soon as you start removing it – you miserably fail.

A related thread on WTL group is dated year 2006. You can find the same dated 2003. Now it’s 2011 and it’s still here.

OK, it might be some bug in controlling code etc, but here is Control Spy 2.0. Here you go:

• rebar
• RB_GETBANDINFO, RBBIM_STYLE
• add RBBS_NOGRIPPER, Send
• remove RBBS_NOGRIPPER, Send
• ouch!

WTL does a cool trick to fool the bastard:

void LockBands(bool bLock)
{
    int nBandCount = GetBandCount();
    for(int i =0; i < nBandCount; i++)
    {
        REBARBANDINFO rbbi = { RunTimeHelper::SizeOf_REBARBANDINFO() };
        rbbi.fMask = RBBIM_STYLE;
        BOOL bRet = GetBandInfo(i, &rbbi);
        ATLASSERT(bRet);

        if((rbbi.fStyle & RBBS_GRIPPERALWAYS) == 0)
        {
            rbbi.fStyle |= RBBS_GRIPPERALWAYS;
            bRet = SetBandInfo(i, &rbbi);
            ATLASSERT(bRet);
            rbbi.fStyle &= ~RBBS_GRIPPERALWAYS;
        }
        if(bLock)
            rbbi.fStyle |= RBBS_NOGRIPPER;
        else
            rbbi.fStyle &= ~RBBS_NOGRIPPER;

        bRet = SetBandInfo(i, &rbbi);
        ATLASSERT(bRet);
    }
}

Pretty cool and works… if you have one rebar only. If you have another one in the same window (e.g. mine is at bottom) – you fail again.

A nasty workaround is to nudge hosting window by resizing it, followed by a full cycle of internal layout updates:

CRect Position;
ATLVERIFY(GetWindowRect(Position));
Position.bottom++;
ATLVERIFY(MoveWindow(Position));
Position.bottom--;
ATLVERIFY(MoveWindow(Position));

If you were adding and removing ActiveX controls programmatically with ATL, you might have noticed a nasty memory leak after final release of the control. Control added, removed and a leaky block is left om heap.

Tracking simple memory leaks is not a rocket science with Microsoft C++ runtime, by adding _CRTDBG_MAP_ALLOC define, as described in Memory Leak Detection Enabling on MSDN:

#if defined(_DEBUG)
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
#endif // defined(_DEBUG)

Having that done, the problem was isolated to an extra instance of CAxHostWindow, which was excessively instantiated and never freed. Luckily, having this done you already have sufficient amount of keywords on hands to locate relevant articles on Microsoft knowledge base:

• PRB: CAxWindow Members Can Cause a Memory Leak
• How to add ActiveX controls to an ATL composite control programmatically in Visual C++

Basically, what is taking place is the behavior by design. That is, AtlAxCreateControlEx API is leaky by design, one gotta just be aware.

CComPtr<IUnknown> pContainerUnkonwn, pControlUnknown;
static LPCTSTR g_pszControlClass = _T("{8856F961-340A-11D0-A96B-00C04FD705A2}");
// NOTE: This results in two instances of CAxWindowHost being created, one of which appears to be leaked (see below), 
//       see http://support.microsoft.com/kb/218442
// 
// Detected memory leaks!
// Dumping objects ->
// {81} normal block at 0x00128928, 244 bytes long.
// Data: <x .       .     > 78 09 2E 01 01 00 00 00 0C 10 2E 01 00 00 00 00 
// Object dump complete.
ATLENSURE_SUCCEEDED(AtlAxCreateControlEx(CT2COLE(g_pszControlClass), m_hWnd, NULL, &pContainerUnkonwn, &pControlUnknown));

A workaround here is suggested by Q218442 and is a replacement of AtlAx* API call with a CAxWindow class:

CComPtr<IUnknown> pContainerUnkonwn, pControlUnknown;
static LPCTSTR g_pszControlClass = _T("{8856F961-340A-11D0-A96B-00C04FD705A2}");
CAxWindow AxWindow;
AxWindow.Attach(m_hWnd);
ATLENSURE_SUCCEEDED(AxWindow.CreateControlEx(CT2COLE(g_pszControlClass), NULL, &pContainerUnkonwn, &pControlUnknown));

The happiness of leak resolution here might be somewhat diluted by a weird assertion failure coming out from the deepness of ATL. The problem comes from ~CAutoStackPtr destructor (undocumented) trying to _freea a memory block allocated by caller’s _malloca. While caller’s _malloca is converted to plain heap allocation (_malloc_dbg) for debugging purposes, _freea is still looking for security cookie which never existed. Hence, assertion failure:

_ASSERTE(("Corrupted pointer passed to _freea", 0));

Unfortunately, it appears that _CRTDBG_MAP_ALLOC macro is not compatible with _malloca/_freea, the problem exists in versions of Visual C++ 2008 (at least) through 2010 SP1.

Vote for the problem on MS Connect: _malloca/_freea mismatch with malloc debug builds (_CRTDBG_MAP_ALLOC)

It was a sort of ridiculous problem: an attempt to instantiate a Microsoft.Jet.OLEDB.4.0 object failed with error. Still some applications are still running without problems connecting to Jet databases, how comes?

There has been a number of posts on Internet, but none of the top ones appeared to be relevant.

The problem is reproduced extremely simple:

int _tmain(int argc, _TCHAR* argv[])
{
    ATLVERIFY(SUCCEEDED(CoInitialize(NULL)));
    {
        //CoLoadLibrary(L"C:\\Program Files (x86)\\Common Files\\System\\Ole DB\\oledb32.dll", TRUE);
        CComPtr<IDBInitialize> pDbInitialize;
        const HRESULT nResult = pDbInitialize.CoCreateInstance(L"Microsoft.Jet.OLEDB.4.0", NULL, CLSCTX_INPROC_SERVER);
        _tprintf(_T("nResult 0x%08x\n"), nResult);
    }
    CoUninitialize();
    return 0;
}

Oops, the code gives error REGDB_E_CLASSNOTREG 0×80040154 “Class not registered”. It looked like system was unable to locate one of the internally used DLLs – oledb32.dll, and if we help by uncommenting the line commented in the code snippet above, the error changes to ERROR_MOD_NOT_FOUND 0x8007007e “The specified module could not be found”.

The problem appears to be that one of the system components, which is involved, “Microsoft OLE DB Data Conversion Library” is registered with the system using a REG_EXPAND_SZ value, to be located using path “%CommonProgramFiles(x86)%\System\Ole DB\oledb32.dll”. It is obvious that CommonProgramFiles(x86) is placeholder to be expanded, but does the expansion succeed?

On my system there was no such environment string, so expansion failed. There however was CommonProgramFiles, and it pointed to correct location. So, this hot fix makes it work right:

BOOL UpdateEnvironmentVariables()
{
    CString sCommonProgramFilesX86;
    sCommonProgramFilesX86.GetEnvironmentVariable(_T("CommonProgramFiles(x86)"));
    if(sCommonProgramFilesX86.IsEmpty())
    {
        CString sCommonProgramFiles;
        sCommonProgramFiles.GetEnvironmentVariable(_T("CommonProgramFiles"));
        if(!sCommonProgramFiles.IsEmpty())
        {
            ATLVERIFY(SetEnvironmentVariable(_T("CommonProgramFiles(x86)"), sCommonProgramFiles));
            return TRUE;
        }
    }
    return FALSE;
}

int _tmain(int argc, _TCHAR* argv[])
{
    ATLVERIFY(SUCCEEDED(CoInitialize(NULL)));
    {
        UpdateEnvironmentVariables();
        CComPtr<IDBInitialize> pDbInitialize;
        const HRESULT nResult = pDbInitialize.CoCreateInstance(L"Microsoft.Jet.OLEDB.4.0", NULL, CLSCTX_INPROC_SERVER);
        _tprintf(_T("nResult 0x%08x\n"), nResult);
    }
    CoUninitialize();
    return 0;
} 

I am under impression that the problem is not specific to Windows 7. I remember similar symptoms on Windows XP system, but they were not critical and not investigated to solution. It rather seems that OLE DB component is registering itself using a reference to environment variable which is not present on a number of systems.