ATL ActiveX control hosting memory leak, _freea assertion failure and more

If you were adding and removing ActiveX controls programmatically with ATL, you might have noticed a nasty memory leak after final release of the control. Control added, removed and a leaky block is left om heap.

Tracking simple memory leaks is not a rocket science with Microsoft C++ runtime, by adding _CRTDBG_MAP_ALLOC define, as described in Memory Leak Detection Enabling on MSDN:

#if defined(_DEBUG)
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
#endif // defined(_DEBUG)

Having that done, the problem was isolated to an extra instance of CAxHostWindow, which was excessively instantiated and never freed. Luckily, having this done you already have sufficient amount of keywords on hands to locate relevant articles on Microsoft knowledge base:

• PRB: CAxWindow Members Can Cause a Memory Leak
• How to add ActiveX controls to an ATL composite control programmatically in Visual C++

Basically, what is taking place is the behavior by design. That is, AtlAxCreateControlEx API is leaky by design, one gotta just be aware.

CComPtr<IUnknown> pContainerUnkonwn, pControlUnknown;
static LPCTSTR g_pszControlClass = _T("{8856F961-340A-11D0-A96B-00C04FD705A2}");
// NOTE: This results in two instances of CAxWindowHost being created, one of which appears to be leaked (see below), 
//       see http://support.microsoft.com/kb/218442
// 
// Detected memory leaks!
// Dumping objects ->
// {81} normal block at 0x00128928, 244 bytes long.
// Data: <x .       .     > 78 09 2E 01 01 00 00 00 0C 10 2E 01 00 00 00 00 
// Object dump complete.
ATLENSURE_SUCCEEDED(AtlAxCreateControlEx(CT2COLE(g_pszControlClass), m_hWnd, NULL, &pContainerUnkonwn, &pControlUnknown));

A workaround here is suggested by Q218442 and is a replacement of AtlAx* API call with a CAxWindow class:

CComPtr<IUnknown> pContainerUnkonwn, pControlUnknown;
static LPCTSTR g_pszControlClass = _T("{8856F961-340A-11D0-A96B-00C04FD705A2}");
CAxWindow AxWindow;
AxWindow.Attach(m_hWnd);
ATLENSURE_SUCCEEDED(AxWindow.CreateControlEx(CT2COLE(g_pszControlClass), NULL, &pContainerUnkonwn, &pControlUnknown));

The happiness of leak resolution here might be somewhat diluted by a weird assertion failure coming out from the deepness of ATL. The problem comes from ~CAutoStackPtr destructor (undocumented) trying to _freea a memory block allocated by caller’s _malloca. While caller’s _malloca is converted to plain heap allocation (_malloc_dbg) for debugging purposes, _freea is still looking for security cookie which never existed. Hence, assertion failure:

_ASSERTE(("Corrupted pointer passed to _freea", 0));

Unfortunately, it appears that _CRTDBG_MAP_ALLOC macro is not compatible with _malloca/_freea, the problem exists in versions of Visual C++ 2008 (at least) through 2010 SP1.

Vote for the problem on MS Connect: _malloca/_freea mismatch with malloc debug builds (_CRTDBG_MAP_ALLOC)