So you got a memory corruption issue with a piece of software. It comes in a unique scenario along the line of having a huge pile of weird code running well most of the time and then, right out of the blue, a corruption takes place followed by unexpected code execution and unstable software state in general.

The biggest problem with memory corruption is that a fragment of code is modifying a memory block which it does not own, and it has no idea who actually is the owner of the block, while the real owner has no timely way to detect the modification. You only face the consequences being unable to capture the modification moment in first place.

To get back to the original cause, an engineer has to drop into a time machine, turn back time and step back to where the trouble took originally place. As developers are not actually given state-of-the-art time machines, the time turning step is speculative.

CVirtualHeapPtr Class: Memory with Exception-on-Write access mode

At the same time a Windows platform developer is or might be aware of virtual memory API which among other things provides user mode application with capabilities to define memory protection modes. Having this on hands opens unique opportunity to apply read-only protection (PAGE_READONLY) onto a memory block and have exception raised at the very moment of unexpected memory modification, having call stack showing up a source of the problem. I refer to this mode of operation as “hardware assisted” because the access violation exception/condition would be generated purely in hardware without any need to additionally do any address comparison in code.

Needless to say that this way is completely convenient for the developer as he does not need to patch the monstrous application all around in order to compare access addresses against read-only fragment. Instead, a block defined as read-only will be immediately available as such for the whole process almost without any performance overhead.

As ATL provides a set of memory allocator templates (CHeapPtr for heap backed memory blocks, allocated with CCRTAllocator, alternate options include CComHeapPtr with CComAllocator wrapping CoTaskMemAlloc/CoTaskMemFree API), let us make an alternate allocator option that mimic well-known class interface and would facilitate corruption detection.

Because virtual memory allocation unit is a page, and protection mode is defined for the whole page, this would be the allocation granularity. For a single allocated byte we would need to request SYSTEM_INFO::dwPageSize bytes of virtual memory. Unlike normal memory heap manager, we have no way to share pages between allocations as we would be unable to effectively apply protection modes. This would definitely increase application pressure onto virtual memory, but is still acceptable for the sacred task of troubleshooting.

We define a CVirtualAllocator class to be compatible with ATL’s CCRTAllocator, however based on VirtualAlloc/VirtualFree API. The smart pointer class over memory pointer would be defined as follows:

template <typename T>
class CVirtualHeapPtr :
    public CHeapPtr<T, CVirtualAllocator>
{
public:
// CVirtualHeapPtr
    CVirtualHeapPtr() throw();
    explicit CVirtualHeapPtr(_In_ T* pData) throw();
    VOID SetProtection(DWORD nProtection)
    {
        // TODO: ...
    }
};

The SetProtection method is to define memory protection for the memory block. Full code for the classes is available on Trac here (lines 9-132):

• CGlobalVirtualAllocator class is a singleton querying operating system for virtual memory page size, and provides alignment method
• CVirtualAllocator class is a CCRTAllocator-compatible allocator class
• CVirtualHeapPtr class is smart template class wrapping a pointer to allocated memory

Use case code will be as follows. “SetProtection(PAGE_READONLY)” enables protection on memory block and turns on exception generation at the moment memory block modification attempt. “SetProtection(PAGE_READWRITE)” would restore normal mode of memory operation.

CVirtualHeapPtr<BYTE> p;
p.Allocate(2);
p[1] = 0x01;
p.SetProtection(PAGE_READONLY);
// NOTE: Compile with /EHa on order to catch the exception
_ATLTRY
{
    p[1] = 0x02;
    // NOTE: We never reach here due to exception
}
_ATLCATCHALL()
{
    // NOTE: Catching the access violation for now to be able to continue execution
}
p.SetProtection(PAGE_READWRITE);
p[1] = 0x03;

Given the information what data gets corrupt, the pointer allocator provides an efficient opportunity to detect the violation attempt. The only thing remained is to keep memory read-only, and temporarily revert to write access when the “legal” memory modification code is about to be executed.

One-shot Read/Write Protection with Guard Pages

Another option granted by memory protection modes is brought by PAGE_GUARD flag. MSDN says:

A guard page provides a one-shot alarm for memory page access. This can be useful for an application that needs to monitor the growth of large dynamic data structures. For example, there are operating systems that use guard pages to implement automatic stack checking.

Setting a guard page mode provides an additional option to trigger an exception with even read access to a protected memory block.

p.SetProtection(PAGE_READWRITE | PAGE_GUARD);
BYTE n = p[0];

CDebugHeapPtr Class: More Options to Catch Memory Corruption Conditions

While setting memory protection attributes on a memory block of interest provides unique troubleshooting opportunities, it still does not cover important typical problems with memory misuse scenarios. Those are writing immediately before the allocated block, and writing immediately after. Having array of N items, this would be writing to indices -1 and N respectively.

To address this scenarios of misuse we can extend CVirtualHeapPtr class so that it could additionally provide “sanity pages” with PAGE_NOACCESS protection at the boundary of allocation. Because virtual memory allocation is granular, we will have to have padding bytes that extend our block to the page boundary, however we have an option to put the padding bytes before or after the payload data block in order to capture after or before memory block writes respectively.

The figure below shows memory layout for the data:

Source code for the CDebugHeapPtr class is available on Trac (lines 155-). The sanity pages create a block of inaccessible addresses which immediately cause access violation exception on either read of write access attempt. Under debugger, those are shows with question marks:

The padding space is pre-initialized with hardcoded value 0×77, and the space is checked for integrity at release of memory block call.

Catching the Exceptions

Having the exceptions generated on run-time, they immediately alter application execution code path and are easy to track and catch. There is no need to bring the feature rich debugger, such as Visual Studio to the production site in order to catch the exception and environment, instead a way simpler tool such as LogProcessExceptions would be able to create a minidump file and write the state of the application. The minidump can be transferred into debugger-enabled environment for detailed check.

Visual C++ .NET 2010 source code is available from SVN.

An updated version of LogProcessExceptions utility is given an additional option to create minidump .DMP files for debugged process on user request. This is in particular useful in conjunction with flag choices (on the previous page of the wizard).

Partial Visual C++ .NET 2010 source code is available from SVN, release binary included (Win32, x64).

LogProcessExceptions utility implements a very basic debugger which attaches (see DebugActiveProcess on MSDN) to a running process and monitors its exceptions. Once exception takes place the utility creates a minidump file for the process (see MiniDumpWriteDump on MSDN) so that exception condition could be investigated off-site using debugger.

If you throw C++ exceptions in your C++ code in exceptional cases which indicate necessity to log the condition and possibly check it later, the utility will get the vital information for the application running at production location in environment without fully featured debugger (note that the utility is a simple download-and-run “wizard” style application, with no installation required), or will simply track the error letting the application continue execution without pretty much of an interruption.

The tool will prompt for debuggee process, and follow with minidump type choices and the debugging operation.

Minidump files are created automatically with the name file including:

• Excecutable Name
• Process Identifier
• Ordinal Number
• Exception Code
• C++/ATL Exception HRESULT Code (esp. for CAtlException exceptions)

The utility is expected to work with Windows XP operating system and more recent.

Partial Visual C++ .NET 2010 source code is available from SVN, release binary included (Win32, x64).

Artwork credits: http://cristoildiablo.deviantart.com/

Because Microsoft C++ Runtime library is distributed as a side-by-side assembly, GetModuleHandle and LoadLibrary is no longer available to locate the library within the process. Both API calls would return NULL handle.

HMODULE hModule = GetModuleHandle(_T("msvcr90.dll"));
HMODULE hModule = LoadLibrary(_T("msvcr90.dll"));

Still, how to locate the modules in order to, for example, be able to check DLL version? Process Status API (PSAPI) can help:

#include <psapi.h>
#pragma comment(lib, "psapi.lib")

...

DWORD nDataSize = 4 << 10; // 4K
EnumProcessModules(GetCurrentProcess(), NULL, 0, &nDataSize);
nDataSize += nDataSize / 2;
CTempBuffer<HMODULE> phModules;
ATLENSURE_THROW(phModules.AllocateBytes(nDataSize), E_OUTOFMEMORY);
if(EnumProcessModules(GetCurrentProcess(), phModules, nDataSize, &nDataSize))
{
    const SIZE_T nCount = nDataSize / sizeof *phModules;
    for(SIZE_T nIndex = 0; nIndex < nCount; nIndex++)
    {
        CPath sPath = _VersionInfoHelper::GetModulePath(phModules[nIndex]); // uses GetModuleFileName
        if(_tcsicmp(FindFileName(sPath), _T("msvcr90.dll")) == 0)
        {
            // NOTE: Here we found it
            ATLTRACE2(atlTraceGeneral, 2, _T("msvcr90.dll version is %s\n"), _VersionInfoHelper::GetVersionString(_VersionInfoHelper::GetFileVersion(sPath)));
            break;
        }
    }
}

ProcessSnapshot is a utility to take a snapshot of process call stacks, and the snapshot taken is written into a human friendly text file.

Additionally to this, the utility has been given a capability to create process minidump files, on user request. The minidump files can be used with debugger to analyze the context of the process using feature rich debug environment, esp. Microsoft Visual Studio. To create a minidump for a process, check a corresponding box and press “Take a Dump” button. A file named “<process-image-name> – <date> <time>.dmp” will be created in the directory of the utility executable.

See also:

• Minidump Files (MSDN)
• How to read the small memory dump files that Windows creates for debugging
• Post-Mortem Debugging Your Application with Minidumps and Visual Studio .NET
• How to View Windows Minidump Files

A binary [Win32, x64] and partial Visual C++ .NET 2008 source code are available from SVN.

Another example of a negligence with a cost of incompatibility and enormous amount of support time. Ahead Nero installs a number of DirectShow filters into $(Program Files)\Common Files\Ahead\DSFilter directory. One of the files is NeResize.ax and it hosts a Nero Resize filter. Let us take a closer look:

CLSID: {30002E0C-C574-481E-A5DE-90AE54A79E10} (note that Nero 8 ships the same buggy stuff with new CLSID of {3D0A27C9-B4D6-487B-AFE4-E3CABD4B81F9} – 11.05.2010)
Merit: 0×00400000 (MERIT_UNLIKELY)
Input Pin’s Media Type: major type GUID_NULL, subtype GUID_NULL
Output Pin’s Media Type: major type GUID_NULL, subtype GUID_NULL

The filter is clearly a video filter:

So the filter register itself under a merit that allows taking it during Intelligent Connect, it registers using media type wildcard which is clearly widely than the filter can affectively operate with and the most interesting part is: with certain video media types the filter crashes (memory access violation) during pin connection negotiation process. That is, inaccurate filter may be crashing third party software it has nothing to deal with at all.

*** Unhandled Exception
Process: 0x000001d4, Thread: 0x00000ce4, Date: 6/29/2009, Time: 11:20:56 AM, Application: C:\Program Files\...
Module: C:\..., Product Version: 1.7.1.1, File Version: 1.7.1.20014, File Time: 23.06.2009, 19:02
Code: 0xc0000005, Flags: 0x00000000, Address: 0x05fc6c65
Parameters: 0x00000001, 0x15be9030

** Call Stack
NeResize!05fc6c65 DllUnregisterServer +21909 @05fc0000
NeResize!05fc7888 DllUnregisterServer +25016 @05fc0000
NeResize!05fc7204 DllUnregisterServer +23348 @05fc0000

Additionally to that the filter does not allow its insertion in debugging environment, and it seems even with Visual Studio running without a debugging session active. Which means that developer may be unaware of issues until incompatibility comes up at a later stage such as testing, or at production site.

It is not the first Nero filter which is bringing real problems. Basically any user who want to keep his system far from issues while still having Nero installed, needs to do find $(Program Files)\Common Files\Ahead\DSFilter directory and immediately rename it to some ~DSFilter in order to invalidate all Nero filters registration.

A few quotes from Guidelines for Registering Filters:

Avoid specifying MEDIATYPE_None, MEDIASUBTYPE_None, or GUID_NULL in the AMOVIESETUP_MEDIATYPE information for a pin. IFilterMapper2 treats these as wildcards, which can slow the graph-building process.

Nero Resize does specify and obviously slows the system down.

Choose the lowest merit value possible. Here are some guidelines:
…
Special purpose filter; any filter that is created directly by the application: MERIT_DO_NOT_USE

Nero Resize uses higher value and thus affects proper applications.

Software developers will be safer to prevent from DirectShow Filter Graph Manager considering the buggy filter to be used during Intelligent Connect by implementing IAMGraphBuilderCallback interface.