Log Process Exceptions: Filters and Email Notification

Moving on with LogProcessExceptions tool which externally monitors (debugs, to be more specific) an application of interest and captures its exceptions writing minidump files for further analysis.

This updates adds two features:

  1. Ability to filter out the exceptions of interest
  2. An email notification on exception, with or without minidump file

Filter

A new property page in the wizard provides one with an option to specify filters/rules to identify exceptions of interest. One rule per line, once exception occurred its codes are compared against rules top to bottom. The first matching rule results in decision whether to log the minidump (and issue a notification) or not. If no filter/rule applicable found, or the entire filter is empty, the exception is logged (positive).

The rule syntax is the following:

  • empty lines are ignored
  • anything to the right from // is considered to be a comment
  • meaningful line has one or more items separated by spaces
  • first item on the line is either - or + to indicate whether the rule will result in skipping the exception (minus sign) or logging in (plus sign)
  • second item is exception code, e.g. 0xC0000005 for memory access violation, 0xE06D7363 for native C++ exception (this constant has a dedicated alias “C++”
  • third item applies to C++ exceptions and matches ATL CAtlException codes, and this allows to filter out specific HRESULTs; it is also possible to specify a range of codes as shown on the screenshot above

Email Notification

Why? Some issues are long to wait, so you don’t want to have it in front of you until the issue takes place. This is when a friendly email from the application would be so much appreciated.

The application gets you that by queuing an email once exception of interest takes place. The email settings are basically those of class described on an earlier EmailTools.dll post. The whole SMTP email class is embedded into this application, and it actually deserves a dedicated post on explaining how to embed ATL COM class with persistence based on COM map, which in turn depends on type library expected to be registered, into another application which additionally runs without registration.

There is a default preset tuned for Google Mail, but there are other options as well. Note that both filter and email settings are saved into registry under HKCU to be reused in next runs of the application.

Small minidump files are attached right to the emails. Larger ones (typically those with Full Memory option checked) are only mentioned. There is a threshold setting that defines how many megabytes is OK to attach the most.

Download links:

LogProcessExceptions: Log Service Process Exceptions

One of the nasty issues with LogProcessExceptions utility was that it was unable to attach to service processes and track them to catch their exceptions.

The actual problem was that the processes were not listed in first place, so there was nothing to attach to. Access and security requirements necessary for a process to debug another process are listed in MSDN DebugActiveProcess article:

The debugger must have appropriate access to the target process, and it must be able to open the process for PROCESS_ALL_ACCESS. DebugActiveProcess can fail if the target process is created with a security descriptor that grants the debugger anything less than full access. If the debugging process has the SE_DEBUG_NAME privilege granted and enabled, it can debug any process.

The utility did enable the SE_DEBUG_NAME privilege, however it was doing it prior to starting debugging session and after the process of interest was already pointed to by user.

This was insufficient because EnumProcesses only lists service processes (not actually exactly services, but processes running in different security context) in case debug privilege is already enable by the time of the API call. The utility now enabled the privilege well in advance and list the services, so can be effectively applied to those.

Download links:

Hardware assisted memory corruption detection

So you got a memory corruption issue with a piece of software. It comes in a unique scenario along the line of having a huge pile of weird code running well most of the time and then, right out of the blue, a corruption takes place followed by unexpected code execution and unstable software state in general.

The biggest problem with memory corruption is that a fragment of code is modifying a memory block which it does not own, and it has no idea who actually is the owner of the block, while the real owner has no timely way to detect the modification. You only face the consequences being unable to capture the modification moment in first place.

To get back to the original cause, an engineer has to drop into a time machine, turn back time and step back to where the trouble took originally place. As developers are not actually given state-of-the-art time machines, the time turning step is speculative.

CVirtualHeapPtr Class: Memory with Exception-on-Write access mode

At the same time a Windows platform developer is or might be aware of virtual memory API which among other things provides user mode application with capabilities to define memory protection modes. Having this on hands opens unique opportunity to apply read-only protection (PAGE_READONLY) onto a memory block and have exception raised at the very moment of unexpected memory modification, having call stack showing up a source of the problem. I refer to this mode of operation as “hardware assisted” because the access violation exception/condition would be generated purely in hardware without any need to additionally do any address comparison in code.

Needless to say that this way is completely convenient for the developer as he does not need to patch the monstrous application all around in order to compare access addresses against read-only fragment. Instead, a block defined as read-only will be immediately available as such for the whole process almost without any performance overhead.

As ATL provides a set of memory allocator templates (CHeapPtr for heap backed memory blocks, allocated with CCRTAllocator, alternate options include CComHeapPtr with CComAllocator wrapping CoTaskMemAlloc/CoTaskMemFree API), let us make an alternate allocator option that mimic well-known class interface and would facilitate corruption detection.

Because virtual memory allocation unit is a page, and protection mode is defined for the whole page, this would be the allocation granularity. For a single allocated byte we would need to request SYSTEM_INFO::dwPageSize bytes of virtual memory. Unlike normal memory heap manager, we have no way to share pages between allocations as we would be unable to effectively apply protection modes. This would definitely increase application pressure onto virtual memory, but is still acceptable for the sacred task of troubleshooting.

We define a CVirtualAllocator class to be compatible with ATL’s CCRTAllocator, however based on VirtualAlloc/VirtualFree API. The smart pointer class over memory pointer would be defined as follows:

template <typename T>
class CVirtualHeapPtr :
    public CHeapPtr<T, CVirtualAllocator>
{
public:
// CVirtualHeapPtr
    CVirtualHeapPtr() throw();
    explicit CVirtualHeapPtr(_In_ T* pData) throw();
    VOID SetProtection(DWORD nProtection)
    {
        // TODO: ...
    }
};

The SetProtection method is to define memory protection for the memory block. Full code for the classes is available on Trac here (lines 9-132):

  • CGlobalVirtualAllocator class is a singleton querying operating system for virtual memory page size, and provides alignment method
  • CVirtualAllocator class is a CCRTAllocator-compatible allocator class
  • CVirtualHeapPtr class is smart template class wrapping a pointer to allocated memory

Use case code will be as follows. “SetProtection(PAGE_READONLY)” enables protection on memory block and turns on exception generation at the moment memory block modification attempt. “SetProtection(PAGE_READWRITE)” would restore normal mode of memory operation.

CVirtualHeapPtr<BYTE> p;
p.Allocate(2);
p[1] = 0x01;
p.SetProtection(PAGE_READONLY);
// NOTE: Compile with /EHa on order to catch the exception
_ATLTRY
{
    p[1] = 0x02;
    // NOTE: We never reach here due to exception
}
_ATLCATCHALL()
{
    // NOTE: Catching the access violation for now to be able to continue execution
}
p.SetProtection(PAGE_READWRITE);
p[1] = 0x03;

Given the information what data gets corrupt, the pointer allocator provides an efficient opportunity to detect the violation attempt. The only thing remained is to keep memory read-only, and temporarily revert to write access when the “legal” memory modification code is about to be executed.

Continue reading →

LogProcessExceptions: Automatically Create Minidump Files on C++ Exception in Monitored Process

LogProcessExceptions utility implements a very basic debugger which attaches (see DebugActiveProcess on MSDN) to a running process and monitors its exceptions. Once exception takes place the utility creates a minidump file for the process (see MiniDumpWriteDump on MSDN) so that exception condition could be investigated off-site using debugger.

If you throw C++ exceptions in your C++ code in exceptional cases which indicate necessity to log the condition and possibly check it later, the utility will get the vital information for the application running at production location in environment without fully featured debugger (note that the utility is a simple download-and-run “wizard” style application, with no installation required), or will simply track the error letting the application continue execution without pretty much of an interruption.

The tool will prompt for debuggee process, and follow with minidump type choices and the debugging operation.

Continue reading →

AMCap issue

While playing around with a camera DirectShow video source filter, AMCap, which is widely used sample and which I believed to be very stable, started crashing in an inner call CDeviceMoniker::IsEqual:

 	devenum.dll!CDeviceMoniker::IsEqual()  + 0x13 bytes
>	AmCap.exe!ChooseDevices(IMoniker * pmVideo=0x003e9a38, IMoniker * pmAudio=0x00000000)  Line 2672 + 0x2a bytes	C++
 	AmCap.exe!ChooseDevices(wchar_t * szVideo=0x0013f5b8, wchar_t * szAudio=0x0013edb0)  Line 2753 + 0x13 bytes	C++
 	AmCap.exe!AppInit(HINSTANCE__ * hInst=0x00400000, HINSTANCE__ * hPrev=0x00000000, int sw=1)  Line 379 + 0x13 bytes	C++
 	AmCap.exe!WinMain(HINSTANCE__ * hInst=0x00400000, HINSTANCE__ * hPrev=0x00000000, char * szCmdLine=0x00161f32, int sw=1)  Line 453 + 0x11 bytes	C++
 	AmCap.exe!__tmainCRTStartup()  Line 578 + 0x35 bytes	C
 	AmCap.exe!WinMainCRTStartup()  Line 403	C
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes

It appeared that while setting a checkmark on proper menu item the code does not check for the moniker tobe not NULL and a NULL IMoniker pointer passed as an argument into IMoniker::IsEqual is not checked inside devenum.dll (which is obviously a bug for an API entry).

To hotfix the problem, it is necessary to add an extra check near line 2650 of amcap.cpp:

    int i;
    for(i = 0; i < NUMELMS(gcap.rgpmVideoMenu); i++)
    {
        if(gcap.rgpmVideoMenu[i] == NULL)
            break;
        // HOTFIX: Avoid calling IMoniker::IsEqual(NULL) due to possible memory access violation
        if(!gcap.pmVideo)
            continue;
        CheckMenuItem(GetMenu(ghwndApp),
            MENU_VDEVICE0 + i,
            (S_OK == gcap.rgpmVideoMenu[i]->IsEqual(gcap.pmVideo)) ? MF_CHECKED : MF_UNCHECKED);
    }

    for(i = 0; i < NUMELMS(gcap.rgpmAudioMenu); i++)
    {
        if(gcap.rgpmAudioMenu[i] == NULL)
            break;
        // HOTFIX: Avoid calling IMoniker::IsEqual(NULL) due to possible memory access violation
        if(!gcap.pmAudio)
            continue;
        CheckMenuItem(GetMenu(ghwndApp), MENU_ADEVICE0 + i,
            (S_OK == gcap.rgpmAudioMenu[i]->IsEqual(gcap.pmAudio)) ? MF_CHECKED : MF_UNCHECKED);
    }