Fooling Around // Software Production Line

Microsoft C++ compiler offers a buffer security check option since quite a long ago. It adds so called “security cookies” onto stack and checks them to be not modified by code in order to early-detect stack buffer overflow conditions. The compiler option /GS is enabled by default in Release configurations and is not applicable to debug code (which has optimizations disabled).

Here comes the trouble: release code might show unexpected message box “Buffer overrun detected!”. I have never seen such after years of development and tens of thousands of lines of code written, but luckily there are colleagues who can contribute a challenging issue to investigate.

The message is shown through C runtime’s __crtMessageBoxA function and is not only annoying and angering the customer, it also stops application operation and terminates it:

By default, an application that fails a security check displays a dialog box that states “Buffer overrun detected!” When the dialog box is closed, the application terminates.

However it is possible to override this behavior:

The CRT library offers the developer an option to use a different handler that would react to the buffer overrun in a manner more sensible to the application. The function, __set_security_error_handler, is used to install the user handler by storing the user-defined handler in the user_handler variable

In my case we would be overriding this to trace call stack at the issue point. However, let us see first what other conditions might cause a message box. CRT source has the following occurrences for__crtMessageBoxA:

• assert.c - assertion failure, which normally only applies to debug code
• crt0msg.c - _NMSG_WRITE function, typically for a fatal run time error messages
• crtlib.c - _CRTDLL_INIT fails to start on a Win32s system (VER_PLATFORM_WIN32s)
• crtmbox.c - implementation of __crtMessageBoxA (which is, among other things binds to API functions on runtime instead of static linking)
• dbgrpt.c - CrtMessageWindow function, applies to debug code only
• secfail.c - __security_error_handler function, the /GS thing

So it appears that hopefully /GS failure is the only surprising condition to display an unexpected message box.

OK, let us just see what we got finally here. Using the MSDN sample code and overriding default check failure handler:

int main()
{
	_set_security_error_handler(report_failure); // <<---------------- Override
	// declare buffer that is bigger than expected
	char large_buffer[] = "This string is longer than 10 characters!!";
	vulnerable(large_buffer);
}

Private handler is:

void __cdecl report_failure(int code, void*)
{
	ATLTRACE2(atlTraceGeneral, 0, _T("Security Error Handler: Code %d\n"), code);
	CONTEXT Context = { CONTEXT_CONTROL };
	if(GetCurrentThreadContext(&Context))
		CDebugTraceCallStack::TraceCallStack(Context);
}

and finally output:

BufferSecurityCheckSample.cpp(20): report_failure: Security Error Handler: Code 1
  BufferSecurityCheckSample!0x00402a2d report_failure (+ 125) [c:\buffersecuritychecksample\buffersecuritychecksample.cpp, 22] (+ 10) @00400000
  BufferSecurityCheckSample!0x00403d56 __security_error_handler (+ 48) [f:\vs70builds\6030\vc\crtbld\crt\src\secfail.c, 79] (+ 6) @00400000
  BufferSecurityCheckSample!0x00402e1d report_failure (+ 25) [f:\vs70builds\6030\vc\crtbld\crt\src\secchk.c, 117] (+ 9) @00400000
  BufferSecurityCheckSample!0x0040142c vulnerable (+ 44) [c:\buffersecuritychecksample\buffersecuritychecksample.cpp, 36] (+ 11) @00400000
  BufferSecurityCheckSample!0x00404019 mainCRTStartup (+ 371) [f:\vs70builds\6030\vc\crtbld\crt\src\crt0.c, 259] (+ 18) @00400000
  kernel32!0x7c817067 RegisterWaitForInputIdle (+ 73) @7c800000

The log file would immediately show the source of the run time problem.

See Also:

• /GS (Buffer Security Check)
• Compiler Security Checks In Depth

Since ATL version 3, and actually it probably existed even since even earlier time, I have seen Microsoft Visual Studio (and its predecessor Visual C++) generating .RGS registration script with hardcoded CLSID value, ProgID values and other strings entered through IDE wizard. While it seems to work nice at the moment of generation, it still gives the problem to lose sync between identifiers if code is copied between projects or a different way. It might be an oversight originally, but was not it the obvious thing to do later to avoid possible typos of inaccurate user? Microsoft Visua Studio 2008 still generates the same code.

Previous versions of development environment offered attributed code option which in fact took care of this problem very well. Although there has been a lot of negative feedback written about C++ attributes, this was a kind of problem attributes left no chances to break: CLSID was specified once only in C++ COM class attribute and the rest of the deal was generated automatically, including IDL and registration sequence.

It is especially unclear why it was left this way because internal ATL classes already have all necessary capabilities to process wildcards. For example, default .RGS script contains an entry for COM class type library, HKCR\CLSID\{…} key, “TypeLib” value. Why would hardcode it if it can be specified through a replacement:

HKCR
{
	NoRemove CLSID
	{
		ForceRemove %CLSID% = s '...'
		{
			'TypeLib' = s '%LIBID%'

And the replacement is so easily added overriding CAtlModule’s virtual AddCommonRGSReplacements:

class CFooModule :
	public CAtlDllModuleT<CFooModule>
{
...
// CAtlModule
	HRESULT AddCommonRGSReplacements(IRegistrarBase* pRegistrar) throw()
	{
		_ATLTRY
		{
			ATLENSURE_SUCCEEDED(__super::AddCommonRGSReplacements(pRegistrar));
			ATLASSERT(!IsEqualGUID(m_libid, GUID_NULL));
			ATLENSURE_SUCCEEDED(pRegistrar->AddReplacement(L"LIBID", StringFromIdentifier(m_libid)));
		}
		_ATLCATCH(Exception)
		{
			return Exception;
		}
		return S_OK;
	}

I thought it would be implemented natively in ATL long ago, along with per-coclass replacements like %CLSID%.

This is how I like my registration script:

HKCR
{
	%PROGID% = s '%DESCRIPTION%'
	{
		CLSID = s '%CLSID%'
	}
	%VIPROGID% = s '%DESCRIPTION%'
	{
		CLSID = s '%CLSID%'
		CurVer = s '%PROGID%'
	}
	NoRemove CLSID
	{
		ForceRemove %CLSID% = s '%DESCRIPTION%'
		{
			ProgID = s '%PROGID%'
			VersionIndependentProgID = s '%VIPROGID%'
			ForceRemove 'Programmable'
			InprocServer32 = s '%MODULE%'
			{
				val ThreadingModel = s 'Both'
			}
			TypeLib = s '%LIBID%'
		}
	}
}

Although I personally have no experience with tricky DirectShow filters that decide on possibility of connection not only looking at media type and other capabilities, there has been a number of cases mentioned that certain video decoders will only connect to renderers or video renderers in order to avoid interception of decoded data.

It was recently mentioned that one of such filters checked Misc Flags obtained through IAMFilterMiscFlags interface to make sure it is connected to renderer downstream. Before we proceed, let us make it clear that it is silly and can only protect from beginner, a complete newbie. At the very least it should have checked peer filter’s CLSID and compare against white list of stock video renderer: Video Renderer, Video Mixing Rendeder 7, Video Mixing Rendeder 9, Enhanced Video Renderer. A custom video renderer or even just a transformation filter with a renderer flag would be able to receive decoded data and make it available for any further processing.

What I am going to do is to take Sample Grabber Filter as a base and make a new filter from it which will only connect to renderer (it will check the flags as described above). Then another filter, which will also use Sample Grabber Filter base, will accurately fool the first one and connect to output of the first filter and will render video to complete the graph.

Read more »

A simple question being asked all over again. Given IBasicVideo, IBasicVideo2, IVMRWindowlessControl or IVMRWindowlessControl9, how to save image to file? It is easy. It is a bit easier with IBasicVideo because it is possible to query this interface directly from graph’s interface, such as  IGraphBuilder, and the call will be forwarded to video renderer. This code assumes internal bitmap format is non-paletted, which I believe is always the case.

LONG nBufferSize = 0;
ATLENSURE_SUCCEEDED(pBasicVideo->GetCurrentImage(&nBufferSize, NULL));
CHeapPtr<BITMAPINFO> pBitmapInfo;
ATLENSURE_THROW(pBitmapInfo.AllocateBytes(nBufferSize), E_OUTOFMEMORY);
ATLENSURE_SUCCEEDED(pBasicVideo->GetCurrentImage(&nBufferSize, (LONG*) (BITMAPINFO*) pBitmapInfo));
const BYTE* pnData = (const BYTE*) (&pBitmapInfo->bmiHeader + 1);
// NOTE: You might wish to handle <=8 bpp bitmaps here
ATLASSERT(pBitmapInfo->bmiHeader.biBitCount > 8);
ATLASSERT(pBitmapInfo->bmiHeader.biCompression == BI_RGB);
ATLASSERT(pBitmapInfo->bmiHeader.biSizeImage);
BITMAPFILEHEADER BitmapFileHeader;
ZeroMemory(&BitmapFileHeader, sizeof BitmapFileHeader);
BitmapFileHeader.bfType = 'MB';
BitmapFileHeader.bfSize = (DWORD) (sizeof (BITMAPFILEHEADER) + pBitmapInfo->bmiHeader.biSize + pBitmapInfo->bmiHeader.biSizeImage);
BitmapFileHeader.bfOffBits = (DWORD) (sizeof (BITMAPFILEHEADER) + pBitmapInfo->bmiHeader.biSize);
ATLENSURE_SUCCEEDED(File.Write(&BitmapFileHeader, sizeof BitmapFileHeader));
ATLENSURE_SUCCEEDED(File.Write(&pBitmapInfo->bmiHeader, pBitmapInfo->bmiHeader.biSize));
ATLENSURE_SUCCEEDED(File.Write(pnData, (DWORD) pBitmapInfo->bmiHeader.biSizeImage));

ATL regular expression code adds dump/debug capabilities with ATL_REGEXP_DUMP preprocessor definition defined (undocumented). Dump output is directed to stdout, so if the application is not console, it has to be redirected in advance, e.g. into a $(BinaryPath)$(BinaryName)-atlrx.h file.

	CAtlRegExp<> Expression; LPCTSTR pszPattern; BOOL bCaseSensitive;
#if defined(ATL_REGEXP_DUMP)
	REParseError Error = Expression.Parse(pszPattern, bCaseSensitive);
	if(Error != REPARSE_ERROR_OK)
	{
		FILE* pStream = NULL;
		if(!GetStdHandle(STD_OUTPUT_HANDLE))
		{
			TCHAR pszPath[MAX_PATH] = { 0 };
			ATLVERIFY(GetModuleFileName(_AtlBaseModule.GetModuleInstance(), pszPath, _countof(pszPath)));
			RemoveExtension(pszPath); // ATLPath::
			_tcscat_s(pszPath, _countof(pszPath), _T("-atlrx.txt"));
			ATLVERIFY(freopen_s(&pStream, CStringA(pszPath), "w", stdout) == 0);
		}
		_tprintf(_T("Error %d\n"), Error);
		Expression.Dump();
		if(pStream)
			ATLVERIFY(fclose(pStream) == 0);
		ATLASSERT(FALSE); // Break into debugger
	}
#else
	ATLVERIFY(Expression.Parse(pszPattern, bCaseSensitive) == REPARSE_ERROR_OK);
#endif // defined(ATL_REGEXP_DUMP)

There is also ATLRX_DEBUG definition, which enables other debug capabilities. CAtlRegExp object is given a virtual function OnDebugEvent through which it prints comments on CAtlRegExp::Match process, which can be also redirected to file similar way.

Given a DirectShow filter graph, what media samples are being sent through particular graph point? DumpMediaSamples utility gives a fast answer to this question. It prints out connection media type details (with details of VIDEOINFOHEADER, VIDEOINFOHEADER2 and WAVEFORMATEX structures corresponding to FORMAT_VideoInfo, FORMAT_VideoInfo2 and FORMAT_WaveFormatEx format types) and IMediaSample details obtained through AM_SAMPLE2_PROPERTIES structure.

First of all, it is necessary to create a graph of interest using GraphEdit utility. At the point of interest it is necessary to insert [an uninitialized] Sample Grabber Filter with the filter name “SampleGrabber” (this is the default name but if you add second filter which will be given a different name and remove first filter then, the utility would fail).

The graph may look like this:

Read more »

See beginning in microsoft.public.win32.programmer.directx.video newsgroup.

This sample is demonstrating COM aggregation to embed an existing filter an re-expose it as a new filter having inner filter pre-initialized.

The Visual Studio C++.NET 2008 projects contains a DirectShow filter class that registers itself under Video Capture Sources category and embeds File Source (Async) Filter inside initialized to stream clock.avi file from Windows directory.

Read more »