/GS Buffer Security Check

Microsoft C++ compiler offers a buffer security check option since quite a long ago. It adds so called “security cookies” onto stack and checks them to be not modified by code in order to early-detect stack buffer overflow conditions. The compiler option /GS is enabled by default in Release configurations and is not applicable to debug code (which has optimizations disabled).

Here comes the trouble: release code might show unexpected message box “Buffer overrun detected!”. I have never seen such after years of development and tens of thousands of lines of code written, but luckily there are colleagues who can contribute a challenging issue to investigate.

The message is shown through C runtime’s __crtMessageBoxA function and is not only annoying and angering the customer, it also stops application operation and terminates it:

By default, an application that fails a security check displays a dialog box that states “Buffer overrun detected!” When the dialog box is closed, the application terminates.

However it is possible to override this behavior:

The CRT library offers the developer an option to use a different handler that would react to the buffer overrun in a manner more sensible to the application. The function, __set_security_error_handler, is used to install the user handler by storing the user-defined handler in the user_handler variable

In my case we would be overriding this to trace call stack at the issue point. However, let us see first what other conditions might cause a message box. CRT source has the following occurrences for__crtMessageBoxA:

• assert.c – assertion failure, which normally only applies to debug code
• crt0msg.c – _NMSG_WRITE function, typically for a fatal run time error messages
• crtlib.c – _CRTDLL_INIT fails to start on a Win32s system (VER_PLATFORM_WIN32s)
• crtmbox.c – implementation of __crtMessageBoxA (which is, among other things binds to API functions on runtime instead of static linking)
• dbgrpt.c – CrtMessageWindow function, applies to debug code only
• secfail.c – __security_error_handler function, the /GS thing

So it appears that hopefully /GS failure is the only surprising condition to display an unexpected message box.

OK, let us just see what we got finally here. Using the MSDN sample code and overriding default check failure handler:

int main()
{
	_set_security_error_handler(report_failure); // <<---------------- Override
	// declare buffer that is bigger than expected
	char large_buffer[] = "This string is longer than 10 characters!!";
	vulnerable(large_buffer);
}

Private handler is:

void __cdecl report_failure(int code, void*)
{
	ATLTRACE2(atlTraceGeneral, 0, _T("Security Error Handler: Code %d\n"), code);
	CONTEXT Context = { CONTEXT_CONTROL };
	if(GetCurrentThreadContext(&Context))
		CDebugTraceCallStack::TraceCallStack(Context);
}

and finally output:

BufferSecurityCheckSample.cpp(20): report_failure: Security Error Handler: Code 1
  BufferSecurityCheckSample!0x00402a2d report_failure (+ 125) [c:\buffersecuritychecksample\buffersecuritychecksample.cpp, 22] (+ 10) @00400000
  BufferSecurityCheckSample!0x00403d56 __security_error_handler (+ 48) [f:\vs70builds\6030\vc\crtbld\crt\src\secfail.c, 79] (+ 6) @00400000
  BufferSecurityCheckSample!0x00402e1d report_failure (+ 25) [f:\vs70builds\6030\vc\crtbld\crt\src\secchk.c, 117] (+ 9) @00400000
  BufferSecurityCheckSample!0x0040142c vulnerable (+ 44) [c:\buffersecuritychecksample\buffersecuritychecksample.cpp, 36] (+ 11) @00400000
  BufferSecurityCheckSample!0x00404019 mainCRTStartup (+ 371) [f:\vs70builds\6030\vc\crtbld\crt\src\crt0.c, 259] (+ 18) @00400000
  kernel32!0x7c817067 RegisterWaitForInputIdle (+ 73) @7c800000

The log file would immediately show the source of the run time problem.

See Also:

• /GS (Buffer Security Check)
• Compiler Security Checks In Depth