A technique for debugging complex parsers.

Created by Peter Kankowski
Last changed
Contributors: IceStudent, Theimeto, Karlis, and Ace
Filed under Win32 programming

Share on social sitesReddit Digg Delicious Buzz Facebook Twitter

Debugging buffer overflows

When debugging a complex parser, you have to deal with buffer overflows. Consider a parser that reads 1-2 characters past the end of buffer when some specific input is provided. The bytes after the buffer are usually readable, but if they are not, the program crashes. It is hard to catch the bug, because it occurres only with some specific memory layout.

In such cases, you need to make the bug occur always. You program will crash more often, but that's good because it helps you to find the root of the problem.

Here is a debugging version of malloc that allocates the buffer at the very end of memory page, so reading one extra character leads to access violation error:

VOID* DbgMalloc(SIZE_T size) {
    SYSTEM_INFO sys_info;

    INT_PTR page = sys_info.dwPageSize;
    assert((page & (page - 1)) == 0); // page size is the power of two
    SIZE_T rounded_size = (size + page - 1) & (-page); // round up to the page boundary

    BYTE* start = (BYTE*)VirtualAlloc(NULL, rounded_size + page, MEM_COMMIT, PAGE_READWRITE);
    DWORD old_protect;
    BOOL res = VirtualProtect(start + rounded_size, page, PAGE_NOACCESS, &old_protect);
    assert(res); UNREFERENCED_PARAMETER(res);
    return start + (rounded_size - size);

Now you can allocate memory with this function when running automatic tests. If the test coverage is close to 100%, you will able to catch all buffer overflow errors.

Download the source code (10 KB)

Peter Kankowski
Peter Kankowski

About the author

Peter is the developer of Aba Search and Replace, a tool for replacing text in multiple files. He likes to program in C with a bit of C++, also in x86 assembly language, Python, and PHP.

Created by Peter Kankowski
Last changed
Contributors: IceStudent, Theimeto, Karlis, and Ace


I use self-made alloc-and-protect-last-page strategy to detect such situations. It looks like DbgMalloc, but allocates one page more and sets last page protection to "no access" to produce a guaranteed exception. Also, this exception can be isolated from other.
Peter Kankowski,
Thank you very much, it's an excellent idea. I implemented it (VirtualProtect function is used to set the next page protection to PAGE_NOACCESS).
No need to reinvent the wheel: gflags -p /enable myprogram.exe has been around for ages.
See http://support.microsoft.com/kb/286470
Peter Kankowski,
Thank you very much, it's a nice tool if you use Win32 allocation functions (e.g., HeapAlloc).

MSVC++ malloc uses it's own heap manager for small blocks of data, so gflags will not work for them. In this case, you have to "reinvent the wheel" and use DbgMalloc.
IIRC MSVC malloc does not use own heap manager since msvc2003 unless _CRTDBG_MAP_ALLOC is defined (_DEBUG only of course).
Peter Kankowski,
I checked it and found that you are right: modern versions use HeapAlloc by default. (I viewed the source code of MSVC 6.0 CRT previously, that's why I made the mistake.)

So, gflags can be useful. I will try this tool, thank you again.
You should also check "Application Verifier" from Microsoft, it has the memory debugging functionality of gflags + some other good features. It also comes with a handy UI for quick setup. That tool really helps you to find bugs amazingly fast.


The debug heap of MSVS has some handy functions as well, for example _CrtSetDbgFlag.
> MSVC++ malloc uses it's own heap manager for small blocks of data
> I viewed the source code of MSVC 6.0 CRT

Even in 6.0 the CRT doesn't use its own manager for small blocks unless it's explicitly turned on. Apparently turning it on can often improve the overall performance even in the code built with the later compiler versions if the code does a lot of small allocations.

(Out of topic: There's a simple rule, convenient for us non native English speakers: If you mean “it is” or “it has” write “it's”, otherwise write “its”)
Peter Kankowski,
Ace, thank you for the tip; it's helpful, as always :).

Karlis, thank you very much; I will try AppVerifier. There was an article about CRT debug heap functions on this blog.
Some years ago, I did very similar work for MSVC6, automating malloc/new replacements. I documented the work here: http://ttsiodras.googlepages.com/HeapCheck.html

I also found a way to do this at link time (i.e. completely automated), but I never got around to documenting it.

Your name: