More and more applications hit the Windows limit of available address space for 32-bit applications, and the whole concept becomes more important for understanding due to necessity to work things around.
A thing, which is more or less easy to understand, is that a user mode 32-bit application can address 2^32 addresses. The addresses are not directly physical RAM and the operating system is responsible for management of the mapping addresses into RAM as a part of virtual memory manager operation. Paged memory organization is well documented on MSDN, and the questions has been raised numerous times. An interesting question is whether a 32-bit application can effectively manage memory amounts exceeding address space limits.
Back in 80386 times, the systems could address megabytes of RAM in 16-bit code through XMS and EMS services. The application could access “high” memory addresses by requesting mapping portions of RAM into lower megabyte address space. In some way similar technique is also here for 32-bit applications in Windows through use of file mappings.
A regular memory backed file mapping requests Windows to reserve a memory block which becomes available for mapping into address space of one or more processes. Creating file mapping itself does not imply mapping and this leaves a great option for the owner to allocate more data than it can actually map into address space: if 32-bit process virtual address space is fundamentally constrained, the file mapping allocation space is more loosely limited by amount of physical memory and paging file. The application can allocate 2, 3, 4 and more gigabytes of memory – it just cannot still map it all together into address space and make it available simultaneously.
The FileMappingVirtualAddress utility does a simple thing:
- on startup it allocates (CreateFileMapping) as many 256 MB file mappings as operating system would allow, and shows it in a list
- each time a user checks a box, the application maps (MapViewOfFile) corresponding file mapping into address space; unchecking a box unmaps the view
- the caption shows currently used and maximal available virtual address space
A plain 32-bit version of the application allocated 51 blocks for me (which totals in 13 GB of memory, with 8 GB physical RAM installed in the system). The allocation takes place immediately because the operating system does not actually make all this memory prepared for use – the actual pages would be allocated and ready to use on demand when the application requires them.
The most important part made so obvious is that the 32-bit application succeeds in allocating well over 4 GB, which is maximal virtual address space it can ever get.
The virtual address space in use is only 1641 MB and another request to map an additional section with MapViewOfFile would fail (the default address space limit is 2 GB) – space fragmentation make mapping unavailable earlier than we actually use the whole space, since the API would need to allocate contiguous range of addresses to satisfy the request.
32-bit application built with /LARGEADDRESSAWARE parameter might manage to do more allocations: 64-bit versions of Windows provide 4 GB of addresses to 32-bit processes. 32-bit operating systems might also be extending the limit in case of 4GB RAM Tuning (which would typically be 3 GB of space for a process).
Finally, 64-bit build of the application is free from virtual address space limit as the limit is 8 terabytes. The mapping is again instantaneous because actual RAM will be supplied on first request to mapped pages only.
A binary [Win32, Win32 with /LARGEADDRESSAWARE, x64] and partial Visual C++ .NET 2010 partial source code are available from SVN.