Tracking Unusable Virtual Memory in VMMap

July 22, 2014

one comment

VMMap is a great Sysinternals tool that can visualize the virtual memory of a specific process and help understand what memory is being used for. It has specific reports for thread stacks, images, Win32 heaps, and GC heaps. Occasionally, VMMap will report unusable virtual memory, which is not the same as free memory. Here’s an example of a VMMap report for a 32-bit process (which has a total of 2GB virtual memory):

vmmap-table

Where is this “unusable” memory coming from, and why can’t it be used? The Windows virtual memory manager has a 64KB allocation granularity. When you allocate memory directly with VirtualAlloc and ask for less than 64KB, say 16KB, VirtualAlloc returns an address on a 64KB boundary. The first four pages (16KB) are then allocated, and the remaining 48KB are marked unused. There is no way to get at this memory by performing another allocation, because VirtualAlloc will always return an address that resides on a 64KB boundary. This memory, then, is unusable.

The fragmentation view in VMMap makes the problem even more evident. In the following screenshot, the yellow dots are 4KB regions that are allocated and usable, and the gray rectangles are 60KB regions that cannot be used. When the whole address space is full of these unusable regions, you’re not getting as much virtual memory as you’re supposed to.

vmmap-fragmentation

Fortunately, it’s pretty easy to track down the source of the offending allocations. Essentially, we’re looking for VirtualAlloc calls where the allocation size is less than 64KB (or, even better: not evenly divisible by 64KB). You can track down these allocations using VMMap itself (it has a tracing mode), or you can attach WinDbg and set a breakpoint:

0:000> bm kernelbase!VirtualAlloc* "r $t0 = poi(@esp+8); .if (@$t0 % 0x10000 != 0) { .printf \"Unusable memory will emerge after allocating %d bytes\", @$t0; kb 4 } .else { gc }"
  1: 76c03e8a          @!"KERNELBASE!VirtualAllocExNuma"
  2: 76bcd532          @!"KERNELBASE!VirtualAlloc"
  3: 76c03e66          @!"KERNELBASE!VirtualAllocEx"
0:000> g
Unusable memory will emerge after allocating 4096 bytes
ChildEBP RetAddr  Args to Child              
00defd48 010a4e34 00000000 00001000 00003000 KERNELBASE!VirtualAlloc
00defe2c 010a5f25 00000000 00000000 7eaa7000 FourKBLeak!allocate_small+0x34
00deff18 010a6989 00000001 012ba870 012bae98 FourKBLeak!main+0x35
00deff68 010a6b7d 00deff7c 7529919f 7eaa7000 FourKBLeak!__tmainCRTStartup+0x199
...

This breakpoint makes sure that the allocation size passed to VirtualAlloc is evenly divisible by 64KB. If it isn’t, the breakpoint stops and prints out the offending allocation and call stack; otherwise, it continues execution. This should make it very easy to catch the source of the small allocations, and hopefully fix them.


I am posting short links and updates on Twitter as well as on this blog. You can follow me: @goldshtn

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

one comment

  1. Pingback: The Morning Brew - Chris Alcock » The Morning Brew #1657