Myth-Busting: Memory-Mapped Files and Shared Memory on Windows

Thursday, January 21, 2016

I am often asked why memory-mapped files can be more efficient than plain read/write I/O calls, and whether shared memory is slower than private memory. These seemingly unrelated mechanisms share a common implementation in the Windows kernel, known as section objects or file mapping objects. Yes, this shared implementation powers memory pages that are shared across multiple processes (by name) as well as file regions mapped to memory pages (even in a single process). If you're interested in a thorough discussion of how section objects work, I must refer you to Windows Internals, 6th Edition. But if you're only here for...
no comments

Windows Process Memory Usage Demystified

Tuesday, January 5, 2016

"How much memory is your process using?" -- I bet you were asked that question, or asked it yourself, more times than you can remember. But what do you really mean by memory? I never thought it would be hard to find a definitive resource for what the various memory usage counters mean for a Windows process. But try it: Google "Windows Task Manager memory columns" and you'll see confusing, conflicting, inconsistent, unclear explanations of what the different metrics represent. If we can't even agree on what "working set" or "commit size" means, how can we ever monitor our Windows...

Large Win32 Heap Allocations Go Directly to VirtualAlloc

Friday, October 23, 2015

The Windows heap manager was designed to avoid the overhead of having to allocate virtual memory directly with VirtualAlloc, among other things. If you only need a 20-byte object, it's a waste to call a system service (involving a user-kernel transition) and allocate a full page. The heap manager avoids that overhead by managing large blocks of virtual memory in user mode---it is implemented in ntdll.dll. However, when you allocate particularly large blocks of memory (>= 512KB at the time of writing), the heap manager doesn't see a reason to interfere, so it just forwards your request to VirtualAlloc. It still knows about...

Tracking Unusable Virtual Memory in VMMap

Tuesday, July 22, 2014

VMMap is a great Sysinternals tool that can visualize the virtual memory of a specific process and help understand what memory is being used for. It has specific reports for thread stacks, images, Win32 heaps, and GC heaps. Occasionally, VMMap will report unusable virtual memory, which is not the same as free memory. Here's an example of a VMMap report for a 32-bit process (which has a total of 2GB virtual memory): Where is this "unusable" memory coming from, and why can't it be used? The Windows virtual memory manager has a 64KB allocation granularity. When you allocate memory directly...
one comment

Diagnosing a Non-Paged Pool Leak with Asynchronous I/O

Thursday, February 20, 2014

I spent a few hours last week chasing a non-paged pool leak caused by a simple Win32 application. After some divide-and-conquer work, we were able to pinpoint the line of code causing the leak -- a seemingly innocent WSARecv call that performs an asynchronous socket receive. How can a user-mode application cause a non-paged pool leak that quickly accumulates to dozens of megabytes of kernel memory? Read on for the details. If you'd like to replicate this problem yourself and experiment with the diagnostic process described below, use the following gist. It's 54 lines of code including error handling and #includes. Capturing...
no comments

An Exercise in Virtual to Physical Memory Translation

Monday, September 23, 2013

In this post, we will explore virtual address translation through a real-world example. If you’d like, the instructions here should be sufficiently detailed for you to perform this experiment on your own, with a kernel debugger (a local kernel debugger session or even LiveKD is sufficient). Let’s start with the basics. For a great walkthrough of how memory translation works on x86-64 and x86, you should read Luke Hutchinson’s blog post. Alternatively, Windows Internals (6th edition) contains an even more detailed description of address translation in the Memory Manager chapter (volume 2). Now let’s go ahead and...
no comments

Windows Memory Manager’s Preferential Treatment of Access Faults in the Interlocked Singly Linked List Pop Implementation

Saturday, August 11, 2012

Special treatment is always a good thing, especially when you find that the kernel part of Windows gives preferential treatment to its user mode counterpart. Executive summary: It turns out that the Windows memory manager’s access fault handler, which handles the exception raised by the CPU when an invalid memory access occurs, has a special case for an expected access violation that might be raised when popping an item from an interlocked singly linked list (SList). The rationale for this special treatment is probably the prohibitive cost of setting up an exception handling frame in the popping function....
one comment

Baby Steps in Windows Device Driver Development: Part 6, Hiding Processes

Tuesday, August 16, 2011

Last time around, we’ve seen how to do something slightly useful in our driver. This time, we’ll simulate a technique used over ten years ago by Windows kernel rootkits to hide a process from tools such as Task Manager. First, some background: the Windows scheduler doesn’t need process information to run code. The scheduler needs access only to threads—threads ready for execution are stored in a set of ready queues. When a thread enters a wait state, the system tracks its information using _KWAIT_BLOCK structures, which again don’t require access to processes. Still, the system keeps track...

Baby Steps in Windows Device Driver Development: Part 5, Monitoring Processes

Saturday, July 2, 2011

The first remotely useful thing we are going to do with our newly acquired knowledge about device driver development is to register a callback for whenever a process is created, and output the information on the parent and child processes. (Frankly, this can be accomplished quite as easily using the WMI Win32_ProcessStartTrace event class, but bear with me here.) The PsSetCreateProcessNotifyRoutine function is a service provided by the process manager in the executive, which allows us to register a callback for when processes are created. This can be useful in the context of a security product, auditing software,...
no comments

SELA Developer Days 2011 – Windows Internals

Thursday, June 30, 2011

The SELA Developer Days conference has been adjourned :-) My one-day session today, titled Windows Internals for Busy Developers, was something I came up with a couple of months ago and was sure it wouldn’t be popular – after all, we have a five-day Windows Internals course and most people interested enough in the subject would want to attend the full training with labs, demos, and detailed walkthroughs of all Windows components. I was surprised to find 16 attendees in my class, all eager to learn about Windows architecture and components, diagnostic tools, kernel debugging, and internal...
no comments