A year and a half ago I touched on the subject of debugging process startup, such as the startup of Windows Services, using the GFlags utility (the ImageFileExecutionOptions registry key).
The general idea is to rely on the Windows loader to launch a debugger instead of the debugged process, and trace your way through the process startup code. Unfortunately, this relies on the debugged process to run in the same session as you—otherwise, you won’t be able to actually see the debugger.
Starting from Windows Vista, Windows services are isolated into a separate session to which you do not have access when you are logged onto the system. The debugger is launched within this session as well, which produces the undesired result of having the service stuck waiting for the debugger, and the debugger stuck waiting for your input which you cannot provide. (To learn more about Session 0 Isolation, check out the trusty Windows 7 Training Kit which covers several application compatibility topics with detailed code examples.)
What can you do to debug service startup on Windows Vista or newer OS versions? All you need is to fire a remote debugging server that debugs the service, and connect to its debugging session from a debugging client. Assuming that your Debugging Tools for Windows installation resides in C:\Debuggers, you can configure the following as the Debugger string in GFlags:
C:\Debuggers\ntsd.exe -server tcp:port=10000 -noio
When you start the service, you will notice an ntsd.exe instance launch in session 0; you’ll need to connect to the debugging session quickly by launching WinDbg (or NTSD), choosing File | Connect to Remote Session, and providing tcp:port=10000 as the transport. (Note that when debugging service startup, you might want to increase the service startup timeout to prevent the SCM from giving up on your service. Unfortunately, this is a global setting; another option is to request additional time from code.)