Special consideration for the ThreadAbort exception
As a rule, I never want to see an unhandled exception crashing my application with the .Net error dialog and without logging the error. Therefor I always hook both the AppDomain.UnhandledException and the Application.ThreadException events to catch those exceptions, log them, and close the application gracefully.
Turns out I overlooked one small detail - the ThreadAbort exception, raised by a call to the Thread.Abort method. It's not a best practice to use it (you should let the thread shut down itself), but it may be used by a 3rd party code, so it can't be ignored.
However, since it's not an actual exception there is no logic in shutting down the application due to this exception.