InfoQ: A Formal Performance Tuning Methodology: Wait-Based Tuning
Although the author writes about java applications the concepts are the same for .NET and supported to some extent by Visual Studio Test Edition
Here's the summery:
"Performance tuning was once more “art” than “science”, but after a combination of abstract analysis and trial-and-error, wait-based tuning has proven to make the exercise far more scientific and far more effective. Wait-based tuning begins by performing a wait-point analysis of an application’s architecture in order to identify technologies employed by the architecture that can potentially cause a request to wait. Wait-points come in two flavors: tier-based wait-points, which are indicative of any transition between application tiers, and technology-based wait-points, which are technology features such as caches, pools, and messaging infrastructures that can improve or hinder performance. With a set of wait-points identified, the tuning process is implemented by opening all tier-based wait-points and external dependency pools, generating balanced and representative load against the application, and tuning backwards, or tightening wait-points to maximize the performance of a request’s weakest link, but without saturating it.
Wait-based tuning has proven itself time and time again in real-world production environments to not only be effective, but to allow a performance engineer to realize measurable performance improvements very quickly."
Read more