Earlier this year I blogged about this importing of understanding and knowing how your application will perform under load. (See it here). After receiving a lot of questions from individuals wanting to learn more about "how" they can load test their applications and the tools that I use when testing applications I thought it would be good to follow up with a post to shed a bit of light on the process.
Disclaimer
Any and all products mentioned in this example are for illustration purposes only. I am NOT being paid or compensated in any way by the referred vendors for mentioning their products, nor am I receiving any affiliation or other credits by anyone following links, etc.
These tools happen to be the tools that I have experience with. They have worked well for me in the past and have provided invaluable information in testing applications large and small.
Planning!
Believe it or not, the most important part of load testing an application isn't actually in the testing process itself. It is in the planning & setup that you do BEFORE you complete the test.
Accurate Scenarios
Part of the planning process is to look at what you are wanting to test. With web applications, we have so many different methods that we can use to test. For example, simple tools can be used that ONLY request the particular server resource (.aspx page, etc) that is necessary for the main request. These types of tools are great to isolate load to your .NET components, however, they can often paint an invalid picture. When an average website could be comprised of more than 40 HTTP requests, only testing a single request in that cycle doesn't stress the server in any noticeable manner. These on-off tests can be great if you know already you have a problem with your code, but they can also send you on a wild goose chase.
Example: In a recent load test up to 50,000 concurrent users for a DNN based website we found the initial choking point to be bandwidth & throughput at the hosting facility with their Load Balancer & Network switches. This was due to the excessive HTTP content for images etc, so much that the 5 web-heads behind the scenes never went past 35% CPU and we failed at only 10,000 concurrent. We put a CDN in place for static content and hit 47,000 users before we hit another limit with a load balancer.
Keep these scenarios in mind when it comes to working with your code. Do you want to test it "as the user would test" or do you want to isolate your testing to see what is going on?
Reasonable & Expected Load Levels
The next part of planning is to analyze your environment, your available resources, the quality of your application and the load that you should reasonably be able to expect your application to handle. I wish I could say that this was an easy and exact science, but it is by far the fuzziest math of all. Your application, what it does, and the way it was developed will have a major impact.
However, a common rule of thumb that I see many people use is that anything more than 100 requests/sec with an IIS server will require either tweaking of the server or additional servers. I've managed a little over 1,000 requests/second with a bit of optimization.
Load Duration & Flow
The next item to plan for is how long will you test, and what is the flow of the test. If the test is a multi-step process, how long do you want to simulate the load? If you are looking to show a user progressing, how long should the virtual users wait before they request the next page? I cannot provide any real guidance here as each situation is different, but be sure to take this into consideration.
Monitoring, Monitoring, Monitoring & More Monitoring
Yes, that might be a bit over-dramatic, but I truly cannot stress this enough. Load testing is expensive, time-consuming and to be perfectly honest can be a real pain. Often times you are testing late at night, or on weekends to avoid impacting items during peak times, or other restrictions. You don't want to have to re-test something just because you didn't have an idea of what was going on.
What kind of monitoring am I talking about? Well, there are a number of things that you might want to have available depending on the nature of the tests. A few tools that I've used are included below.
- NewRelic - A great utility for monitoring application performance, response time, outbound requests, SQL activity, and even a bit of primitive .NET investigation.
- ANTS Profiler (Red Gate) - This is a detailed .NET profiler that can give you line-level timings and other information on your application. it does need to be installed on your server and set up before you start, but its information can be invaluable.
- Performance Monitor - This is just your out of the box resource monitor in Windows Server. Many people forget this simple utility that grants us exposure to CPU, Memory, Disk, and other resource utilization information. It's free, everyone has it, so why not use it!
- IIS Logs - These are a great item to collect after a test as you might be able to do some analysis on the response & processing time for your application.
- Windows Event Logs & Application Logs - Be sure to track and keep these as well, you never know when you might want to try and find an error that was encountered during a test.
This is just a few of the items I've used. If you are working on a specific issue, feel free to post in the comments section below and I an try to point you in the right direction.
Executing Your Tests
This is often viewed by individuals as the "Fun" part of the process. You get to see your application get abused and see where things break. However, it is important to be diligent during this process and do all of the testings that you should. Not only should you be using your load testing tool(s) of choice to initiate the test, but it is very important to manually check in on the site. When subjecting a site to heavy load from third-party solutions it is possible that a performance bottleneck could be occurring between the test facility and the site, and NOT impacting others. So if your load test shows slow responses, make sure you can validate!
The testing tools that you use for execution can vary a lot. However, I've used the following tools at various times and it will start you down the road to selecting the tool for your environment.
- Loader.io - This is a great simple tool, fast to set up and easy to get results. However, this tool is a "single request" type of tool by default. Not a big deal, but something you want to be aware of.
- Blitz.io - This is another good tool, the pricing is right as well.
- LoadStorm - This is a great comprehensive load testing tool. You can do user flows, full HTTP request tracking and all kinds of fun things. I've used this for over 200,000 virtual users of testing in the last 2 months and have been quite impressed
- LoadImpact - This tool seems to be fantastic, but I do not have any personal experience with it.
Regardless of the tool that you use, get your test done!
Analysis and Re-testing
This is an imperative part of the process. Review all the information that you have collected. Identify a hypothesis, make a SINGLE change and then retest. I cannot stress the importance of following the scientific method when working through performance issues. If you change more than one thing, how do you know what actually fixed the issue? Or what if something doesn't work as expected, what broke?
Summary & Shameless Plug
I hope that this information has been helpful. In the past two years, I've done a lot of talking at conferences, on-site presentations to development teams, and my own load testing for clients. In many ways it is more of an art than a science as each situation is different, each site has its own quirks and needs to be handled differently. If you are totally lost with your load testing or application needs, reach out to someone with experience. You will be amazed at what can be accomplished with a 2-4 hour training session for those inexperienced with load testing.
Happy Coding!