I've recently started a new job (Java web development) - watch this space, I'll be blogging about it very soon - and have needed to set up some external monitoring and performance testing quickly. I've been using Selenium for functional testing for a few years now - and was excited when I came across browsermob - which allows you to use selenium scripts to perform external system monitoring and performance testing at load. It does so by spawning real firefox instances across virtual machines running inside Amazons EC2 cloud infrastructure. This allows you to use test servers located in seven countries:
- Washington DC
- SF Bay Area, CA
- Dublin, Ireland
- New York
- Amsterdam
- Singapore
- Dallas TX
With the monitoring component, you can choose several countries to ping your site.
With the performance testing component, you choose a single country to run your tests from.
There are two variety of users - Real Browser Users (RBU) and Virtual Users (VU). RBU allows you to run selenium scripts out of a real browser, VU allows you play back raw http request/response. VU has a much cheaper infrastructure cost.
Both monitoring scripts and performance scripts use the browsermob javascript api - this also provides access to the selenium api (for RBUs). The documentation, FAQs and blog articles are all very well-written and extremely helpful.
There is a local script verfication harness you can download to develop your scripts and test (it's pretty cool - you actually see firefox being remote controlled locally via Selenium RC).
There is a mechanism to upload csv files (containing user login creds and test data), and also convert your existing selenium functional test scripts.
Performance tests can be scheduled - you can pick the number of users you want, you can choose to ramp up (or down) users, or run at a constant rate.
You can choose to run several test scripts during a test session (you give each script a percentage to determine how many times it gets executed). There is no facility (at present) to create predefined test scenarios for later scheduling.
Test results are pretty stunning, they provide an amazing amount of data to drill down on (they also allow you to download a mysql database containing all of your test results for later analysis).
If a test fails, it records the actual browser session and produces a screenshot for later viewing.
All graphs have very good explanations on how to interpret the results. Although keep in mind, server side monitoring provides the other half of the picture (so make sure you have nmon/xymon/perfmon running on your servers).
Pricing seems pretty good for what you get - a fraction of competing commercial load testing products (and you get the infrastructure to boot).
I still need to verify a few things about the amount of included cloud dollars with the company before I comment further - the marketing material simply states "ample included":-/ I don't think there is a competing technology in this league - the features are staggering, it's easy to use and the documentation is great.
If you are building an externally facing website and have a requirement for stress and volume testing I would highly recommend this product.