The current project we are working on has 16 separate Eclipse projects. Running unit tests in Eclipse using the launch target only works by scanning one project at a time. To ensure that all tests are run we built a test scanner that scans all of our projects. This works great but makes for a long test run. To improve throughput we take advantage of our multi-core boxes and multi-thread our unit and integration tests by using the JUnit ActiveTestSuite. This will run all its tests (TestSuites) in separate threads.
Essentially we scan for test cases and apply TestCase filters to group them into TestSuites. Examples of our filters are below.
- Separate TestSuite per project with a further breakdown as follows:
- Separate TestSuite for long running TestCases flagged using an annotation
- Separate TestSuite for Tests that need HomeState in a Database. Incur the cost of setup once.
- Separate TestSuite for Tests that do not need homestate but are arbitrariy split into groups based on simple hashcode mod calculation
This turns our 16 project unit test run into a single launch target that takes advantage of threading. All our 2400+ tests run in approximately 160 seconds. No excuses for forgetting to run tests before checkin.
Here is a picture of the CPUs while the tests are running.
We have some added tracking built in. We track the cost of Setup and TearDown per TestSuite run. We also log all unit tests that are not marked “Long Running” that take > 2 seconds. This way we can decide how to re-group the tests accordingly.
For Integration tests we take the same approach. We take it a little further and remember statistics for each TestCase. That way we can balance the integration tests across threads minimizing the total time required to run them. Currently our build machine runs 6 separate Threads of Selenium with separate, dynamic homestates per thread. All 48 integration tests take approximately 9 minutes to run compared to 40 minutes single threaded.