FS JUnit – Retry handling

We in the FS (Financial Services) architecture group are working on a significant upgrade to our integration, regression and unit testing framework. During this process we experienced periodic test failures that seemed to be “timing” issues. After some investigation and discussion with the team, it was determined that certain tests periodically failed. We simply had to re-run them to get a green bar. We did not have time budgeted to pause and fix another project’s tests.

Taking a page out of our colleagues’ play book back at Intelliware H.O. (Test Server with Retries), we implemented a retry mechanism for our tests. We added an annotation to our FSJunit project which is detected by our multi-threaded FSQueuedTestClassRunner. This annotation specifies the maximum number of attempts to run a test method. This annotation can be at the Test Method or Test Class level. Setting the maxAttempts=2 for all of our 138 integration/regression tests was accomplished by dropping the annotation on the base Web Test class. This gave us a consistent green bar. In post-run analysis it was determined that <5 tests (out of 138) needed re-run.

@RunWith(FSQueuedTestClassRunner.class)
@TestMaxAttempts(maxAttempts=2) //All Test Methods will attempt to pass at most 2 times
public class StringUtilitiesTest {

@Test
@TestMaxAttempts(maxAttempts=5) //This Test will will attempt to pass at most 5 times
public void testProcessString() {
   ...
}

Fixing the test is ultimately the right thing to do. Sometimes you just don’t have the time to fix it right then and there.

Kevin

It's only fair to share...
Share on FacebookGoogle+Tweet about this on TwitterShare on LinkedIn

Leave a Reply