Latest Entries »

A couple of weeks ago, the Sydney testers meet up group got together to talk about how to introduce testing to an organisation. Drinks were had (thanks SoftEd!), stories were shared, debates were had and friendships were formed. Here are a few of the points that I took away from the discussion.

Introducing testing is not just about introducing testers
Introducing testing to a new team means changing the way the team develops software. It’s not enough to simply add testers and expect testing to happen. Integrating testing into the software development lifecycle is something that involves the way every team member works.

Authority is important
If you are to be a catalyst for change, you need enough authority to be able to instigate the change. Often this kind of authority can come with being an outside consultant. It was noted that sometimes the fact that you are being paid a huge sum of money in order to introduce new practices can lead to you having a stronger voice in the organisation. Being a contractor can have similar advantages. It was discussed that, as a contractor, you’re less affected by the politics of the organisation and this enables you to introduce change more easily.

Developers can test too
There was some debate about the difference between testers and developers, and how to tell if someone is “destined” to be a tester versus a developer. Testers can program, and developers can test, and it’s beneficial for both testers and developers to have these skills. The difference is the individual’s personal preference, and level of skill in these areas.

Self-marketing
The value in testing isn’t always readily apparent, especially to high-level managers. Demonstrating the value in testing activities is an important part of introducing testing to an organisation. Metrics such as bug counts can be used to help show value, but they can also be misused. For example, if testers help developers to improve their own testing skills then the amount of reported bugs may decrease, and this would look bad for a team that uses bug reports as a metric to show test team effectiveness. For this reason it’s important to demonstrate that testing is more about preventing bugs than finding bugs.

To automate or not to automate
Introducing automated GUI tests is a project risk, and is an activity that is often underestimated in terms of time cost and skill requirement. There is always a maintenance cost that needs to be considered when introducing automated tests. Automation metrics can also be easily misused. On the other hand, if considered well, automated testing can be a valuable asset to many software development teams.

The tester mindset
Testers do need a negative mindset to perform some kinds of testing activities. But good testers need to be able to switch out of this mindset at will or they’ll just become unpleasant to work with, and this is not good self-marketing for the testing team.

A few follow-up topics were suggested, including “how to recruit testers” and “how blurred should the line be between testers and developers?”

You can find information on upcoming Sydney Tester Meetups on the meetup homepage.

Recently, a link came my way about Aspiritech, a testing company that only hires testers with Asperger’s Syndrome or high-functioning Autism. I’d heard of companies like this before, and was curious about the advantages and limitations of this testing approach, and about what made software testing so suited to this demographic. So I expressed some interest on Twitter, and was subsequently introduced to Michael Drejer. Michael has Asperger’s Syndrome, and is a software tester working at BOAS Specialister in Denmark. So he was able to answer a few of my questions over Twitter.

Michael:
Oh yeah, @Aspiritech they’re good guys. I work for the company that started the whole asperger-software tester thing. Or well, we are a separate company now from the original Specialisterne. We are called BOAS Specialister. Specialisterne is Danish for The Specialists (and BOAS is a contraction of words Live on your own Asperger).

Trish:
Cool. How does the test approach of these orgs differ from other orgs?

Michael:
They don’t really. Except we’re better and more thorough. We follow test cases/plans when we have them, or write them then execute. The main difference is that you have to have an asperger diagnosis to be hired. And the work environment is more asperger friendly.

Trish:
Is all of the testing done with test cases, or are there also exploratory sessions done without test cases?

Michael:
We do bug hunts as well. Some things are easy enough to do without test cases. But it’s good to have a structured process to work from, that’s one of the important things.

Trish:
Interesting. Do you work directly with developers, or does your team work externally from the main dev team?

Michael:
We work only as external testers. We usually report the bugs in a bug report software, depending on what the client uses. Over the years, we have done a lot of testing for Lego on their website, shop and flash games for example.

Trish:
Cool. What makes the structure of the testing so important?

Michael:
It’s the asperger thing. We really don’t like when things aren’t structured. It’s what makes it difficult for us to find work, because a lot of jobs and places of work don’t have that kind of structure we need. That’s why QA testing is so good for us. And because we have an extraordinary ability to notice details, we find more bugs than most other testers do.

Ah I see. Do you think your preference for structure helps you spot software defects more easily?

Michael:
Structure prevents me from losing my mind in frustration, allowing me to focus on the testing. The work environment is also an important factor. I don’t like a lot of light, and I don’t like having my back towards the door. My co-workers have other issues, mostly sensory-related.

Trish:
Interesting. I know a few test approaches that aren’t as structured though. I guess that wouldn’t work as well for you guys? Like for example, working without specification documents, and/or not using test cases.

Michael:
When testing websites we rarely use test cases (I basically make them in my head and remember for testing again). BTW, a couple of us are ISEB certified. I think it depends on the scope of the software. Big software, like the one we’re working on now, needs test cases.

Trish:
Ah sorry, when you said structured, I thought you meant test cases. What did you mean by structured?

Michael:
I do mean test cases. But when it’s smaller jobs, it’s okay, we don’t need test cases for that. If it’s a complex software program, then we need to know what to do. It is just how we work most optimal, and it can stress us out very easily if we don’t have test cases, or at least some sort of recipe for how to do the testing. We just don’t always have the privilege of having test cases.

If you’d like to learn more, Michael wrote this guest blog post about working as a software tester with Asperger’s Syndrome. It was recently published in the book “The Thinking Person’s Guide to Autism”.

There’s a plague upon the testing craft, and it’s called “fake testing”. You could also call it “ineffective testing”. Here’s James Bach’s guide to faking a test project.

I believe fake testing is generally practiced by testers who just don’t know any better. We can’t really blame them. Most testers “fall in” to the profession. There’s a widely held belief that testing is an unskilled profession that can be done by anyone. So when it comes to adding testers to a project, often “anyone” is hired.

We could try to educate those “anyone” testers, but there’s not much point. They are the symptom, and not the problem. The problem is a lack of understanding of testing by the people who are hiring these testers in the first place. The problem is that testing is easy to fake, and its effectiveness is difficult to measure.

This is why we end up with testing certification. Hiring managers don’t know enough about testing to recognise good testers, so they’re just turning to some kind of authority to tell them what a good tester looks like.

If we want to attract more skilled professionals to software testing, then we need to work on educating the developers, the project managers, the product managers, the recruiters and the other people involved in hiring testers in the first place. If they recognise good testing, then they’ll know how to recognise good testers. Understanding how to effectively test software is something that the whole software development team should know about to some extent anyway.

So how do we do it? Well, for starters, maybe we should stop siloing ourselves as a community. We should be networking with the rest of the IT community, doing talks at developer and project management conferences, submitting articles to a broader audience. I’m sure some of us already are. It would be great to introduce more testing-focused coursework in universities. I know it was an area that was definitely lacking in my computer science coursework.

It’s been so good to see non-testers coming along to the Sydney Tester Meetups. I think we need more online resources, so that it’s easy to direct people to good sources of information about software testing. Something like “good testing starts here”. Dot com.

What do you think? How can we help educate the greater IT community about software testing?

I’ve talked to a lot of unhappy testers – smart, talented testers who always feel like they are working at odds with their teams in order to advocate quality. In this kind of situation, it’s easy to believe that quality is dead. In my last post, I speculated that perhaps it just isn’t cost-effective anymore for companies to have a primary quality focus. In which case, as an advocate for quality, how could a tester ever have real job satisfaction?

Software isn’t just a “see who can make the most money” game. There are people out there who believe in making good products.

The quality of the product is limited by the quality goal of the company. Testers, developers, designers – we all help the company achieve that goal. If the goal doesn’t align with our own personal ideas of quality, then we will not have much job satisfaction.

Whether you like software to look beautiful, or produce huge profits, or be driven by beautiful code, or make your customers love you, you want to build something that makes you proud.

Jerry Weinberg famously said that quality is value to someone at some time. What is quality to you, at this time? Are you working at a place that has the same quality definition as you? If not, then you have two choices – adjust your personal definition of quality to match the company’s, or find a company that has the same values as you do.

Every successful company is making quality software, by some person’s definition of quality. If we want to have real satisfaction in what we help create, we need to find or create a company with quality values that align with our own, and then work out how we can best serve to help achieve that goal.

“Quality is dead,” declared Goranka Bjedov. And I thought to myself, I knew it.

For some time now I’ve been concerned about this. I’d always seen specialised testing as a way to turn a mediocre-quality product into a high-quality product. But the evidence before my own eyes is that the market is full of buggy software. The worst part is, people accept buggy software as the norm and learn to live with it.

We say that nobody can ever be “done” testing, because there is always more than can be tested. There comes a point in any project where the cost of continuing to test outweighs the benefit in releasing earlier, but with unknown bugs.

“The truth is, bad software makes more money” – Goranka Bjedov, STANZ 2011.

I recently watched a project going through a fairly difficult stabilization phase. Too much change towards the end of the development cycle and not enough testing at the start meant that most of the bugs were being found at the end, so the team was looking at a few more weeks of catch up time to find and fix all of the bugs. But, for reasons that I still don’t completely understand, all of the bugs were postponed and it was released anyway. The next two weeks were a mad scramble to fix production bugs, followed by several weeks of less frantic fixing. I was ready to label the project a colossal failure. But the product was actually very well-received by most customers. By the end of the first week, customers were singing praises for the new feature and the project was hailed as a success.

It left me wondering what the point was of having testers on that project at all. I was told that, without testers, it would have been a whole lot worse. Is that rewarding though? Working to make something “not as bad as it could be”?

After STANZ, I stayed with my friend Richard for a few days. Richard is a medical doctor, training in radiation oncology. I asked him if it was depressing being in a job treating cancer, which is a condition that’s difficult to treat and not completely curable.

“Oh not at all” he said. “There are some practical things you can do, to improve quality of life”.
“But,” I pressed, “when people think cancer, they usually think that’s it for them. It seems like you’re just fighting an uphill battle.”
“Everyone dies eventually, of something,” he explained. “But I know that there’s things I can do that means they’re not going to die today. And the quality of life they have in the meantime can be made a lot better.”
I nodded. “I guess that’s true.”
“Once you accept mortality, medicine’s not that hard. It’s actually quite rewarding.”

Perhaps we need a mentality shift, and a different way of seeing our role. If we see ourselves as the cure for bad software, then we’re going to be repeatedly disappointed.

I emailed my friend James Martin for his perspective, and he offered this one:

How about working to make something ‘better’. What if all any of us did, regardless of job title, was try as hard as we could to make the best stuff possible, accepting the fact that even the best of us has only a fragile grasp of what it takes to make something that another person will be delighted by.

Personally, it helps me to think that my whole job description is ‘to make things better’. That way I can solve problems using whatever tools feel right for the task at hand and I don’t feel constrained by someone else’s opinion of who I should be and what I should do.

As usual, the people setting the ‘standards’ for our ‘profession’ do so without knowing our context. But we can set our own standards for ourselves and play by our own rules. I like to think that you and I did that last year. I still think of what we did at Campaign Monitor as my best work.

If quality is dead, what are we doing here?
If we are adding value, what is it and how do we measure it?

When WebAii encounters a modal dialog of any kind, its default response is to sit there staring at it for eternity, causing your entire test suite to hang. It’s not exactly ideal. So, here is some hackery I put together to prevent this from happening.

Stuff that might be important to know:
- I’m using WebAii version 2.0 beta, which might not be the latest version but is pretty close to it. I did make some effort to try to work out what the latest version is, but it was far too hard to work it out from the website. So if the latest version handles this better, we’re all in luck I suppose.
- We’re converting all our tests to WatiN. Try it today!

1. Start an unexpected dialog monitor before each click action, based on the assumption that dialogs generally appear as the result of a click.
2. Set up a second click action for when you’re *expecting* a dialog, so that your unexpected dialog monitor doesn’t handle the expected stuff.

public void ClickElement(Element element, bool isDialogExpected)
{
    Assert.IsNotNull(element, "Error: element to be clicked was null");
 
    if (!isDialogExpected)
    {
        // Handler for unexpected dialogs that may appear
        var anyDialog = new GenericDialog(ActiveBrowser, "", true) {HandlerDelegate = HandleUnexpectedDialog};
        Manager.DialogMonitor.AddDialog(anyDialog);
        Manager.DialogMonitor.Start();
 
        Actions.Click(element);
 
        Manager.DialogMonitor.RemoveDialog(anyDialog);
        Manager.DialogMonitor.Stop();
 
        if (_unexpectedDialogAppeared)
        {
            Assert.Fail("An unexpected dialog appeared after clicking: " + element + ". The dialog caption was: " + _unexpectedDialogCaption);
        }
    }
    else
    {
        Actions.Click(element);
    }
}

3. Set up a handler for unexpected dialogs, which attempts to read the dialog text, close the dialog and gracefully fail the test with a message that includes the text of the dialog. It’s great to take a screenshot at this point too, if you can.

Important note: Make sure you’re identifying the dialog as a dialog attached to the browser process. Otherwise tests will fail when they encounter *any system dialog at all*. I can’t tell you how disheartening it is to find that all your tests have failed because your VM’s antivirus software needs updating.

public void HandleUnexpectedDialog(IDialog dialog)
{
    if (dialog.Window.OwnerProcess.ProcessName == "iexplore")
    {
        _unexpectedDialogAppeared = true;
        TestHelper.CaptureScreenForTestLog(ToString());
        try
        {
            _unexpectedDialogCaption = dialog.Window.Caption;
        }
        finally
        {
            var anyDialog = new GenericDialog(ActiveBrowser, "", true) { HandlerDelegate = KillBrowserBecauseOfTooManyDialogs };
            Manager.DialogMonitor.AddDialog(anyDialog);
            Manager.DialogMonitor.Start();
 
            dialog.Window.Close();
 
            Manager.DialogMonitor.RemoveDialog(anyDialog);
            Manager.DialogMonitor.Stop();
        }
    }
}

4. If the dialog fails to close properly, kill all browser processes. Note that it’s passing the failure reason through, so that the test doesn’t just fail because of lack of running browsers.

private static void KillBrowserBecauseOfTooManyDialogs(IDialog dialog)
{
    KillBrowserProcesses("More than one dialog appeared in sequence, so I thought it best to just kill all IE processes and start afresh.");
}
 
public static void KillBrowserProcesses(String reason)
{
    CaptureScreenForTestLog(reason);
    foreach (Process p in Process.GetProcessesByName("iexplore"))
    {
        p.Kill();
    }
 
    Assert.Fail(reason);
}

5. In dialog handling for expected dialogs, also monitor for unexpected dialogs. For example, during a file upload, if the file is not found, an unexpected dialog might appear. Kill it on sight.

/// <summary>
/// Opens the File Upload dialog and attempts to upload a specified file. 
/// (note, in firefox, the windows are called "File Upload")
/// </summary>
/// <param name="reference">The reference to the HtmlInputFile control used for opening the File Upload dialog</param>
/// <param name="fileName">The combined filepath + filename of the file to upload</param>
public void UploadFile(String reference, String fileName)
{
    // A flag to indicate whether a file not found error has occurred
    _unexpectedDialogAppeared = false;
 
    _count = 0;  // number of times a dialog was opened
 
    _fileName = fileName;   // save filename into global variable
    String alertTitle = (ActiveBrowser.BrowserType == BrowserType.InternetExplorer) ? "Choose File to Upload" : "File Upload";
 
    // Handlers for unexpected dialogs that may appear if the file cannot be found
    var logonPrompt = new GenericDialog(ActiveBrowser, "Connect to ", "Cancel", true) { HandlerDelegate = HandleDialog };
    var alert = new GenericDialog(ActiveBrowser, alertTitle, "", false) { HandlerDelegate = HandleDialog };
 
    Manager.DialogMonitor.AddDialog(logonPrompt);
    Manager.DialogMonitor.AddDialog(alert);
    Manager.DialogMonitor.Start();
 
    // Click the field so that the file upload dialog appears
    if (ActiveBrowser.BrowserType != BrowserType.FireFox)
    {
        ClickWhileExpectingDialog(reference);
        alert.WaitUntilHandled(10000);
    }
    else
    {
        ClickWhileExpectingDialog(reference);
        //inputField.MouseClick();
        alert.WaitUntilHandled();
    }
 
    Manager.DialogMonitor.RemoveDialog(logonPrompt);
    Manager.DialogMonitor.RemoveDialog(alert);
    Manager.DialogMonitor.Stop();
 
    // Fail the test if the file was not found
    if (_unexpectedDialogAppeared)
    {
        Assert.Fail("An unexpected dialog appeared while attempting to select file: {0} for upload, the caption was: {1}", fileName, _unexpectedDialogCaption);
    }
}

6. Test it. How else will you know if it works? This test references a simple html file with a button that launches a file upload dialog.

using MbUnit.Framework;a
using System.Windows.Forms;
 
namespace Tests.Common.Test
{
    public class HandleDialogTests
    {
        WebBrowser _webBrowser;
        private GetTestData _testData;
 
        /// <summary>
        /// Initialization for the class. Creates test data required for test cases in this class.
        /// </summary>
        [SetUp]
        public void MyTestInitialize()
        {
            _testData = new GetTestData();
            _webBrowser = new WebBrowser(_testData);
        }
 
        /// <summary>
        /// Clean up for the class.
        /// </summary>
        [TearDown]
        public void MyTestCleanUp()
        {
            TestHelper.CaptureScreenOnFailure(ToString());
            _webBrowser.TestCleanUp();
        }
 
        [Test]
        public void UnexpectedBrowserDialog()
        {
            // Unexpected browser dialogs should cause tests to fail
            _webBrowser.LaunchBrowserAndNavigateTo(StringFormattingHelper.GetCurrentWorkingDirectoryAsUri() + "/resources/fileUpload.html");
            _webBrowser.Click("id=fileupload");
        }
 
        [Test]
        public void UnexpectedWin32Dialog()
        {
            // Unexpected win32 dialogs should not cause tests to fail
            new System.Threading.Thread(() => MessageBox.Show("background thread modal box")).Start();
            _webBrowser.LaunchBrowserAndNavigateTo(StringFormattingHelper.GetCurrentWorkingDirectoryAsUri() + "/resources/fileUpload.html");
            _webBrowser.Click("id=link");
        }
 
        [Test]
        public void FileDialogTimeout()
        {
            _webBrowser.LaunchBrowserAndNavigateTo(StringFormattingHelper.GetCurrentWorkingDirectoryAsUri() + "/resources/fileUpload.html");
            _webBrowser.UploadFile("id=fileupload", StringFormattingHelper.GetCurrentWorkingDirectoryAsUri() + "/resources/file.txt");
        }
 
        [Test]
        public void UnexpectedDialogWhileUploadingFile()
        {
            _webBrowser.LaunchBrowserAndNavigateTo(StringFormattingHelper.GetCurrentWorkingDirectoryAsUri() + "/resources/fileUpload.html");
            _webBrowser.UploadFile("id=fileupload", "badFileName");
        }
    }
}

A few people have asked me how the Sydney tester meetups began.

Back in early 2010, I started working with a guy called James. James and I could talk about testing pretty much indefinitely. Then one day, James and I heard about another keen tester named Marlena who had just moved to Sydney to work for Atlassian. She invited us out to meet her after work one evening, so James and I set out to the Red Oak brewery cafe for a few beers and a chin wag.

Well, we nerded it up like nobody’s business. Talking testing with other testers is fun, we said. We should do it more often, we said. So we kept meeting up and having some beers and talking some testing and we liked it a lot. Then we said you know what would make this even better? More testers. So we put the word out that’s how the Sydney tester meetups were born.

At first we tried to spread the word via Twitter, email and word of mouth, and we had some monthly meetups in bars. There were no topics or talks – we’d just sit around and have a few drinks and chat for a while. But we found it hard to get more than a few people to attend. Finding interested testers was difficult, and the bars could be pretty noisy.

It was pretty close to a year since that first tester meetup, when the three of us met at Darling Harbour for the last time. James was moving back to England and Marlena was moving back to the USA. Our meetup group numbers had dwindled down to a total of four that night. That could have been the end of the Sydney tester meetups. But our fourth member was Anne-Marie, who had just moved to Sydney from Ireland. And Anne-Marie wasn’t ready to let the meetups die just yet.

Anne-Marie approached SoftEd for sponsorship, mainly so that we could get a bigger and quieter regular venue for our meetups, and perhaps entice people with some free refreshments. SoftEd happily agreed, and they also sponsored our new Meetup.com subscription (which is only about $4/month). SoftEd was also running James Bach‘s Rapid Software Testing course in Melbourne, so they talked to him about doing a talk at our tester meetup, to which he agreed.

Meanwhile, I and asked Andrew from Atlassian if they would be willing to host the next meetup. Atlassian was happy to host, so that was our venue problem solved. Google was also supportive, and offered to host a future meetup.

So I think through the power of Meetup.com and the drawcard of James Bach, we managed to get around 50 people for this meetup. I had to close it off to new RSVPs a week before the event, because the venue couldn’t host any more! Bruce picked up some snacks for hungry testers. The talk was very well-received, and it was followed by an enjoyable dinner with a few of the testers there.
Since then, I’ve organised a couple of meetups at City Hotel, where we have a quiet room to ourselves and a bar tab courtesy of SoftEd. The first was a games night, where we divided into groups and played some games designed to challenge some of the same logic, lateral thinking and reasoning skills that we use in testing. The second meetup had no specific format – just a chance to get to know other testers in the group. The attendance for each was somewhere between 10 and 20 people – testers and other IT folk from all kinds of workplaces, just talking testing.

The thing I love most about these meetups is that everyone there offers a different perspective, and an enthusiasm for what we do. One of the greatest things to hear is someone saying “Huh. I’d never thought about it that way.”

If you’re in Sydney and would like to come to one of our meetups, just sign up at our meetup.com site to stay updated about our upcoming events.

(A conversation snippet from the hogfish.net archives, circa 2007)

Trish says:
aw man

Trish says:
/me tries to remember what the file made yesterday was called

Trish says:
/me sifts through files called things like jam.xml, ponymeat.xml, pork.xml, raisins.xml, great.xml, jelly.xml

Trish says:
oh scone.xml

Trish says:
it’s all so obvious now

Dewi says:
/me laughs

Dewi says:
I hope you’ve got test cases to ensure they are always well-formed and conformant to their schemas!

I’ve heard that people can identify with either visual, auditory or tactile methods of learning, or some combination of the three. This is called Fleming’s VARK model of learning. The concept is pretty simple. Visual learners learn best with visual aids, like diagrams, pictures and books. Auditory learnings learn best when they’re listening to things like lectures and discussions. Tactile learners learn best when they’re doing things, like experiments and demonstrations. Most people will probably use a combination, but show a preference for one method more than the others.

I know from experience that I’m a very visual learner, so I love drawing diagrams and writing very structured documents to visualise information. Tactile learning works well for me too, but I usually struggle with audio information. So mind maps are a really useful tool for me. I can create mind maps very rapidly because each node already has the context of the previous node.

When I’m using mind maps for requirements analysis, I’ll make initial nodes for each main component, then break each component down into smaller feature nodes. I’ll add nodes for any questions or test cases that I think of along the way. I used to make the questions and test cases a different colour, but I found that it was a waste of time because I’m really just capturing raw data, which forms the basis of my questions and test cases. Once I’ve written test cases, I don’t really look at the map again.

I think this is because mapping isn’t so much an information capturing exercise as it is a learning exercise. Visualising requirements helps me to form a more complete mental model of the system under test, which helps me to work out scenarios based on this understanding.

I tried to use mind maps for mapping out functional areas of a system, but it didn’t work because mind maps only allow a tree hierarchy so they can’t show an adequate level of interconnectedness between functional areas.

I think that at their best, mind maps will be a visual representation of the author’s own thought process. For this reason, they may not be useful to anyone other than the author (especially non-visual learners). This has been my experience, but perhaps others are finding it useful in collaboration if they are using them in a different way.

If you’ve ever maintained a large suite of GUI automated tests, then you’re probably familiar with this scenario: You run your suite overnight and everything passes except for a couple of tests which fail unexpectedly. When you re-run them, they pass. These “random failures” cause your suite to look less than reliable, and you find yourself rerunning the suite every morning, or perhaps just rerunning the failed tests. You may find that your team starts to lose confidence in the quality of the tests. After all, they seem to be failing randomly for no good reason.

Well, there’s always a reason. In the case of our tests, they were encountering occasional issues like server timeouts, network issues, browser flakiness and a myriad of other environmental problems that quite frankly weren’t worth fixing at the time. I don’t like pandering to the diva that is GUI automation – every time I have to spend time fixing something just to make automated tests work, this increases the maintenance cost of automated testing and decreases its value. Automated testing is supposed to save us time, not create more work for us.

So here’s a cheap solution to this problem – just run the tests again. Each test gets one more chance to pass, and if they fail twice then they fail for good. Since GUI automation is generally plagued with more false negatives than false positives, I’m not too concerned about the risk of tests passing incorrectly. But even so, it’s good to have a record of the tests that failed the first time, just to glance over. Just in case.

My colleague Pete came up with this solution for our test suite, which will work for tests driven by Gallio (MbUnit). It’s a modification of an attribute that lets you run tests multiple times. He’s changed it so that they will only run a second time if they fail. Unfortunately in the standard test output, it hides the failure message, but you can still see it in the Gallio test report, which we use to view screenshots of test failures anyway.

It’s certainly helped the reliability of our test suites, and increased our team’s confidence in the value of the automated tests.

using System;
using Gallio.Framework;
using Gallio.Framework.Pattern;
using Gallio.Model;
using Gallio.Common.Reflection;
using MbUnit.Framework;
namespace SecondChanceExample
{
  public class RepeatOnFailureAttribute :TestDecoratorPatternAttribute
  {
  private readonly int _maxNumberOfAttempts;
  public RepeatOnFailureAttribute(int maxNumberOfAttempts)
  {
    if (maxNumberOfAttempts &lt; 1)
    throw new ArgumentOutOfRangeException("maxNumberOfAttempts", @"The maximum number of attempts must be at least 1.");
    _maxNumberOfAttempts = maxNumberOfAttempts;
  }
  protected override void DecorateTest(IPatternScope scope, ICodeElementInfo codeElement)
  {
    scope.TestBuilder.TestInstanceActions.RunTestInstanceBodyChain.Around(delegate(PatternTestInstanceState state,     Gallio.Common.Func inner)
    {
      TestOutcome outcome = TestOutcome.Passed;
      int failureCount = 0;
      // we will try up to 'max' times to get a pass, if we do, then break out and don't run the test anymore
      for (int i = 0; i &lt; _maxNumberOfAttempts; i++)
      {
        string name = String.Format("Repetition #{0}", i + 1);
        TestContext context = TestStep.RunStep(name, delegate
      {
        TestOutcome innerOutcome = inner(state);
        // if we get a fail, throw an error
        if (innerOutcome.Status != TestStatus.Passed)
        {
         throw new SilentTestException(innerOutcome);
        }
      }, null, false, codeElement);
      outcome = context.Outcome;
      // escape the loop if the test has passed, otherwise increment the failure count
      if (context.Outcome.Status == TestStatus.Passed)
      break;
      failureCount++;
    }
    TestLog.WriteLine(String.Format(
    failureCount == _maxNumberOfAttempts
    ? "Tried {0} times to get a pass test result but didn't get it"
    : "The test passed on attempt {1} out of {0}", _maxNumberOfAttempts, failureCount + 1));
    return outcome;
  });
  }
}
}