I just spent the last 3 days doing the Rapid Software Testing Course run by James Bach. It was an excellent and fun course and I learned heaps from it – highly recommended for anyone interested in testing. One of the things James taught was a “blink test” – basically a test where you run a heap of data in front of you very fast to look for patterns or problems.
So I was thinking today about uses for GUI automation and how GUI automation often blindly flies by obvious bugs, and how I’ve often found bugs while watching automated smoke tests run. If you run a huge suite of test cases overnight it’s not worth watching the whole suite run because that would take hours. But hey, what if we just grab the automated tests that run in the area that we’re interested in and watch them while they run? Not only would it offer us an inexpensive way to spot interesting things, but it serves the dual purpose of being a soft audit for the automated tests, to make sure that they’re still doing their jobs properly.
Does anybody already do this? An automation blink test?
« Test certification advantage Why aren’t there more women in IT? »
I’ve seen a cool example of this in an enormous test lab. The testers there set up automated long-sequence load tests on literally hundreds of machines with hundreds of monitors. Every now and again they would walk down the aisles between the machines, glancing at the monitors for things out of the norm.
One could also record and fast-replay a screen on a particular machine, or scan the log files with one finger on the down-arrow key.
—Michael B.
Hi Trish,
I find this an excellent way to blink test web forms and designs. I often do as you described to look for alignment issues between pages. Moving slowly through often doesn’t show the problem as obviously as a fast run through the site.
Browser behaviour differences are also a good use of fast running UI automation.
Glad you enjoyed the course. I really need to get myself on this course sometime very very soon.
Rob..
Hi Thrish. I oftenly do the same thing you described. Automation suites does tend to fly by bugs obvios to human eyes so why not keep an eye on it ever once in a while.
Most GUI test automation tools have capture screenshot methods e.g. Selenium RC has: http://release.seleniumhq.org/selenium-remote-control/1.0-beta-2/doc/server/org/openqa/selenium/server/commands/CaptureScreenshotCommand.html
Writing a test framework that takes a screenshot when tests fail to assist investigating automated test failures is a common use case, but so is taking screenshots throughout automated test execution (not just on failures) and scripting a tool to convert those images into a movie which can then be played back as fast or slowly as needed. Perfect for blink testing.
Taking screenshots though can slow down test execution (and take up disk space) so it’s a good idea to implement a flag in your framework to enable/disable it.
Hi Trish,
Something we did at a former workplace of mine: in addition to a suite of Selenium automated tests, we had a routine visual walkthrough to look for things that they wouldn’t catch: broken CSS, overlapping elements, other visual glitches, etc. Before very long, we got the idea to “automate” the visual walkthrough by writing a Selenium script — putting the walkthrough on autopilot, as it were. You could pause the script’s execution at any point to investigate something that had gone awry. Not having to do all the pointing and clicking by hand meant we could get through a visual walkthrough in something like 25 minutes instead of the hour it used to take.
Thanks guys, it’s great to know that other testers have been using this successfully. Recording using a screen capture tool is a great idea as well.
Andrew, I like the idea of screenshots during automation and have tried them out in our current WebAii framework, but getting Gallio to play nice in order to screenshot on failure has been a bit of a fiasco, so that one’s sitting on the backburner for now. That’s a great idea, making them into a movie to review later, I’ll keep that in mind if we get around to getting the screenshots working.
Great ideas, you’ve all really given me something further to think about.
Hi Trish,
I posted my comment a while ago with links pointing to “Tour Bus” heuristic by Cem Kaner and James Bach… probably, links were the reason my input went as spam.
Anyway, you can find it by googling. Search for “bbstScripting2005.ppt”
Thanks,
Albert
Thanks Albert, I’ll check that out. Sorry about your comment not showing up, I can only assume like you say that my spam filter must have swallowed it.
Hi Trish,
I do as well, frequently. I briefly describe something similar here: http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=103
references:
Computer Assisted Testing: http://www.kaner.com/testarch.html
Tool-Supported Testing: http://www.satisfice.com/presentations/agileauto.pdf
-Jonathan