übertest

Automated tests - everybody wants them, right? I’ve seen the hunger in people’s eyes when they hear about a suite of tests that run themselves. It sounds so high-tech and glamorous - machines running all your tests for you so you just have to check the results at the end of the day.

But it’s never as simple as “One Automation Please!” “Sure, here you go sir.” Test automation is an incredibly tricky thing to get right. After some lengthy research on the topic, I’m not even sure if anybody has managed to get it completely right. It’s a new concept, so it’s still evolving, but it seems like a few organisations have managed to get to a place where they are reasonably satisfied with their automation approach.

Before I discuss a few techniques, I’d just like to point out a few things about automation.

1. Automation is not a product. Do not deliver it to your customers. There are so many reasons not to do this. First of all, what are they going to do with them? Run them against their production system? Are you expecting the production system to change over time so much that it will need testing again? And if it does run and tests do fail, will the customer know how to interpret the results? Will your automated tests produce test data in their production system? Automation is a tool, a technique, a means to an end.

2. Automation is an investment. Do not use it unless you know you will profit from it. It takes a long time to automate a test. It takes even more time to maintain a suite of automated tests while the application under test is still under construction. Work out a way to decide whether automation is really saving you time, and check on it at regular points in the project lifecycle. If you are not getting any return on your automation investment, cut back.

3. Automation should not be a project deliverable. Automating tests is a technique designed to benefit the testers. If you list it as a project deliverable, you risk committing your test team to arbitrary goals, such as “Automate 100 test cases”. It should be a test team prerogative to decide when it is appropriate to use automation and how much to use, just like any other test technique or tool.

The most important thing to decide with regards to test automation is the purpose of it. What is your goal? How will you know when you are done? Here are a few goals I’ve seen in action:

Automate x% of all test cases.
Advantages: An easy target to measure.
Limitations: No prioritisation of test cases and goal may be unrealistic. When the team is under pressure to meet this goal, the testers usually automate the “easy” test cases in order to reach the target, which are usually trivial tests. In addition, x% of all test cases may not be automatable, which will result in testers writing “near enough is good enough” tests that may validate the wrong thing. The danger of this is that you will see a large amount of passing results, but they may not be accurate.

Automate all primary or high priority test cases.
Advantages: Ensures the most important areas are tested. An easy target to measure.
Limitations: Goal may be unrealistic for the available resources. If the whole test team is spending their time trying to reach the automation goal, who is looking for defects? It is difficult to scale back this goal because all of the targeted tests are high priority.

Automate smoke tests only
Advantages: An easy goal to achieve and easy to maintain. Will be used often enough to guarantee good return on investment.
Limitations: May not be enough to justify spending $x,000 on automated testing tool licenses. While suitable for very small teams, larger teams could benefit from automating more tests.

There is no definitive “right” approach, unfortunately. Here are a few approaches I am trialling at the moment.

1. Automate all smoke tests. Categorize regression test cases by product risk. Assuming a team of x testers automating for y hours per day, a maximum of z amount of test cases can be automated. Therefore, the goal is to automate the top z amount of highest risk test cases.

2. Automate all smoke tests. Identify the test cases that will take the longest time to execute manually. Rank them in order of longest to shortest time. Timebox time spent on automation to x hours per week per tester. The goal is to automate as many of the identified test cases as possible within the allotted time, in order of ranking.

Further reading: Just Enough Software Test Automation by Mosley and Posey.

The V model - A methodology applied to a task so brain-numbing that only a can of V will get you through it
Automation - A program that looks busy all the time and tells you that all the tests pass. If the tests don’t pass, it means it’s broken.
Browser compatibility testing - A long-winded exercise designed to tell you that hardly anyone uses Safari anyway.
Usability testing - And you thought triage was too long already.
Scope creep - That bastard who keeps adding new features and hitting on the female employees in the lunch room.
Agile - An excuse to do whatever the hell we want and call it a process.
Stand-up meeting - A pre-coffee meeting that everybody hates, except the project manager.

Intermittent defect - A bug that makes you look like a liar.
Can’t repro - The response you’ll get back for all your future defects after you’ve raised an intermittent defect.
Legacy System - The legacy of a project team who are no longer around. It’s usually evident as to why.

Before you say I’ve ripped off The Braidy Tester, let me get in first by saying yes, I’ve shamelessly copied his idea. Go read his list as well.

My colleague and I currently have no “proper” functional performance testing tools, and there are no immediate plans to purchase any. So instead, we’ve been working on a little side project to get Rational Functional Tester to generate performance testing results while it’s doing its usual functional testing.

We were originally using the logInfo() method, but then realised that it would be a pain to extract the performance times out of the usual RFT log file. So in the end, we came up with this:

PerformanceRat.java

Important notes:

It will output a comma delimited file that you can open in excel for easy reporting. For example, each line will look like:
Tue Jun 24 14:10:28 EST 2008, Logging In, 2.5

We haven’t managed to get the waitTime value to work for any value less than 0.3 seconds. We generally find that 0.5 second increments are sufficient for our tests.

You will need to set up one or more common methods in your SuperClass, such as getDocument(). If you haven’t set up a SuperClass and a Framework, start here:
Using the Find() Method
Setting up a SuperClass (for XDE but the principles largely apply to Functional Tester)

You will need to create a folder on your C drive called TestLogFiles or change the filepath in the java file to suit an existing folder of your own.

We have occasionally encountered the exception “No screen point found for object” if we run the same test several times in a row. No idea why this happens, but I’ll post details if we figure it out.

A few articles I’ve come across lately that are a good read:

When NOT to fix bugs: Hard-assed Bug Fixin’ - Joel Spolsky

What makes a good tester: Hallmarks of a Good Tester - Michael Hunter

What makes testers different: It’s different in Test - Harry Robinson

Recently on my current project, we had a defect raised by the customer that actually turned out to be a feature they had requested at the very beginning. My boss was amused, but all I thought was “well, it’s about time”. It seems I am cynical enough to regard “Customer logs feature as a defect” as a project milestone. It got me thinking about some other “hidden” project milestones.

Basically, if these milestones don’t occur, I start to feel a bit anxious and start to wonder what’s the matter with the project. MSF’s “zero bug bounce” is one of those milestones - the expectation that the bug count will go up again after it’s hit zero for the first time.

Here’s a few more:

• Complete or partial GUI redesign
• Customer requests urgent feature that they forgot at the start of the project
• Environmental fault blocks test progress

Anyone got any more?

Okay, this made me giggle. Yup, I’m a geek.

Take a squiz at Jeff Fry’s article “Incentives for Developers and Testers“. I found the article from the same comment on the Google Testing Blog that he wrote it in reply to:

I have a simple question as a former-programmer, now business guy, between dev and QA.

Why not put pay performance targets on both sides as incentives per testing release?

In other words, each QA person gets $10 for each bug they find (up to 10, or whatever). Each development person gets $10 for the number of bugs not found under a certain target (10, or whatever).

Seems like an easy way to get people more motivated about the whole testing process.

When I saw the comment, my first reaction was pretty similar - “Ooh, there’s a dangerous thought”. Jeff’s article explains my reaction a lot better than I ever could, and further elaborates on the link between salary and job motivation (or lack thereof).

It reminds me of something Scott Adams once said about how the requirements given to the guys who designed the pyramids were probably just the area of the base, and the height of the structure. The Egyptians were probably expecting something in the rectangular prism family, and instead got the most efficient design to meet those requirements.

No I don’t mean actually testing computer games, I’m talking about making a game out of testing. “Gee whiz golly, are you saying you can make work FUN?” I hear you ask sarcastically. Well hey mate, I’m not promising anything yet, but just take a look at the words of these fine folks:

Productivity Games - Using games to improve quality by Ross Smith
QA Deathmatch by Scott Hackett

I’m a big fan of the old bug bash, but not so sure about the QA Deathmatch idea (obviously I’d rather have dedicated testers than trying to make developers enjoy doing testing on top of their regular work).  Generally  the idea has potential, but also runs the risk of ending up like chore wars. And we all know how that turns out.

The following conversation took place last Friday night between me and my significant other:

Mister (seeing me using computer): Are you doing a web check-in?
Me: What? What do you mean?
Mister: A web check-in. Are you checking in on the web?
Me: Why on earth would I be doing that? What the…(pause)…oh you mean for the FLIGHT.
Mister: Huh? Yeah, what did you think - (realises and laughs) You thought I was asking if you were checking in FILES?

Note to self: Must un-geek my brain on Friday afternoons.

You know, I once came in partway through a project with test cases that were a little light on the details, like:

Move team.

It was a little perplexing to say the least. How do I move a team and what’s meant to happen once I do it? The other testers were just as confused as I was by this cryptic instruction. I ended up mandating that test cases have steps and verification points, believing that more information was better. I stood by that belief for quite some time, until I came in partway through a different project with test cases which were VERY thorough, a bit like this:

Start application
Is the username text field there?
Is the password text field there?
Is the Login button there?
Type your username into the username text field.
Type your password into the password text field.
Click the Login button.
Did the welcome pageload?
Did the picture of Snarfy the Cow appear at the top of the page?
Does the text at the top of the page say “Welcome to Snarfy’s barnhouse of fun!”?

and I don’t mean that was one test. I mean that was part of every single test case that was written. Some of these test cases had more than 150 verification points. Can you imagine how long that would take to test?

Honestly I believe if you want to try either of these options, there’s a much easier test case to write:

Load gun
Point at foot
Pull trigger

because either way, you’re going to end up with frustrated crazy testers and who knows what that will lead to? A desolate wasteland of despair where untamed bugs feast on the grisly remains of your software, that’s what. What a terrible thought.

So just try to aim somewhere in the middle, yeah? No need to list out every little step along the way, that’s what user guides are for after all. No need to be too general in order to increase test coverage, that’s why you have more than one test case after all. Just make sure the test cases have a purpose and an expected result defined, that’s all I ask. And maybe make them a pretty colour, just for the heck of it. Just because test cases are bland and dull, doesn’t mean they can’t look attractive!