5 Tips to Thwart Testers
If you’re a web developer who does not enjoy seeing bugs assigned to you, you’re in luck – I’m about to reveal 5 easy tips to prevent bugs. Try these tricks on your own code before you check in your change, and you may find less bugs in your inbox the next morning.
- Run the code and verify visually that it does what it’s meant to do. Very important.
- Try it out in IE (or another different browser). These are my favourite browsers for testing, in order: Internet Explorer, Safari, Chrome, Firefox. Why is that? Because that’s usually the reverse order of favourite browsers for programmers. Any UI-level change is almost guaranteed to have issues in another browser.
- Update to the latest version and compile before testing on your machine. Other people’s changes could affect yours.
- Try it out in the development environment. A change of environment could affect the way your feature behaves.
- Try the change in an end-to-end scenario. So for instance, you’ve changed one page of a user signup process. The first thing I’ll do is check that page. The second thing I’ll do is sign up the user and login. The way one element is displayed is often dependant on previous steps, and often affects subsequent steps.
And, bonus tip:
- Go one step beyond the obvious. The third thing I’ll do is think – is there another way to do this same operation? For example, if changing a file saving operation, as well as testing that the drop down menu command works, try the keyboard shortcut as well.
Follow these steps, and I can assure you those pesky testers will be easing up on the "create new bug" button in no time.
Developer maturity model
The other day I was thinking about developers and their attitudes towards testers and came up with this maturity model:
Level 1: "We don't need testers" The pre-tester attitude of developers who have never worked with a tester, and have never seen the need for one.
Level 2: "The testers are criticizing my work" A common attitude of developers who have only recently begun working with testers for the first time. These developers may react defensively to bugs raised, and will often be very aggressive or uncooperative with testers.
Level 3: "The testers are there to validate my work" Developers with this attitude have recognised testers as a useful resource, but have come to depend on them to pick up obvious bugs in their work. This may lead to careless work.
Level 4: "Testers help me turn good work into excellent work" These are developers with a high level of experience, who take care in their work. They recognise that if they deliver quality work to the testers, then the testers will have time to find more complex bugs. This will result in a high standard of work at a faster rate.
Level 5: "Testers help the team to produce excellent software" These developers understand the difference between writing code and making software. They have undergone a paradigm shift towards realising that quality is the whole team's responsibility.
I ran this by some friends of mine (who are "level 5" software developers, of course). I initially thought levels 4 and 5 would be combined, but after discussing with Rob Sanders, I decided that they are probably separate mentalities. When asked what triggered the move from one level to the next, they replied simply "experience".
Checklists > scripts
Earlier this year, I started work testing a system that had been developed over about 5 years and there were no documented test cases. There was a huge suite of automated tests, but no test cases whatsoever. I didn't have time to write 5 years' worth of test scripts for a system I knew nothing about, so I decided to try something radical - checklists instead of scripts. For an excellent description on the difference between checklists vs scripts, see Cem Kaner's presentation on the value of checklists (an article I only found just now, thanks to Quick Testing Tips).
I have to say, I had a few doubts about the approach, and at first thought maybe it could only be a temporary solution, but I have to say it has proven to be quite effective and efficient.
Some benefits are:
- Lots of time saved upfront not needing to write test scripts
- Quick and easy maintenance - checklists take much less time to update than detailed scripts
- Checklists force me and other testers to think about how the system works, and find out how to do things if we don't know how (the users might not know how either - how easy is this information to find?)
- Checklists allow a different path to be taken each time (ie. different things are done to achieve the same goal), which results in greater coverage, and more bugs found
Some disadvantages:
- If non-testers have to use the checklists, they may not cover as much ground as experienced testers, and test coverage may decrease (compared to using test scripts)
- Some information that might have been included in scripts is lost. I supplement this by keeping some reference information about the application in a wiki (Fitnesse actually, so automated tests can one day live on the same page as the relevant information)
If anyone else has had experience using checklists in place of scripts, I'd be very interested in hearing about the results - good or bad.
Performance testing on a budget
If you're a software tester, sooner or later someone will ask you to be involved in performance testing. If you don't know much about performance testing, it can be pretty daunting. Even if you do know a lot about performance testing, it can still be pretty daunting, especially if there are strict time and budget restrictions...and I'm inclined to guess that there usually are. A little while ago, Anne-Marie Charrett invited me to contribute to her article "Ways to approach application performance testing on a tight budget", which is definitely worth reading for anyone involved in performance testing.
My advice here comes in the form of a story, which I think we can all learn from. Once upon a time, there was a big web app project. The project was almost complete and was due within just a couple of weeks. Around this time, a tester called Mish joined the project and the other testers told her that she was to take over the task of performance testing from them. Well! Mish had never done performance testing before, and told them so. The other tester told her he hadn't done it before either, but they were on a tight budget and they just had to make do with what they had. The other tester explained to Mish that he had been using JMeter to record benchmarks against an older version of the application, and he was going to compare the times to the new version of the application.
Immediately Mish realised it was never going to work. The new version of the application was running on a complicated 6-box cluster environment, whereas the old version of the application was running on an old abandoned workstation that someone had found under a desk somewhere, with a big sign on it saying "DO NOT TOUCH". Mish tried to point this out, but was told it was the best they could do and she would just have to deal with it. So Mish abandoned benchmarking and instead just measured the loading times of some operations in the application, presented the results and hoped for the best.
Management said that the results were fine and that they were ready to deploy. So then they added the production data to the application's database. Hundreds of thousands of rows of production data. Well, didn't that slow down the performance considerably! The project was delayed another month as a result.
What can we learn from this story?
- Always perform production testing on an environment that is similar to the production environment in as many ways as possible. This includes the hardware, the software, the other software running on the server, the test data, and so much more!
- Performance testing, if not done correctly, can lead to very inaccurate results.
- There is MUCH more to performance testing than just page load times. Research this skill thoroughly before just trying to wing it.
- Performance problems are often not trivial to fix. Do not leave performance testing to the end of the project lifecycle. Start early!
The joy of bugfinding
I said this over dinner the other night and was told I had to blog it.
I test .NET web applications. If I see that lovely yellow Server Error stack trace, it's a good day. If I see it 2 or 3 times, it's a really good day.
But it has to be challenging, it's no fun if I just login and bam, there it is. That's not a bug-finding achievement, that's just pathetic. It's like if you're hunting deer and a deer stands in front of you and shoots itself in the head. What a let-down!
The bearer of bad news
I was just thinking about this the other day, and then just now I saw Dave Whalen's post on "ugly baby syndrome". Software testing's a difficult job, and I don't mean technically challenging, I mean socially difficult. Often we're associated with an attitude of negativity. This is our conundrum:
- Our job is to find faults in the software.
- If we say the software is fine and it isn't, it looks like we didn't do our jobs.
- If we say the software isn't fine and it is, it looks like we don't know what we're talking about.
- If we say the software isn't fine and we're correct, we've done our jobs right, but now other people have to do more work to fix it.
- If we say the software IS fine and we're correct, then champagne and pats on the back all around (how often does that happen?)
We'd all like to be able to say the software's fine, but it's a risky move - we can never state with absolute certainty that the software is bug-free. So testers say things like "all of our tests have been run and no bugs have been found". Or "to the best of my knowledge, the system should operate as per requirements". You'd think we'd make pretty good politicians, really.
In addition, testing is one of the more thankless jobs in IT - how often do we get thanked for the bugs we prevented? After I took this job, I was given kudos for a job well done after a release and I was totally blown away, because that hardly ever happens in testing.
So here's my advice to any testers, or wannabe testers:
- Be prepared to look like an idiot. Sooner or later you're going to raise a bug that's not really a bug, or raise hell over some bug that only occurred because you had a donut leaning on the Windows key. Big deal. Raising non-bugs is an unfortunate side-effect of testing. Better to raise the bug than risk the bug being missed.
- Stay real, but stay positive. Sounds strange to stay positive when your job is focused on the negative. It's a perspective thing- you're not breaking the product, you're helping the team fix the product. Remember you're allowed to compliment the product as well.
- Be objective. You could try using passive voice, if you find objectivity difficult. Don't say things like "this is crap" or "this is broken". Instead describe the behaviour you're seeing. It's okay to have opinions, but make sure you differentiate between your personal opinion and your objective analysis, e.g. "the new sign-up screen works correctly, but I personally found the layout a bit difficult to read." (instead of "the layout on the sign-up screen is too difficult to read")
- Stay cool in the face of danger. Sooner or later you will encounter hostility - you're the messenger, and you may get shot because of it. Just keep your cool, stay positive, and try to work with the person (or persons!) to focus on the solution rather than the problem.
- Promote your work. The benefits of testing are not always obvious to the rest of the team, or to the higher-ups. Nobody is going to dig deep to find the benefit in your contribution - it's up to you to show them what you've achieved. Report on how many bugs you found for a release (and therefore prevented from entering production). Share metrics with the team showing an increase in quality. If developers are producing better work (and you're finding less bugs), comment on how much this has improved.
Computer Science degrees and relevance to the real world
I was reading this article and it reminded me of when I first entered the workforce and wondered just how relevant my university degree really was to an actual software development job. I still think computer science university courses could be a lot better, but I don't agree with modeling it on actual real-life projects, reasons being:
- They can learn this better from doing work experience (which looks great on the resume too)
- It teaches a single way of doing things, which doesn't encourage adaptability to different processes and methods
- It will probably teach them all to be programmers
The reason universities teach theories and concepts is because it gives a solid, adaptable grounding in a field. Information Technology is a field that is constantly and rapidly evolving. Universities are training people for IT jobs that don't even exist yet. How do they do that?
Computer science graduates don't all end up as programmers. They can end up as testers (hi!), database administrators, business analysts, project managers, web designers, system administrators, and heaps of other jobs that probably have yet to be invented.
Actually, at my university (University of Queensland), we actually did a team project subject with monthly deadlines, documentation required and version control. I don't think it was really that useful, because it wasn't real. In a student project, you hand in your work, it gets graded, and that's the end. In a real project, you deliver your work to the customer and if it's not good enough then you have a responsibility to fix it, and explain why it wasn't done right the first time.
So, what improvements would I like to see made to university courses? For starters of course I'd like to see more testing concepts introduced. In fact, there could be a whole heap of useful non-programming subjects added like requirements gathering and interface design.
I hope as the software industry evolves, universities will incorporate software testing theories into their education system. Then maybe someday soon we'll see computer science graduates who want a career in testing, rather than seeing it as the job you take when you can't get a real programming job. :)
Test tool roundup – data generation
I guess not all of us have the time and imagination to think up realistic test data, especially when we need about 8,000 rows of it. Here are some tools that might help you out:
Lorem Ipsum - The classic "filler" text used by typesetters everywhere. This stuff is great because you can generate specific character amounts, and it has no unusual formatting (just full stops, commas, whitespace and alphabet characters). Not very realistic? True, but if you want realistic, you can go to just about any online document and grab text from there. This is nice and sanitised so you know exactly what's in there.
GenerateData.com - Great for creating large amounts of custom data, particularly user data. It generates the data in a few different formats, including Excel, CSV and SQL - really handy for importing.
Fake Name Generator - Creates user data based on gender, name set (language) and country. Great for seeing how your app fares against foreign names. For fans of The Sims, it even has a generator for naming a whole Sims family.
I found out about some of these tools from Quick Testing Tips and @dailytestingtip. Thanks guys!
Daily Testing Tip
If you're a tester on twitter, follow @dailytestingtip for handy testing tips each day. You may even see a tip from me or the Maverick Tester occasionally.
If you're not on twitter, why not? Come on, get with the times and join the fun!
Testing vs Checking
I wish I had seen this post - Testing vs Checking by Michael Bolton - about a year ago when I was trying to explain this concept to my manager at the time. The manager's opinion was that testing = checking, and as soon as checking was done, that meant that testing was done. The terminology of "checking" vs "testing" didn't occur to me at the time, and I poorly tried to explain it as "basic testing" and "real testing", which was not received well.
Michael Bolton's explanation is very good. Bookmark this article now. You will need it in the following situations:
- When you're explaining to someone why a tester's sole purpose is not to check a developer's work for them
- When you need to explain why testing is not finished just because all of the test cases have passed
- When you need to explain why actual human testers can never be replaced by automated test suites
- When you need to explain why testers do need expected results (specifications, requirements, etc) in order to do checking
- ...and probably many more situations where you find yourself having to explain what it is that testers actually do
Thanks Michael. You are an asset to those of us cursed with feeble explanations.