Latest Entries »

Positive reinforcement

Filed Under: Office survival, Rants by Trish Khoo
Feb.19, 2010

“That’s one of the reasons we have testers. A great tester gives programmers immediate feedback on what they did right and what they did wrong. Believe it or not, one of the most valuable features of a tester is providing positive reinforcement. There is no better way to improve a programmer’s morale, happiness, and subjective sense of well-being than a La Marzocco Linea espresso machine to have dedicated testers who get frequent releases from the developers, try them out, and give negativeand positive feedback. Otherwise it’s depressing to be a programmer. Here I am, typing away, writing all this awesome code, and nobody cares. Boo hoo.” – Joel Spolsky (full article)

There’s something that’s been bugging me about this quote for the past couple of weeks, and I think I finally figured out what it is. As a tester, it’s important to establish a certain dynamic between oneself and each developer. This is because our job is essentially to find flaws in the stuff they create. Nobody really likes having their mistakes pointed out to them. If care is not taken, it’s easy for developers to resent testers for doing their jobs. Imagine if a tester said:

“This feature you’ve made is full of bugs. You’ve done a terrible job.”

Obviously this is going to make the developer upset. The main issue here is that the tester has passed judgement on the developer – “You’ve done a terrible job”. If a tester is to work well with developers, the tester cannot pass judgement on the code. Testers report what they see, they don’t say if it’s good or bad. They report expected and actual results, not correct and incorrect results.

If we flip this around and the tester says:

“This feature you’ve made is brilliant. You’ve done a fantastic job!”

This may make the developer happy, but the tester is still passing judgement on the code. What happens when the developer creates another feature and asks the tester “what do you think?” Will the tester say “well it’s okay, but there are a lot of bugs here, do better next time”? This is not a good dynamic between the tester and the developer – it’s not the tester’s place to say whether the feature is good or bad. The feature is not made for the tester. The feature is made for the customer.

Furthermore, GUI-level testers will rarely see whether a developer has done a great job or not. Developers rarely write the specifications for the product they build, and many do not even design the interfaces. If a developer comes up with some brilliant solution to a really challenging technical problem, it’s unlikely that the tester will ever know about it – they don’t see this process, they only see the end result. The only measure a tester has is how many bugs are found. And this isn’t necessarily an indication of a job well done either – it doesn’t take into account the complexity of the task or the time allocated to the task. And it’s not a tester’s job to research these factors in order to tell the developer whether they have done a good job.

This kind of positive reinforcement is best left to the development manager, not the test team.

So what do developers think of this? Seasoned developer and self-professed IT tinkerer Rob Sanders says:

An interesting topic.  My first thought is that a developer has the wrong mindset if they are looking for credit for a specific formula or process that they write or implement.  You’re right, what the rest of the world cares about is: quality code which produces the expected result within the scope of “acceptable performance”.

Therefore, what a developer should look for in their output is defect-free code (or as close to it as humanly possible) which meets specifications and runs as fast as they can tune it.  Credit for actual implementation specifics should come from other developers (let’s face it, in most cases only other developers normally would understand and appreciate it anyway).

So, I would expect kudos from QA Engineers if I produced a feature with lots of functionality (and good performance) and a low bug count.  I wouldn’t expect credit because I used design pattern X, Y or Z.  That should come from my supporting team of developers and architects.  On the flip side, I’d expect to have defects and performance issues pointed out.  The area which seems to generate the most friction seems to be (from my experience) when QA, Business Analysts and Developers have different interpretations of the stated requirements.

What do you think?

read more

5 Tips to Thwart Testers

Filed Under: Bugs, Manual testing by Trish Khoo
Jan.05, 2010

If you’re a web developer who does not enjoy seeing bugs assigned to you, you’re in luck – I’m about to reveal 5 easy tips to prevent bugs. Try these tricks on your own code before you check in your change, and you may find less bugs in your inbox the next morning.

  1. Run the code and verify visually that it does what it’s meant to do. Very important.
  2. Try it out in IE (or another different browser). These are my favourite browsers for testing, in order: Internet Explorer, Safari, Chrome, Firefox. Why is that? Because that’s usually the reverse order of favourite browsers for programmers. Any UI-level change is almost guaranteed to have issues in another browser.
  3. Update to the latest version and compile before testing on your machine. Other people’s changes could affect yours.
  4. Try it out in the development environment. A change of environment could affect the way your feature behaves.
  5. Try the change in an end-to-end scenario. So for instance, you’ve changed one page of a user signup process. The first thing I’ll do is check that page. The second thing I’ll do is sign up the user and login. The way one element is displayed is often dependant on previous steps, and often affects subsequent steps.

And, bonus tip:

  1. Go one step beyond the obvious. The third thing I’ll do is think – is there another way to do this same operation? For example, if changing a file saving operation, as well as testing that the drop down menu command works, try the keyboard shortcut as well.

Follow these steps, and I can assure you those pesky testers will be easing up on the “create new bug” button in no time.

read more

Developer maturity model

Filed Under: Random musings by Trish Khoo
Dec.24, 2009

The other day I was thinking about developers and their attitudes towards testers and came up with this maturity model:

Level 1: “We don’t need testers” The pre-tester attitude of developers who have never worked with a tester, and have never seen the need for one.

Level 2: “The testers are criticizing my work” A common attitude of developers who have only recently begun working with testers for the first time. These developers may react defensively to bugs raised, and will often be very aggressive or uncooperative with testers.

Level 3: “The testers are there to validate my work” Developers with this attitude have recognised testers as a useful resource, but have come to depend on them to pick up obvious bugs in their work. This may lead to careless work.

Level 4: “Testers help me turn good work into excellent work” These are developers with a high level of experience, who take care in their work. They recognise that if they deliver quality work to the testers, then the testers will have time to find more complex bugs. This will result in a high standard of work at a faster rate.

Level 5: “Testers help the team to produce excellent software” These developers understand the difference between writing code and making software. They have undergone a paradigm shift towards realising that quality is the whole team’s responsibility.

I ran this by some friends of mine (who are “level 5″ software developers, of course). I initially thought levels 4 and 5 would be combined, but after discussing with Rob Sanders, I decided that they are probably separate mentalities. When asked what triggered the move from one level to the next, they replied simply “experience”.

read more

Checklists > scripts

Filed Under: Manual testing, Test cases by Trish Khoo
Dec.23, 2009

Earlier this year, I started work testing a system that had been developed over about 5 years and there were no documented test cases. There was a huge suite of automated tests, but no test cases whatsoever. I didn’t have time to write 5 years’ worth of test scripts for a system I knew nothing about, so I decided to try something radical – checklists instead of scripts. For an excellent description on the difference between checklists vs scripts, see  Cem Kaner’s presentation on the value of checklists (an article I only found just now, thanks to Quick Testing Tips).

I have to say, I had a few doubts about the approach, and at first thought maybe it could only be a temporary solution, but I have to say it has proven to be quite effective and efficient.

Some benefits are:

  • Lots of time saved upfront not needing to write test scripts
  • Quick and easy maintenance – checklists take much less time to update than detailed scripts
  • Checklists force me and other testers to think about how the system works, and find out how to do things if we don’t know how (the users might not know how either – how easy is this information to find?)
  • Checklists allow a different path to be taken each time (ie. different things are done to achieve the same goal), which results in greater coverage, and more bugs found

Some disadvantages:

  • If non-testers have to use the checklists, they may not cover as much ground as experienced testers, and test coverage may decrease (compared to using test scripts)
  • Some information that might have been included in scripts is lost. I supplement this by keeping some reference information about the application in a wiki (Fitnesse actually, so automated tests can one day live on the same page as the relevant information)

If anyone else has had experience using checklists in place of scripts, I’d be very interested in hearing about the results – good or bad.

read more

Performance testing on a budget

Filed Under: Automation, Manual testing by Trish Khoo
Dec.18, 2009

If you’re a software tester, sooner or later someone will ask you to be involved in performance testing. If you don’t know much about performance testing, it can be pretty daunting. Even if you do know a lot about performance testing, it can still be pretty daunting, especially if there are strict time and budget restrictions…and I’m inclined to guess that there usually are. A little while ago, Anne-Marie Charrett invited me to contribute to her article “Ways to approach application performance testing on a tight budget”, which is definitely worth reading for anyone involved in performance testing.

My advice here comes in the form of a story, which I think we can all learn from. Once upon a time, there was a big web app project. The project was almost complete and was due within just a couple of weeks. Around this time, a tester called Mish joined the project and the other testers told her that she was to take over the task of performance testing from them. Well! Mish had never done performance testing before, and told them so. The other tester told her he hadn’t done it before either, but they were on a tight budget and they just had to make do with what they had. The other tester explained to Mish that he had been using JMeter to record benchmarks against an older version of the application, and he was going to compare the times to the new version of the application.

Immediately Mish realised it was never going to work. The new version of the application was running on a complicated 6-box cluster environment, whereas the old version of the application was running on an old abandoned workstation that someone had found under a desk somewhere, with a big sign on it saying “DO NOT TOUCH”. Mish tried to point this out, but was told it was the best they could do and she would just have to deal with it. So Mish abandoned benchmarking and instead just measured the loading times of some operations in the application, presented the results and hoped for the best.

Management said that the results were fine and that they were ready to deploy. So then they added the production data to the application’s database. Hundreds of thousands of rows of production data. Well, didn’t that slow down the performance considerably! The project was delayed another month as a result.

What can we learn from this story?

  • Always perform production testing on an environment that is similar to the production environment in as many ways as possible. This includes the hardware, the software, the other software running on the server, the test data, and so much more!
  • Performance testing, if not done correctly, can lead to very inaccurate results.
  • There is MUCH more to performance testing than just page load times. Research this skill thoroughly before just trying to wing it.
  • Performance problems are often not trivial to fix. Do not leave performance testing to the end of the project lifecycle. Start early!
read more

The joy of bugfinding

Filed Under: Bugs, Random musings by Trish Khoo
Dec.08, 2009

I said this over dinner the other night and was told I had to blog it.

I test .NET web applications. If I see that lovely yellow Server Error stack trace, it’s a good day. If I see it 2 or 3 times, it’s a really good day.

But it has to be challenging, it’s no fun if I just login and bam, there it is. That’s not a bug-finding achievement, that’s just pathetic. It’s like if you’re hunting deer and a deer stands in front of you and shoots itself in the head. What a let-down!

read more

The bearer of bad news

Filed Under: Office survival by Trish Khoo
Dec.03, 2009

I was just thinking about this the other day, and then just now I saw Dave Whalen’s post on “ugly baby syndrome”. Software testing’s a difficult job, and I don’t mean technically challenging, I mean socially difficult. Often we’re associated with an attitude of negativity. This is our conundrum:

  • Our job is to find faults in the software.
  • If we say the software is fine and it isn’t, it looks like we didn’t do our jobs.
  • If we say the software isn’t fine and it is, it looks like we don’t know what we’re talking about.
  • If we say the software isn’t fine and we’re correct, we’ve done our jobs right, but now other people have to do more work to fix it.
  • If we say the software IS fine and we’re correct, then champagne and pats on the back all around (how often does that happen?)

We’d all like to be able to say the software’s fine, but it’s a risky move – we can never state with absolute certainty that the software is bug-free.  So testers say things like “all of our tests have been run and no bugs have been found”. Or “to the best of my knowledge, the system should operate as per requirements”. You’d think we’d make pretty good politicians, really.

In addition, testing is one of the more thankless jobs in IT – how often do we get thanked for the bugs we prevented? After I took this job, I was given kudos for a job well done after a release and I was totally blown away, because that hardly ever happens in testing.

So here’s my advice to any testers, or wannabe testers:

  • Be prepared to look like an idiot. Sooner or later you’re going to raise a bug that’s not really a bug, or raise hell over some bug that only occurred because you had a donut leaning on the Windows key. Big deal. Raising non-bugs is an unfortunate side-effect of testing. Better to raise the bug than risk the bug being missed.
  • Stay real, but stay positive. Sounds strange to stay positive when your job is focused on the negative. It’s a perspective thing- you’re not breaking the product, you’re helping the team fix the product. Remember you’re allowed to compliment the product as well.
  • Be objective. You could try using passive voice, if you find objectivity difficult. Don’t say things like “this is crap” or “this is broken”. Instead describe the behaviour you’re seeing. It’s okay to have opinions, but make sure you differentiate between your personal opinion and your objective analysis, e.g. “the new sign-up screen works correctly, but I personally found the layout a bit difficult to read.” (instead of “the layout on the sign-up screen is too difficult to read”)
  • Stay cool in the face of danger. Sooner or later you will encounter hostility – you’re the messenger, and you may get shot because of it. Just keep your cool, stay positive, and try to work with the person (or persons!) to focus on the solution rather than the problem.
  • Promote your work. The benefits of testing are not always obvious to the rest of the team, or to the higher-ups. Nobody is going to dig deep to find the benefit in your contribution – it’s up to you to show them what you’ve achieved. Report on how many bugs you found for a release (and therefore prevented from entering production). Share metrics with the team showing an increase in quality. If developers are producing better work (and you’re finding less bugs), comment on how much this has improved.
read more

Computer Science degrees and relevance to the real world

Filed Under: Rants by Trish Khoo
Dec.02, 2009
I was reading this article and it reminded me of when I first entered the workforce and wondered just how relevant my university degree really was to an actual software development job. I still think computer science university courses could be a lot better, but I don’t agree with modeling it on actual real-life projects, reasons being:
- They can learn this better from doing work experience (which looks great on the resume too)
- It teaches a single way of doing things, which doesn’t encourage adaptability to different processes and methods
- It will probably teach them all to be programmers
The reason universities teach theories and concepts is because it gives a solid, adaptable grounding in a field. Information Technology is a field that is constantly and rapidly evolving. Universities are training people for IT jobs that don’t even exist yet. How do they do that?
Computer science graduates don’t all end up as programmers. They can end up as testers (hi!), database administrators, business analysts, project managers, web designers, system administrators, and heaps of other jobs that probably have yet to be invented.
Actually, at my university (University of Queensland), we actually did a team project subject with monthly deadlines, documentation required and version control. I don’t think it was really that useful, because it wasn’t real. In a student project, you hand in your work, it gets graded, and that’s the end. In a real project, you deliver your work to the customer and if it’s not good enough then you have a responsibility to fix it, and explain why it wasn’t done right the first time.
So, what improvements would I like to see made to university courses? For starters of course I’d like to see more testing concepts introduced. In fact, there could be a whole heap of useful non-programming subjects added like requirements gathering and interface design.
I hope as the software industry evolves, universities will incorporate software testing theories into their education system. Then maybe someday soon we’ll see computer science graduates who want a career in testing, rather than seeing it as the job you take when you can’t get a real programming job. :)

I was reading this article and it reminded me of when I first entered the workforce and wondered just how relevant my university degree really was to an actual software development job. I still think computer science university courses could be a lot better, but I don’t agree with modeling it on actual real-life projects, reasons being:

- They can learn this better from doing work experience (which looks great on the resume too)

- It teaches a single way of doing things, which doesn’t encourage adaptability to different processes and methods

- It will probably teach them all to be programmers

The reason universities teach theories and concepts is because it gives a solid, adaptable grounding in a field. Information Technology is a field that is constantly and rapidly evolving. Universities are training people for IT jobs that don’t even exist yet. How do they do that?

Computer science graduates don’t all end up as programmers. They can end up as testers (hi!), database administrators, business analysts, project managers, web designers, system administrators, and heaps of other jobs that probably have yet to be invented.

Actually, at my university (University of Queensland), we actually did a team project subject with monthly deadlines, documentation required and version control. I don’t think it was really that useful, because it wasn’t real. In a student project, you hand in your work, it gets graded, and that’s the end. In a real project, you deliver your work to the customer and if it’s not good enough then you have a responsibility to fix it, and explain why it wasn’t done right the first time.

So, what improvements would I like to see made to university courses? For starters of course I’d like to see more testing concepts introduced. In fact, there could be a whole heap of useful non-programming subjects added like requirements gathering and interface design.

I hope as the software industry evolves, universities will incorporate software testing theories into their education system. Then maybe someday soon we’ll see computer science graduates who want a career in testing, rather than seeing it as the job you take when you can’t get a real programming job. :)

read more

Test tool roundup – data generation

Filed Under: Tools by Trish Khoo
Nov.30, 2009

I guess not all of us have the time and imagination to think up realistic test data, especially when we need about 8,000 rows of it. Here are some tools that might help you out:

Lorem Ipsum – The classic “filler” text used by typesetters everywhere. This stuff is great because you can generate specific character amounts, and it has no unusual formatting (just full stops, commas, whitespace and alphabet characters). Not very realistic? True, but if you want realistic, you can go to just about any online document and grab text from there. This is nice and sanitised so you know exactly what’s in there.

GenerateData.com – Great for creating large amounts of custom data, particularly user data. It generates the data in a few different formats, including Excel, CSV and SQL – really handy for importing.

Fake Name Generator – Creates user data based on gender, name set (language) and country. Great for seeing how your app fares against foreign names. For fans of The Sims, it even has a generator for naming a whole Sims family.

I found out about some of these tools from Quick Testing Tips and @dailytestingtip. Thanks guys!

read more

Daily Testing Tip

Filed Under: News by Trish Khoo
Nov.11, 2009

If you’re a tester on twitter, follow @dailytestingtip for handy testing tips each day. You may even see a tip from me or the Maverick Tester occasionally.

If you’re not on twitter, why not? Come on, get with the times and join the fun!

read more
Powered by WordPress. Theme: Motion by 85ideas.