Expert Game

Twelve years after I started my testing career, I still feel like a student.

That’s why I felt so honored last week in being asked to give the Commencement Address for ITT’s spring quarter graduation ceremony.

One of their career advisors had seen me give a guest lecture to a software development class and thought I’d be a good fit, and I knew right away what I wanted to say to them.

I mean, I imagined a few rows of graduates in cap and gown, tassles ready to be turned, ready to party. What could I possibly say to them that had any hope of getting their attention?

What came to mind was something that helped me when I first got started.

I decided to tell them this:

“In programming, there’s a concept known as an If / Then statement. If condition A is met, THEN do instruction B. The code is compiled, it runs, it looks for the condition, and executes some kind of algorithm when it finds the right condition.

But in my career as a test manager, Ive learned that humans are programs, too. We are only as good as our programming. We get tired, nervous, moody when certain conditions are met.

So I offer this snippet of code to add to your existing programming in case those conditions occur. This is that If-Then statement:

IF:

  • you are able to get important things done
  • you are seen learning things on your own
  • you are seen trying to do things even though you don’t know how
  • you don’t hide your ignorance, but also don’t rest on it
  • you share freely the things that you know
  • you honor what other people know
  • you know more often than not how to find out what you don’t know
  • you know how to ask for help
  • you offer help to people ON THEIR OWN TERMS

THEN

  • no one will care whether you succeed by learning or succeed by already knowing
  • no one will care if you mess up occasionally because they assume you learn from it
  • no one will care if you forget (or don’t know) any given fact or method at any given time

… and you will be treated as if you’re smart and useful, even though everyone knows you have a lot to learn

This is a formula that has been true for me, a journalism major, author’s son, and autobiographer philosopher who is now considered an internationally recognized expert in exploratory software testing.

Baby Oracles

On February 16, I became a father.

This is related to testing in more ways than I can count, but let me give you my favorite.

Ever since I announced my wife was expecting, all kinds of parents have given me all kinds of advice, and now it’s time to fulfill the promise I made to most of them — to report how right they were.

In testing, “rightness” is determined by oracles — methods or principles used to recognize a problem. Without oracles, testing isn’t testing, it’s “touring.”

Oracle #1: Most full-term babies weigh between 5 and 9 pounds.

This oracle is probably why the medical staff gave an audible “whoa” when little Charlotte weighed in at 11 pounds. It’s also why I said “whoa” when another baby delivered that day weighed in at 13 pounds.

Oracle #2: Babies eat every two to three hours.

Yep, and here’s an interesting systems-thinking corollary. Charlotte, being a bit bigger than most newborns has a stomach that holds more milk, which means she’s satisifed for a longer time, which means she sleeps longer, which means *I* sleep longer.

Oracle #3: Babies are sometimes jaundiced when they are born.

True for Charlotte, but the other oracle here when a nurse said that her “Billy Rubin” level needed to be watched over the next couple of days. I didn’t know who Billy Rubin was, but suspected he was either a doctor or a scientist who had something to do with the discovery of jaundice. Now, some of you are laughing at this because you have an oracle that tells you that “Billy Rubin” is actually not a person but “bilirubin” — an artifact of the blood that the liver needs to process. I found this out by doing an internet search — a common course of action for all concerned parents who are hear a doctor’s concern and need to know more. Google is the world’s greatest source of oracles. One of my favorite Google hits was a reference to the novel, Silence of the Lambs. Hannibal Lector, trying to be clever, led the kidnapped girl’s mother, a senator, on a wild goose chase by saying the killer was Billy Rubin. Clarisse later figured out it was false oracle.

Mythbusters is my favorite show on TV right now because they take urban legends that have become “fact” by so many people who trust them (e.g. “drinking pop rocks and soda will make your stomach explode”) and they stage experiments to prove or disprove them.

So when you’re testing, think about how it is that you know what you know. Are you sure?

What is rapid test planning?

Rapid test planning is a process of cultivating test ideas within minutes, not days. Its opposed to rigorous test planning which strives to find every possible test. The trade-off is time vs. thoroughness because often were given tight constraints for testing. Management wants test results quickly and we can either freeze, get angry, or see what we can do.

I’ve experienced paralysis and anger many times in my 12 years of testing, but the more I stay in this business, the more I find myself seeing what I can do. These days, I consider it a challenge when a client says you have three days to test.

Enter rapid test planning. For me, it involves one or two exploratory testing “sessions” that result in a list of test ideas, issues, and risks. Exploratory testing is test design and execution happening at the same time, sometimes in the same second, as the tester gathers and analyzes data. The tester decides whats important, and the emphasis is on learning about the product.

A session could be 30 minutes or an hour or three hours, but the mission is the same what can we test?

At Quardev, clients usually don’t give us a lot of time to test, so our promise to them is to focus more on delivering information than creating written test cases. Thats why I often use rapid test planning.

Think of an American-style football game. You have 4 quarters, or roughly one hour of clock time to beat the other team by designing plays that advance the ball down 100 yeards of turf while the other team tries to stop you. Plays can involve throwing the ball or running with the ball until you reach the end of the field for a score. When the other team has the ball, it’s your turn to stop them from scoring, so each team has plans for offense and defense.

Let’s say my football team is the software were testing. Lets also pretend that I am a coach being interviewed. The sports reporter asks me a spontaneous question during the off-season to catch me off-guard to get a good sound bite for his story:

If you were playing the Seahawks today, what would be your game plan?

In answering rapidly to give them a great quote, I might think in terms of the following 5 elements:

  • Structure: The team’s composition.
  • Function: The team’s capabilities.
  • Data: What the team might “process” – i.e. what we do with the ball.
  • Platform: What the team depends upon.
  • Operations: How Ill use my teams skills.

Using a mnemonic like SFDPO is considered a heuristic a fallible method for solving a problem. Its not meant to be thorough or foolproof, just enough to start you on the road quickly to a useful solution. Using this particular heuristic (remembered as SFDPO or San Francisco Depot), I might say something like:

Ive got a deep bench (STRUCTURE), so Im confident that our past performance on special teams against the Seahawks has been great (FUNCTION) when we pass the ball instead of run it (DATA). Given my star players arent that healthy right now (PLATFORM), Id start them in the third quarter, especially if that wind picks up out there (OPERATIONS).

This sound bite gives you an idea of how a coach might inventory their ideas on-the-fly. As coach, I might be wrong about the plan since its just a gut check, so Ill likely adapt it once the game starts and get more information.

For more heuristics like SFDPO to use in rapid test planning, click here.

Execution

Ok, so heres how I do it:

  • The gauntlet is thrown down: Test this! You have 3 days.
  • I ask questions to get more context for my mission or information to enhance my plan
  • I start the session (setting the table for dinner, in effect), asking for anything I can get:
    • the software
    • specs or product docs
    • project docs
    • marketing literature
    • names of knowledgeable project staff who can help me later
    • a projector and a PC with a web connection (to do on-the-fly research)
  • I think of things that trigger other ideas
  • I end the session and go over it with as many stakeholders as possible
  • go to step 2

OR

Declare that the plan is good enough for now to get testers engaged on their own exploratory testing sessions to find bugs with minimal supervision. Click here for ideas on how to tell if your plan is good enough.

For more on the mechanics of exploration and analysis, see this paper.

Other good sources:

  1. My brother James Bach at his testing consulting company Satisfice, most famous for offering the Rapid Software Testing course
  2. Michael Bolton, principal consultant at DevelopSense, who also gives the Rapid Software Testing course.
  3. Robert Sabourin, principal consultant at Amibug.
  4. James Lyndsay, principal consultant at Workroom Productions.

All-Pairs

I got called into a conference call today because of an emerging question a client had about combination testing.

They wanted us to test their online survey, an application which consisted of several questions, most of which had an array of answers from which the respondent could pick. At the end of the survey, the respondent gets a score.

One section of the survey had 40 questions. Each question had anywhere from 2 to 28 possible answers. This made a total of 23 quintillion combinations to test.

So what to do when they only have a few weeks to release?

Enter a heuristic combination method called all-pairs. All-pairs is a mathematical technique that takes a table of options and pairs an option in one column with each option in each of the other column.

This approach is useful for flushing out the risk bugs that exist when two options are paired.

There are two FREE tools I know about that will do this kind of analysis: all-pairs tool from James Bach and PICT, a free tool from Microsoft.

Using both tools, I found that the 23 quintillion options were paired to just 673 test cases. The tools produce a nifty Excel-readable table of test cases, consisting of rows for the tester to follow — telling them which answers to select for each question.

The important point to note when using all-pairs is that it is a heuristic — a fallible mehtod for solving a problem — aka “a rule of thumb.” As such, it should be told to stakeholders that it’s not perfect, but it can help expose certain risks quickly.

For the PICT tool (which also does triples and a variety of other options), click here. For my brother’s free all-pairs tool written in Perl, click here.

TV Shows for Testing

I love tv. Perhaps too much. But I don’t sweat my addiction too much because most of the shows I like have a testing component to them.

Notice that your favorite shows, movies, books all have an important element that you’ll see in testing.

Conflict.

The difference between “desired” and “actual” is called a problem (It’s also called a bug.)

There are many defintions of testing, but my favorite is the discovery of problems as you assess capabilities.

Here’s my list of shows and movies that focus on the juxtaposition of problems and capabilities:

  1. Iron Chef — Food Network — chefs have one hour to cook 5 dishes for a panel of judges. A secret ingredient is revealed at the beginning of the show that the chef must use in all of the dishes — all the while competing with another chef.
  2. Mythbusters — Discovery — Two special effects guys with an extensive array of tools and props, set out to confirm or refute several urban legends.
  3. Ramsay’s Kitchen Nightmares — BBC America — a notoriusly irascible chef serves as a consultant to see if he can turn around England’s failing restuarants.
  4. Survivorman — Discovery — a guy drops himself into remote places like wilderness, desert, and snow pack armed only with a camera and a leatherman tool and the clothes on his back. His mission is to get himself out of danger, filming his journey.
  5. America’s Test Kitchen — PBS — culinary experts try out different tools, gadgets, and recipes, sometimes head-to-head.

Movies I can watch over and over again for their testing parallels:

  1. Apollo 13 — astronauts trapped in a failing capsule with only a few days to live.
  2. Super Size Me — a healthy man eats nothing but food listed on McDonald’s menu for breakfast, lunch, and dinner 30 days. What happens to him?
  3. The Matrix — Life is a nothing but a computer simulation — what bugs have you seen in the program?

Seattle CSI Files

Here are the notes I took after Mark Hanf, a detective from Seattle CSI came to speak to (QASIGwww.qasig.org) on 1/18/07:

  • “We are asked to go to different locations”; parallel: think about testing on different computer platforms.
  • “Look up, not just straight ahead”; parallel: change your perspective when thinking of software tests to run.
  • “Look in the garbage; we go into toilets quite a bit”; parallel: software bugs could reside in places we don’t associate with normally having problems.
  • “Proper documentation with photos”; parallel: we often document our tests and report our finding with screenshots.
  • “Can’t be afraid of heights”; parallel: can’t be afraid of testing on new platforms.
  • “Sometimes you have to match the bullet even though the crime is ‘solved'”; parallel: even though you have found the bug, there may be another cause.
  • “Crime scenes might have CS gas residue”; parallel: “we may be digging in an area that complicates our ability to find bugs.
  • Tools: reflective UV imaging screen, forensic stepping plates, sifting screens; parallel: we have special tools as well (inControl, log file tracing, LoadRunner).
  • “We must gather, document, and demonstrate in court that we did everything possible”; parallel: software projects have “bug juries” that we are often called in to testify in front of to make our case.
  • “We study different disciplines: entomology, odontology, etc.”; parallel: we also study different domains… cognitive psychology for usability, brain physiology, and Crime Scene Investigation!
  • “Everybody’s interested in coming in and going right to the dead body”; parallel: We go right for the features that attract us or that are easy to test.
  • “Detectives should cut their own path to an outside crime scene”; parallel: there is more than one way to reproduce or find a software bug.
  • “You get to the scene, are briefed in an initial walkthrough”; parallel: we have client kick-off meetings that tell us what to focus on and where bugs may likely be hiding.
  • “Footwear impressions and fingerprints are there whether we see them or not”; parallel: same is true for software defects… they are almost always hidden.
  • “Sometimes you’re concerned about the floor, but can’t deal with it then and there”; parallel: bugs mask other bugs… we’re concerned about one feature but may not have time to test it right then.
  • “Take photos with scale and without scale”; parallel: when filing a bug, think about its impact not just to the user but on other programs on the system.
  • “Juries expect a lot more, so in some cases, we have to entertain (re: animation) as well as inform”; parallel: sometimes filing a bug is not enough, we have to be an advocate for what we find.
  • “Defense attorneys could discount elements of our case, so we have to be thorough and careful”; parallel: same is true when we deal with programmers we have to anticipate scrutiny.
  • Photogrammetry‚ a series of digital photographs in succession; parallel: we have mouse click and keystroke recording tools to document the repro of bugs.
  • Talked about how a boyfriend/girlfriend got into a fight and then violence happened; parallel: we develop user stories and scenarios to test for bug pathologies in software.
  • “We can’t say this is what happened, but we can give a logical range of possibilities”; parallel: we’re not always sure what the fault is, but we can suggest possibilities.
  • Projectiles go through glass and leave different signatures; parallel: same is true for bugs… programs leave different signatures on how they use memory or install files.
  • “We have to do presumptive tests sometimes (like the bullet through rubber)”; parallel: we also have to check our basic perceptions to make sure that a bug is really what we think it is.
  • “We take elimination fingerprints to rule out different suspects”; parallel: we do follow-up tests or peripheral tests to rule out other causes.
  • “Keep an open mind… don’t make your evidence fit your theory”; parallel: be mindful of your biases… don’t be fooled into thinking that this is a bug you’ve seen before.
  • “There is a high cost for processing evidence, homicides get priority”; parallel: there is a cost to doing tests… high risk features that lead to crash, hang or data loss get priority.
  • “Temporal evidence: fingerprints can last a long time… might have been there from months before”; parallel: this is the Primacy Bias… a bug might have started weeks ago and shown itself now.
  • Harris vs United States, 1947: Only human failure to find it, study and understand it, can diminish its value; parallel: exactly the same for software testing.
  • “Two heads are better than one; parallel: paired testing, “fresh eyes find bugs.”
  • Staff: Team Lead, Sketch preparer, Photographer, Recorder, Specialists; parallel: Team Lead, Recording Tools, Subject Matter Experts.
Scroll to top
Bitnami