Quardev Quarterly Q3, 2008
In this issue:
Regardless of the current state of the economy, everyone is looking for value - but value can't always quantified; it comes in many forms:
- a trusted partner who you can call on anytime to help you work through an issue or project.
- a testing resource who can give you just enough testing to ensure you reach the goals of your release.
- a writer who understands what you are trying to communicate and can ensure your readers understand too.
This month we've chosen to focus on value. Where it's found and how to quantify it in your business decisions. As a leader in the technical services industry we are always looking for the best way to package and deliver our services to our clients.
Thank you for reading the Quardev Quarterly. We hope that some of the viewpoints we've been able to share in this issue will give you more insight to some of the ways we operate and may help you in your daily work life.
Enjoy the newsletter with our compliments and please contact us with questions, comments, or article ideas.
All our best and thank you for reading;
-The Quardev Crew!
Why SDET?
By Jon Bach, Manager for Corporate Intellect, Quardev, Inc.
About 10 years ago, it became outmoded to hire former data processors or writers or teachers or displaced home-makers to be testers, despite the imagination and enthusiasm (and even track record) of these kinds of candidates. It seemed that the people who had been finding good bugs for years were being forced into being programmers under the banner of "working smarter, not harder." Good, inventive exploratory test engineers were told that they must learn a programming language and write tools to automate their testing - to take a more white-box approach - or else lose their job.
For these past ten years, the major trend in hiring software test engineers has been the search for Software Design Engineers in Test (SDET).
I think I know why.
Why? The mindset was that because the world is an increasingly Internet-driven world, testing needed to be performed faster and more thoroughly.
Often, this was the mandate facing many companies in 1998, and it persists today. But I have an idea of why this happened.
There used to be testers known as "tool guys." Few in number and good at their job, they were people in Test who focused on giving test case engineers and test executioners ways to test more of the product, or use a different lens to see the product, or to extract more information from the product. Usually small teams of two or three, these tool guys walked the border between Test and Development departments, literally. They went to Dev meetings, they went to Test meetings. Their value was that they were humble, hard-working, smart people - but unlike so-called "functional testers," they could whip up small programs in a day or two. They served testers who had specific needs.
For example, as a functional tester with a stack of test cases on my desk to run or a stack of cool exploratory test charters to run, I could go to my tool guy and ask things like:
- "Can you give me a hook into this feature - maybe a log file or something so I can see what's going on?"
- "I need to generate 50 different kinds of data to put into this input field. Can you write me a program to do that while I run these tests?"
- "These conditions need to be simulated as states, can you build me something to put the software in X state whenever I want so I can run these sets of tests?"
The tool guy would ask a few questions to refine the mission, they'd take notes, and they'd say "Let me see what I can do."
They'd come back with something that usually worked, or had a few bugs that could be fixed in a few hours. In fact, these tool guys were so good at it, word spread about their value. The mentality of Program Managers seemed to be, "Why keep a functional tester if these tool guys are more skilled?"
Then, as if overnight, it became, "... if we had all of our functional testers as programmers, we'd have a better project!"
And that's the way it seems to be to this day. These days, most job descriptions for testing have a required entrance criteria for programming skill and they all seem to carry the same title: SDET.
The importance of imagination and thinking like a user seems to have vanished in favor of people who can write code that tests code.
Former Political Science majors, Journalism majors, Chemistry majors, Communication majors now have no place - unless they picked up some coding skill along the way. I'm worried about the mentality of interviewers who place prime value on programming skill, and not on thinking and reasoning skill.
It's often the case where the best bugs found on projects are not found with automated tests. Even automated regression does not always catch problems. In 2000, I was hired by a major printer manufacturer to see why so many of their good bugs came from exploration, not automation. They had thousands of automated regression test cases and although they were finding some problems, the best bugs came from when staff was allowed to "play around" with the software.
Many articles have been written about the dangers of having an unyielding mindset about test automation, and they have one thing in common. It takes a savvy human to know if automation is giving your project the value it needs.
For example, automation may be best for the following scenarios:
- tedious tasks that a human does not have time for
- repetitive tasks that a human would be driven crazy from
- tasks that require fast computing power, such that no human can come up with the results in the time available
As for the value of an SDET, there's no question. Testers hired to have a programming focus can aid a test project. It's an approach to using computers for what they're designed for - speed, computing power, repetition. But an SDET is a specific role, not a solution. I see SDETs these days as those "tool guys" who have been promoted in executive management. They may like being "legitimized" in this way, but it's not their style.
A true SDET serves the Test Team. They question their value, they look for ways to help all kinds of existing test techniques be more efficient.
But how can that begin to pierce the management attitude that programmers are better testers?
Traditionally, testers have been good at finding and programmers have been good at fixing. Those activities are two different mentalities that can't exist in the same space. It's because of confirmation bias. When you're finding problems, you want the system to fail so you will be more likely to notice failures. When you're fixing problems, you want the system to work, so you will be more inclined to see it working.
So if all testers should be programmers, there may be a risk that they will only write programs that confirm the program works, other than programs to aid in finding failure - seeing how many spectacular ways the program crashes.
How about a compromise?
Maybe the most value a test professional brings is in knowing when a programmatic solution is better than a non-programmatic one. Maybe the most value they bring is those little tools that make the human testers day more efficient. Maybe it's in the way they straddle that line between the Programming and Test departments, neither in one camp nor another, but living both cultures simultaneously.
If you find yourself as a manager in this position, thinking that an automated solution is better, think about this question: "What did you see or hear that makes you believe that in this case, the solution should come from someone who knows how to program?" It could be that you're right - you need one of those "tools guys" to augment your existing strategy. But if you're thinking that programmers are just smarter and faster and better than those who do not know how to program, be mindful that the role of testing is to find failure, not confirm the product is working (absent of failure) as individual automated tests tend to do.
References:
http://googletesting.blogspot.com/2007/10/automating-tests-vs-test-automation.html
http://www.satisfice.com/articles/test_automation_snake_oil.pdf
http://blogs.msdn.com/steverowe/archive/2008/02/26/when-to-test-manually-and-when-to-automate.aspx
http://www.softwarequalitymethods.com/H-Papers.html#TestTest2
Testing Is Consulting
Jim Bullock, Rare Birds Enterprises, "Conscious Development"
"Why do you ship without fixing all the bugs?"
We had a captive Development VP to interrogate, so the questions at QA-SIG got interesting. Behind that tester's question was another: "Are defects found and not fixed wasted work?" Behind that: "What's my value if you do this?" Given the chance, he asked them.
Of course, simply observing how they are used shows that known, shipped bugs have many kinds of value:
- Put them in release notes, to avoid them.
- Project the next release better because you know more.
- Get a sense of your risk. How broken is it?
- Notice things you might want to improve in how you do development.
- etc.
We use defect information these ways and more. Defects influence all sorts of things, even when they are not fixed. When the VP answered: "Yes, I want to know." his demeanor was worth more than hours of argument. This guy takes chances all the time. Information drives down his risk. To him, knowing about unfixed defects has great value. He looked hungry when he said it.
That tester was in a consulting relationship and didn't know it. He thought his only contribution was when bad software behavior got fixed. He didn't see that his value came when the information he created influenced development, the VP and the business. Fixed bugs are just one kind of influence, from one kind of information.
"Consulting is influencing others at their request." - Jerry Weinberg
By Weinberg's definition, testing is consulting - it influences go-to-market decisions, development processes and tasks. It turns out the VP-guy was asking for these kinds of influence. Along with his development team knowing about brokens to be fixed, he also wants anything that might reduce the unknowns in his world.
Testing's consulting relationship can get out of whack. Testers sometimes try to force influence they don't have, like a veto on shipping. You only have a veto when your clients give it to you. Some testers insist on techniques - the One True Thing to Do(tm). Well, only if someone wants the information it produces, only if information produced this way will influence someone else. Sometimes testers get bent when their defects aren't fixed, but who said that always makes sense? Maybe, like the tester in this story, your bugs have other influence.
Testers miss value they could provide by missing both influences and the need to be asked. VP-guy can live with some product errors. He can't live with a product that gets rejected by the customer. Along with defects to fix, he's asking to be influenced about whether he should ship or not. He's not asking for testing to take over deciding when to ship. He's asking for information to help him decide.
If you look for influence being asked of you as a tester, you can see many chances to help. If you look for chances to influence with your work, you can put more offers out there:
- Here's what our testing can produce. Do you want some of that?
- Here's what our testing is producing. Do you want more of that? Should our testers stop?
- What do you want to know about your system? What more might we do?
- As we design tests, we think of risks. Would you like to hear about those?
Sometimes, the relationship between testers and their clients is broken beyond misaligned influence. I've seen "testing theater" - lots of visible "testing" effort while the information changes our minds about exactly nothing. Sometimes, testing is a dump for a whole project's problems. "We screwed up, so we'll make you own product quality. BTW, because we're behind, you have a compressed testing schedule." You may agree to take those jobs, but they are not testing. You are being paid for something else. By noticing the relationship, at least you know the game.
Bad relationships won't be solved by testing technology or techniques. It is, however, amazing what addressing the relationship directly will do. Try saying: "I can't do that." Or, you could hear "It's all your problem." as an offer vs. blame, and take them up on it: "I'm hearing that the product's quality is my job. Would you like to hear what we're going to do about that?" Most of the time, you will not get to finish that last bit before they tell you how in charge they are. Doing so, they've straightened out the relationship for you. (Shhhh. Don't tell them.) Now, you can make a sensible offer of information that you can provide to influence others if they want: "Well, since you are responsible for the product's quality, here are some things testing can find out for you. Do you want some of that?"
I was puzzled thinking about the fragment of my history that prompted Jon Bach to ask for this article. Years ago some colleagues and I created an automated testing practice. Yet, there wasn't much to do with testing techniques in what I did. I mostly helped people straighten out the relationships between testing and their customers, sponsors, and peers. Fix that and good things follow. It came down to being explicit with: "What influence would you like from testing?"
I was mostly a consultant about the relationship between testers and their customers, starting with the idea that: "testing is consulting."
Effective Technical Communication Adds Value to Business
Andrew Ho, Technical Writer, Quardev, Inc.
The service a technical writer provides can add significant value to a business and enhance its bottom line. Effective technical communication helps a business reduce operating costs, increase sales, manage risk, improve productivity, and clarify its vision for the future.
Technical writers help specialists sort out "brain dumps." Technical writers edit a document not just for grammar and syntax, but also for logic, organization, and presentation. For writers, editing is fundamentally about piecing together ideas in a logical sequence easily understood by an intended audience. Good technical writers will ask technical specialists meaningful questions about evidence and causal relations between distinct ideas, such as how someone benefits from a particular application. What may be obvious to the specialist may not be obvious to potential investors and customers.
Cost of Ineffective Technical Communication
One way to assess the value of a technical writer is to calculate the cost of ineffective technical communication. Organizations with stellar products and services that try to reduce costs by producing documentation at minimal expense can find that costs incurred from ineffective documentation can surpass the amount spent on ensuring high quality technical communication. For instance, imagine a great software program that's accompanied with a confusing manual. With high expectations, customers purchase the program. Now imagine the results:
- Frustrated customer base
- Increased costs to handle customer questions regarding installation and use of product
- Lower return on investment than customers expect
- Fewer customer referrals than the company expects
Organizations don't purposely give their customers confusing manuals. The problem is that some organizations don't realize they're sending incoherent explanations and instructions to their customers. After all, the entire development team understands the manual just fine!
Regardless of the writing and editing skills of the developer(s), it's often necessary to have someone detached from the specifications and the code to provide an unbiased perspective on the clarity and completeness of technical documentation. An experienced technical writer can give that perspective. And with that perspective, you can add value to your products and services by making them easier for your customers to understand and use.
Cost of Not Having Technical Documentation
A surprising number of organizations lack proper internal technical documentation. This isn't a minor detail - the repercussions can be severe. Even if your business is a software gaming company that doesn't have to worry about the FDA shutting it down for not having adequate documentation of internal procedures, organizations without coherent, version-controlled records of workflow processes and standard operating procedures are at risk if an employee who is the only person familiar with a vital part of your business disappears. Why allow a company to be crippled by one person's absence?
Well written workflow processes and standard operating procedures are also training manuals that help ensure the consistency of products and services, without limiting employee creativity. Companies that don't want to constrain their developers and consultants with too many rules but want to guarantee to their customers the quality of their products and services can seek a technical writer to find the right words that strike the right balance.
Having a technical writer manage and optimize internal technical documentation can reduce business risk and enhance internal quality control processes. A technical writer can work with technical specialists to write, edit, organize, and maintain documentation that is coherent and controlled for audit.
Add Value with Effective Technical Communication
Technical writers can add value to your ideas. The process of discussing an idea with a technical writer can lead to better developed and more refined ideas. Thinking about writing is thinking about ideas and how to best organize and package them. Technical specialists willing to set aside a little time to discuss a technical writer's edits of a proposal, white paper, specifications document, or even a memo often discover that doing so not only prepares them to speak more convincingly about their ideas and vision, it also generates and refines new ideas. The authors of Guide to Effective Military Writing state that "Badly organized writing is evidence of a badly organized mind. No one's going to accept the conclusions of a badly organized mind." Likewise, organized writing reflects an organized and logical mind.
Tech Talk: Testing Tools You Might Find Valuable
Contributors: Jon Bach, Ross Mortimer, Richard Bustamante, and Jacob Stevens, Quardev, Inc.
Below are some of the tools our testing staff finds most valuable:
Spector:
Spector is a commercially available recording tool meant to spy on your friends and relatives as they use their computer, but it's wonderful for testing to capture mouseclicks, screenshots and keystrokes.
Retail: 100.00
Link: http://www.spectorsoft.com/
BlueBerry Test Assistant:
Same as Spector, but records audio as well and includes its own bundled player so that people on the project can easily play your recordings if they are attached to your bug reports.
Retail: 225.00
Link: http://www.bbsoftware.co.uk/BBTestAssistant.aspx
Microsoft OneNote:
Fantastic for note-taking during testing, as you can also easily record your voice notes. The best feature is that the note-taking and audio are synchronized and searchable - the epitome of recording testing for archiving.
Retail: 100.00
Link: (60-day free trial) - http://office.microsoft.com/en-us/onenote/default.aspx
PerlClip:
Any time you're confronted with a text field, you've got several opportunities to stress it. PerlClip allows you to easily create a wide variety of strings of text to paste, including a long series of repeated strings, counter strings, strings with all character codes, even a string created from exporting the text from any document you specify.
Retail: Free
Link: http://www.satisfice.com/tools.shtml
InCtrl5:
This is a tool for tracking changes to your system after doing anything - usually installaing something. It lets you take a snapshot of your current system (files, registry, memory) at one point in time and compare that snapshot to one taken at a later time. InCtrl tells what was added, changed, or deleted, which is ideal for tracking machine states.
Retail: Free
Link: http://www.pcmag.com/article2/0,4149,25126,00.asp
Virtual PC:
This is a way to run several fully functional operating systems (Windows XP, Windows 2000, Windows Millenium) on one machine. No more building boxes for every OS or platform you need tested. If the hardware doesn't matter in your testing, Virtual PC is the way to go. You can save several different machine states as separate files that can be attached to bug reports.
Retail: Free
Link: http://www.microsoft.com/windows/products/winfamily/virtualpc/default.mspx
Sysinternals:
This suite of utilities is a must for testers. Includes file and disk monitoring, process monitoring, network utilities, system information reports, string searching, commands for rebooting remotely, and tons of other cool permutations of whatever you might need to do for file and system analysis.
Retail: Free
Link: http://technet.microsoft.com/en-us/sysinternals/default.aspx
VMWare Workstation:
VMWare hosts and operates virtual machines and allows the user to take snapshots (saved states) of those machines and store them in a library for easy access, which can be invaluable.
Retail: 189.00
Link: http://www.vmware.com/
Wireshark:
Wireshark monitors network traffic going across a selected network adapter, and provides the user with real time data about what's happening (as opposed to Ethereal which aggregates the results at the end). This can be really helpful in testing pings and other traffic.
Retail: Free
Calendar of Events - Quarter 3 2008
September 10, 2008
QASIG
September 29 - October 2, 2008
STARWEST 2008
October 13-15, 2008
PNSQC
For more information about any of these events please see our Web site: www.Quardev.com/events
Quardev is looking for Great People
We are always looking for great people to join our team.
At Quardev you will work hard, you will enjoy a great working environment and benefits, and you will be building a solid, interesting, and flexible career where you can learn and grow.
If a career with Quardev sounds interesting, or sounds like it may be a fit for someone you know, check out our Careers page or contact us today via Quardev contact page.
We'd love to meet you!