Losing is not how you lose. The way you lose is by winning so easily that you don’t learn things. In order to win, you have to lose a lot. Perhaps you have to lose the most.
– Northernlion

One thing that video games ever rarely capture is acknowledging when the player has won. Sure there are win states and lose states, but those ever rarely map to a players own definition of winning and losing. This spans beyond the scope of video games and into sports and any other form of competition as well.

That’s because win-conditions are easily created and agreed upon through quantitative measures (kick the most goals, finish the fastest etc.), but a qualitative account of winning and losing is highly subjective and varies depending on the day and probably the mood someone is in at the time. Sometimes, all it takes to “win” in any activity is to have a good time and unwind, other times it might be to deepen your understanding of the activity you are doing and your relation to it. Sometimes its about aiming for something so unachievable that all you can really do if fail. But you learn from that failure and in doing so, get a little closer to truly winning. I get a lot of that in testing.

I think the path forward in stepping into the testing industry is to face failure with a positive mindset. Testers seek out failure, we are also challenged by failure. We need to learn from failure. But the only way we fail, the only way we win, is by being inquisitive and daring to shoot our thoughts out into public space and watch to see how they either sink or swim.

The one certainty we testers are afforded is that we will always receive an answer at some point to the question “Did we catch all the bugs that mattered?” Someone is going to spot that bug. Log that ticket. And that is going to be an opportunity to learn from and understand if the bug(s) that were missed belong to a larger family of bugs that had not been accounted for. An opportunity to assess where our weaknesses lie and how we might evolve them into strengths.

Testers truly fail when:

• Confronted with overwhelming complexity they shy away and say its too hard
• Important bugs evade them and rather than adapting their approach, choose to ignore those bugs
• They unquestioningly take the path of least resistance
• They fail to define their craft for themselves

Insanity is doing the exact… same fucking thing… over and over again expecting… shit to change…
– Vaas, Far Cry 3

If there is one thing we can say about Vaas, he was definitely insane. After 3 attempts of trying to kill Jason in different wacky styles, not once did he confirm Jason’s death. Never checked his pulse or anything else like that. He failed his own definition really, which is kind of ironic. I think the definition of insanity is a little more nuanced than this, but one important principle it does highlight is the importance of variability. You aren’t going to learn a great deal of anything if you never vary your approach. Certainly, environmental variables might play a factor in providing a different outcome, but I have typically found approaching a problem from different angles elucidates a solution much faster.

The key is being comfortable with failure and understanding the likelihood of it. Learning to fail gracefully and recognizing when you have failed. When you are well acquainted with failure, you start to win more. Keeping this in mind helps me interface with some of the testing heavyweights I have come to respect and helps build up enough confidence to challenge them on their assertions and ideas. More often than not, I’ll fail, but I’ll learn a helluva lot in the process.

I have told you before, there’s no escaping the nature of the universe. It is that nature that has again brought you to me. Where some see coincidence, I see consequence. Where others see chance, I see cost.
– The Merovingian, The Matrix Revolutions

The Grey Heuristic – Any amount of change, removal or addition of components within a system induces an opportunity cost.

The Grey Heuristic is essentially repurposing the economic term of ‘Opportunity Cost’ for the needs of testing. The New Oxford American Dictionary defines Opportunity Cost as “the loss of potential gain from other alternatives when one alternative is chosen”. I am skeptical as to the accuracy of this definition.

Any pool of alternatives we can select from may not be limited to gains, but also losses. And the form of those “Gains” or “Losses” may not be linear, they could be exponentially better or worse in relation to other alternatives when applied in a specific context. What alternatives confer gains or losses will inevitably change over time. The fundamental goal of this heuristic is to help me maintain a tester mindset amongst people who have unwavering faith in the stable evolutions of their technology.

The Merovingian alludes to this in his speech. Testers work in an industry that seizes upon different sorts of chance to provide technological solutions, but that chance is always associated with some sort of cost. We must assess what that cost is and if its a price worth paying for.

I wouldn’t categorise this heuristic as one that finds bugs, what it does do is provide direction for thoughts when learning about a product. To me, this then represents a control-type heuristic; something that applies a particular scope to my thoughts to help concentrate my investigative efforts in a certain way.

There’s more to this heuristic than I can currently fathom at this stage, but I find it profound when applying this to the technologies we have come to rely upon. I’ll be doing more thinking around this and can see myself revisiting this post in the future.

Neo: I just have never…
Rama-Kandra: …heard a program speak of love?
Neo: It’s a… human emotion.
Rama-Kandra: No, it is a word. What matters is the connection the word implies. I see that you are in love. Can you tell me what you would give to hold on to that connection?
Conversation in Mobil Ave, The Matrix Revolutions

One of the activities I recently took upon myself was to consider for myself what my definition of ‘testing’ was. You hear definitions from so many other people about what testing is, but you still have to interpret that definition for yourself. Without stopping to think whether you have truly grasped what someone else’s definition is you may find yourself ill-equipped to defend that definition. “Do I understand it as they say it?” I would find the answer to that question hard to believe if the answer was “Yes”. Just as what Rama-Kandra implies in The Matrix: Revolutions, we interpret all the information we come across, its what makes us human and unique from one another, its why the role of software testing exists today.

So lets get on with it, here is my definition of software testing:

Testing is a systematic approach to systematically learning about something within a given context.

A systematic approach to – The first thing you would have noticed is the strikethrough text. This represents my first definition, I was unhappy with labeling learning as a ‘systematic approach’. It doesn’t make sense to me, it implies that there are other approaches to learning. But, I think learning is fundamentally systematic. When I say systematic, I am happy to use the dictionary’s meaning of “done or acting according to a fixed plan or system; methodical” The only difference is whether the systematic aspect is tacit or explicit.

We might question if children are ‘methodical’ when it comes to learning. But simply asking the question “How do children learn?” already implies that there is some methodology at play that can be discerned. Common answers found online state that children learn through observing, playing and experimenting. So the systematic aspect of their learning would be the processing of external stimuli to arrive at a decision.

systematically – Any astute thinker would then be asking why I have bothered to keep “systematically” a part of this definition at all. If it is true that all learning is systematic, then its a redundant phrase. Like ‘a four-cornered rectangle’, why should I bother describing an attribute of something it innately has?

It’s something I still occasionally wrangle with, but take stock of what most tester’s tout as an important attribute: being a “quick learner”. I think that is a core attribute for a tester to possess. But it’s importance is rivaled by its vagueness. A tester that says they are a quick learner is like a chef that says they are a good cook. I’m inclined to ask for more information – what empowers your learning? What makes you quick? Quick in relation to what? I think when we say we are quick learners, what we are really trying to say is that we possess refined and flexible learning styles that are easily applicable to certain types of applications under test.

I leave ‘systematically’ in the description because the systems of learning testers employ should be orders above anyone that does not actively study testing. I want to draw attention to the fact that our systems of learning are exceptional and meaningful.

learning about – Yes, our jobs are to find bugs and report the ones that matter to the people that matter so they can make informed decisions, so why not instead have “a collection of bug finding activities” instead of “learning about something”? Because I can’t bug find without first learning about the product I am testing.

Whenever I start testing, I don’t know where the bugs are, I can posit where they might be and make an argument for certain plans and strategies based on my presumptions about what the product is meant to do and what it is meant to achieve, but until I start experiencing the product¹, I am blind to bugs. Therefore, the activity I am doing first and foremost when I am testing something is learning, and I make it my job to learn as fast as possible and apply my learning skills in such a way that primes me to encounter and recognize that I have encountered bugs that matter.

something – Testing is not limited to just software and hardware. It is applicable in various facets of our lives. Try testing the way you walk, or the ‘best’ way to sleep. You might scoff at that, but it is my current hobby to learn about lucid dreaming and part of that is founded in understanding our sleeping patterns. Look up WILD and WBTB methods for naturally inducing lucid dreams. But I digress, it just so happens that software/hardware is where testing is needed the most, it is not to say it is the only place that testing is used.

within a given context – I’m still 50/50 on this particular phrase. We are back to tackling our four-cornered rectangle. Learning is always encapsulated within some sort of context, so why mention it? I don’t think I have a full answer at this point, but I feel the need to include that phrase anyway. Maybe to make clear to others that do not study testing, that context is a key factor that directs what we learn and how. At least it serves as a talking point should anyone inquire.

1. Experience of the product can begin before the product is even built through conference with stakeholders, developers, and end users, testing can begin once someone has formulated and shared the ‘idea’ of the product.

---

That is my current definition for testing. I’d call it my second definition as well, I would not be surprised that I revisit and amend the definition several times over the years to come. What I like about this definition is that it is a definition that makes room for good testing and bad testing. I don’t think a definition that constricts the view of testing to just good testing could be called a full and complete picture of testing… whether that’s meaningful or not is a discussion for another day.