As my colleague walked the participant out of the room and back to reception, I sat reflecting on the interview. I felt a warm glow, it reminded me why I love doing what I do — to help solve people’s problems. During the last hour, as my colleague probed a willing participant with seemingly disconnected questions, I got a real sense that the product we were creating was going to ease her pain.
My colleague returned, looking slightly more drained than before the interview had began. No longer in facilitation mode, she was free to return to a more relaxed state. She sat back down and let out a deep breath. I didn’t get the sense that she was feeling the same warm glow I felt.
“I thought that didn’t go so well.” she confessed. No doubt I responded with a puzzled look. “I thought she was just telling us what we wanted to hear,” she continued, “I didn’t feel like it was going to solve any real need for her.” And with that my warm glow faded to an uncertain greyness.
It felt, as she recounted her experience, that we had almost been in different rooms. Had I not witnessed the same thing she had? And if not why not? What else was at play that could cloud my judgement?
The problem with the lean, empowered, we-do-our-own-research product team
I’ve designed and conducted many qualitative research studies over the last few years. I often work in small product teams of three or four people and advocate that all team members get involved in research in some way. When this happens, each function — design, product management and engineering — brings their unique insight to the process and everyone benefits from building a shared understanding within the team.
However, there’s one major problem with this lean approach of an empowered product team conducting their own research. The team are invested in the outcome. This is often overlooked but it explains why I had a different experience to my colleague in the testing session that day. For my colleague was a dedicated researcher and as much as she was a part of our team she was indifferent to the outcome. A designer who has spent the last three days sketching various ideas wants at least one of them to succeed. A developer who spent the last three weeks building a prototype wants it to work. A product manager who got stakeholders’ buy-in on a direction wants it to be proven.
Whenever a team member desires an outcome bias is present. That outcome can be as explicit as wanting a particular idea to succeed or can be vague in a sense of just wanting any positive outcome so the team can move forward. Sometimes research can provide no clear direction, but often we’re not open to this outcome because it’s costly. To say a team can be immune to bias is hopeful at best. To say that a team can be immune to bias whilst also being invested in the success of the product is myopic.
Some teams may believe they are above this, that bias is minimised due to their experience and methods of practice. This quote may give such teams pause.
“Recent evidence suggests that people tend to recognise (and even overestimate) the operation of bias in human judgment — except when that bias is their own.” —Emily Pronin, Perception and misperception of bias in human judgment
Understanding possible bias
As I experienced, bias can affect individuals to the point where two people can partake in the same research session but come away with contrasting experiences. A process known as selective perception provides a useful reference for understanding this phenomenon.
Selective perception is the tendency not to notice and more quickly forget stimuli that cause emotional discomfort and contradict our prior beliefs
It’s important to recognise that selective perception is an important human trait that helps us get through everyday life. It uses past experiences and beliefs to help filter out unnecessary information. Without it we would constantly react to every individual stimulus during the day, and become completely overwhelmed.
Selective perception is not a single bias, it is the sum of multiple cognitive biases. In a research setting selective perception can affect how a team conducts their research, how they experience the research session, their memory of the sessions, and the process of making sense of it afterwards.
When conducting a qualitative research session these biases may include:
- Confirmation bias: The tendency to search for, interpret, focus on and remember information in a way that confirms one’s preconceptions.
- Experimenter’s or expectation bias: The tendency for experimenters to believe and certify data that agree with their expectations for the outcome of an experiment, and to disbelieve, discard, or downgrade the corresponding weightings for data that appear to conflict with those expectations.
- Pro-innovation bias: The tendency to have an excessive optimism towards an invention or innovation’s usefulness throughout society, while often failing to identify its limitations and weaknesses.
During the period of synthesis or sense-making, teams are subject to further memory biases. Some of these may be:
- Fading affect bias: A bias in which the emotion associated with unpleasant memories fades more quickly than the emotion associated with positive events.
- Bizarreness effect: Bizarre material is better remembered than common material.
- Hindsight bias: The inclination to see past events as being more predictable than they actually were; also called the “I-knew-it-all-along” effect.
- Rosy retrospection: The remembering of the past as having been better than it really was.
The missing function
Working alongside a researcher brought my own bias to my attention. I’m now convinced that the best way to mitigate a team’s bias is with a dedicated researcher. That researcher should be someone who is not invested in the success of the product and is only acting to serve the team through testing assumptions and uncovering more problems.
Having this kind of person on a team is unnerving at first because their default mode is scepticism. In a team of optimists trying to push things forward the researcher provides a necessary friction. Their indifference to the product keeps them questioning whether it needs to exist at all, long after the rest of the team have begun to believe their own stories.