User testing best practices 2 / 2
User testing best practices for product managers or UX researchers. 5 users are enough for any tech. user testing. Involve all your important stakeholders into testing sessions as observers.
9. 11. 2020
In the first chapter of this user testing blog post, you could found my reasoning, why are live 30 min. video calls the best method of executing easy and effective user research and also what are the best internal and external sources for recruiting your testers. But now let’s go to the more controversial topic - What is the required number of testers - 10? 100? 1000?
How many users do I need to involve?
The first answer that comes to mind is “as many as possible!”... Well, it is not so straight forward. First of all it would be very costly and moreover, increasing number of testers will not bring such additional benefits as one would expect.
There are many studies that suggest +/- 5 testers as an optimal number for any tech. product user research. That’s right! Only 5 users are enough to uncover 85% of issues or user insights. Look at the famous graph below, made by UX research guru Jakob Nielsen
Let’s assume that there are 10 major value adding problems hidden in our product (or 10 important insights in the customer journey that we have to understand). Nielsen’s research was measuring how many of existing problems / insights we get uncovered while testing with increasing amount of testers.
Do we want to uncover all 100% of problems or insights? Ok, but then we need to test at least with 15 users. Recruiting 15 users can be pretty demanding and following execution, evaluation and processing of those tests will take at least few weeks.
But what will happen if we involve “only” 5 users? Well, we will collect 85% of existing insights, which is still a pretty high number right? It is not 100% but to uncover the maximum of insights we would need to do 3x more tests and thus spend 200% more resources. But the additional value would be only 17% more uncovered insights. Pretty clear inefficiency, huh?
Of course there are cases where you need to be perfectionist and get all 100% of problems - for example in life-dependent medical products or highly critical business services. But in standard situations and typical commercial products we should be totally fine with 85% discovery rate.
Actually the best approach from my experience is to be iterative and rather do a few smaller test rounds than one large overwhelming testing exercise. It is already a pretty tough PM job to process those 85% of insights - analyse them, specify them, prioritise them and implement them. So having more issues to work on, will only increase the product backlog and keep us longer away from testing those new improvements.
If we are able to tame our perfectionism and get along with insights from 4-5 testers, we can focus on less things, work in shorter cycles and repeat the testing more often with new versions of our continuously improving product.
But is 5 really enough?
There is one more concern related to the number of testers. To write it as a quote, it would sound something like this: “Ok, I am fine with uncovering only 85% of all insights, but how can I be sure that these insights are the most important ones and why should I act only upon 5 user voices?"
Well first of all we are doing a qualitative research here, not a quantitative one. So we are looking for insights and themes, not hard numbers or statistical significance. Our goal is to get in touch with real users in order to get better understanding where to focus our improvement efforts.
If we see that 3 out of our 5 testers struggled with some product area it is enough to claim that we should focus on that product part. It would be useless to hypothesise at this point whether maybe another 5 users would have completely different problems. Well, they wouldn't! It would be a huge statistical exception if we found different or more critical problems after testing with another group of users (from the same user segment)
However the best way to solve these kinds of concerns, is to involve all important decision makers into the product testing sessions as observers. Actually, me not succeeding to persuade all relevant stakeholders to join the user testing sessions were maybe one of my most crucial failures.
You can do your best to explain to your business stakeholders that users are struggling with some of our features and that we should rethink it completely. But if they join the testing sessions and directly see that repetitive user frustration, it gives them a completely different perspective.
If your stakeholders see after 5 testers that the majority of discovered issues are repeating, they will fully agree that there is no reason to continue with another users and that we should rather quickly go in front of a whiteboard to start working on those identified issues.
So if it is at least slightly possible, try to get all your important decision makers into the room with testers, so that they can experience it all together with you. But explain precisely what it means to be an observer. They really cannot interact with testers or jump into the testing process. There should be always only 1 person leading the live session and communicating with testers to avoid any confusion. We should make it as easy as possible for testers to behave as in a “normal” situation.
It is usually super hard for CEOs or other non-product people just to sit there and listen to users how they struggle with our product. There is always a strong urge to jump in and explain to testers how to actually use their beloved product in a desired manner. But we are not on a sales session, nor are we here to do a product demo. We are here to sit quietly, observe user behaviour and feel their pain. Maximum what we can do is to ask few questions to get better understanding of user intentions and needs.
Do I need to test with users if I know my product so well?
Very interesting is also the beginning of Nielsen's graph, where we see that zero testers will bring you zero insights. Duh!
I have heard so many times from various business people that they don’t need to spend time on user testing, because they know their customers and products very well and there are anyway so many things to build or fix even without user testing.
The problem is that these business people and industry experts are not typical users of their product. In this case their knowledge is actually their disadvantage and since they are too deep in the topic, they interact with the product in a very different way than a typical user would do. It is great to use their knowledge during a user testing outcomes analysis, but avoiding user testing at all and counting only on internal ideas is a very risky approach.
Already 1 tester can show you 25% of problems and get you enough food for thought. But involvement of the 2nd and 3rd tester will allow you to start seeing the patterns and main themes of users' struggles. And this is the main goal of any user testing. We are not trying to get from users some precise orders what to do with our product. We only need to collect insights about the areas where to focus and then internally dig deeper into those areas.
However all of this is just a theory. If I shall give you 1 advice only, I would recommend you to be flexible and adjust the number of testers in “real time” based on the collected outcomes. This means to stop testing the moment you start to feel that you gained enough insights and on the other hand increase the testers group if you still haven’t gotten the critical "aha moment" that you'd expected.
This can sound maybe a bit vague, but I can assure you that you will naturally feel those aha moments without any struggle.
Do you know this typical story of a product?
Somebody has a great new idea, writes down initial product plan & requirements, draws high fidelity designs and then quickly move forward to develop an MVP (which unfortunately often takes few months) ... and as time flies by, they are stuck in a vicious “builder’s cycle” where they have to quickly design new features because developers are waiting for them and on the other side you have to develop new features because they were planned in the initial product plan.
Errrr … typical highway to hell 🔥
Important piece of a puzzle missing in this story is - user testing! Any kind of user testing.
Before defining product requirements we should map the typical user journey at least with few potential users
Before finalising the designs we should confirm our proposed product flows through wireframes testing
And definitely before starting any development we should receive feedback from real people on our clickable prototypes
So what are the best practices regarding user testing and how to avoid common pitfalls? From my experience as a product manager, there were usually following 3 typical concerns connected to user testing in tech companies:
What kind of user testing method should we use?
Where should we recruit them from?
How many users do we need to recruit?
Since it is a pretty broad topic I will split this article into two parts. First two questions will be tackled in this blog post and the 3rd will follow in User testing best practices (2/2).
What kind of user testing should we do?
There are many testing methods, based on the phase of your product lifecycle and the goal of your testing. But for me the most valuable were always live moderated testing sessions (either onsite or online). During a live session you can directly discuss with users their behaviour and their intentions, go into necessary details and thus get a broader context of their actions.
It is usually also the most costly and time consuming approach, but I would always prefer to have a 30 minutes discussion with 2 users than receiving 20 survey responses or seeing 20 screen recordings.
The value of user testing is not so much in the hard facts or particular answers, but rather in understanding user's empathy with your product, motivation to use it and those “soft factors” that can be gathered only via direct discussions with real people.
You don’t need to have a full fledged UX lab with a mirror wall to conduct live user testing. It is totally enough to schedule a video call with each tester where you can ask them to share their screen in order to observe and record how they interact with your product, prototype or wireframe. And you keep it short! 30 minutes should be enough for a skilled product manager / UX researcher to test needed product parts and avoid so called “survey fatigue” when testers become bored or tired.
Moderated video calls also allow you to have unlimited internal observers with you in the room or in the call, so that they can see the user interaction with their own eyes (importance of which I will mention later - as my most crucial lesson learned).
But take into consideration that any data collection requires user consent, so ask your testers clearly at the beginning if they agree with being recorded, introduce all other internal participants in the call and to comply with the motto of GDPR - data minimisation - try to collect as little personal data as possible. The best approach here is not to connect your testing notes with any particular user. Just take it all as general insights and it doesn’t matter whether they came from user A or user B.
There is also an option called “coffee-shop testing” which means to go anywhere among people (like a coffee house, coworking space, conference, etc.) and ask them to do a 10 minutes product review with you. This approach is pretty cheap and easy, however I will generally not recommend it since your tester should be as close as possible to the potential final users of your product. So some random coffee-place people can skew your test results, since they don’t feel those user pains that your product is trying to solve.
And this leads us to the topic of “how to lead a user testing session?"
Our goal is to understand as much as possible the perspective of our users without navigating them or influencing them in any way. Actually the best approach is talk as little as possible and let them talk at least 80% of the whole session. This implies that testing moderators are good listeners but this holds true for product managers in general :)
The worst thing are of course self-confirming questions like “Look at this hidden dropdown menu here! This can be pretty useful for you, right?” It seems like an extreme case but I’ve honestly heard such a question so many times. At least try to ask open ended questions like “What is your opinion on this dropdown menu below” or ideally do not navigate users at all and wait until they find and comment those features by themselves. If they haven’t found that feature or didn’t say anything about it, probably it was not such an impactful feature as you assumed.
The only way I like to navigate user behaviour is to stop them before some action (like clicking on the “Continue” button in an app flow) and ask them “What do you expect to happen after you click on this button”. This usually provide great insight into users' expectations and their understanding of the flow.
To sum it up, keep it simple and take the whole testing session as a natural discussion between two people - which really doesn’t require any extraordinary conditions other than a zoom call.
However, spend a proper amount of time on selection of your user base. They should be highly relevant for your product and the main criterion is whether they struggle with the same issues as your product is targeting. Ideally define your user segmentation before the testing so that you have a clear profile of testers to approach.
How to recruit your testers?
We are lucky to live in this age, where recruiting of relevant testers is really not that hard. You have plenty of paid tools & services that can help you with the recruitment process based on your specific user segments and for a reasonable price.
But before you jump into these external means, try to check your internal sources. If you already possess a database of at least a few clients or leads, go on and approach primarily those. You would be surprised how many of them will be totally okay to have a 30 minutes video call with you - mainly if your product brings some value to them. And there are also multiple options how to motivate them - from a freebie access to your paid product, up to a basic Amazon gift voucher, which btw. works really well.
Another ways how to recruit testers from your current user base are for example
Using HotJar pop-up polls or other live in-app forms of targeting users directly in your product with a question whether they are willing to join your research (ideally after some value adding moment like successfully finishing your flow in the app)
Or using customer service communication to recruit your users after successfully solving their service case (e.g. at the end of a pleasant chat conversation, after you successfully upgraded an user to a higher subscription tier)
Well, in case that you don’t have any usable user base yet or you cannot approach them you can still count on one of the research recruiting tools like respondent.io, [userinterviews.com] (https://www.userinterviews.com) or [usertesting.com] (https://www.usertesting.com). You can nicely target there exactly what type of users you need to recruit - based on industry, demographics, or any other criteria that you would specify. From my experiences it costs on average $300-$400 to hire & successfully execute 5 highly relevant user testing sessions via these platforms. Which is definitely not so much money compared to the value it will get you.
And in the next chapter - User testing best practices 2/2 - I will try to explain why 5 users are totally enough for a user testing session and also how an ideal testing session should look like.