The difference between moderated and unmoderated tests is that in the case of moderated tests you have to be present (on site, via Skype, etc.) to guide the participants through the tests, whereas with unmoderated tests you don’t need to be present, because the participants conduct their testing independently and, so to speak, moderate themselves.
There are pros and cons for both of them, so we’ll briefly introduce you to both moderated and unmoderated user tests, before finally deciding what kind of user testing we would do if we were you.
Moderated User Tests
Moderated user tests could be described in one sentence like this: You ask somebody to use a product (a website, an app, a can opener, whatever) and think out aloud while you watch them and listen carefully to what’ s crossing their mind.
What’s most important in these moderated user tests, according to our experience, is to avoid creating an uncomfortable exam situation, because if it happens, your test participants will be afraid of doing something wrong: They won’t say everything that crosses their mind, so that you won’t actually be able to understand their behaviour anymore. However, since the mere mention of the word “test” is enough to remind people of their school days and make them nervous, we’ve become used to having to assure each participant right at the beginning of the test that it’s impossible for them to do anything wrong, because it’s not them we’re testing, but the product.
But to say that we’re testing the product and not our test participants is only the beginning. The next thing we need to do is clarify that our ultimate goal is to discover as many problems as possible, and certainly not to have the test participant pass the test. Without saying this, it may happen that people avoid giving negative feedback because they don’t want to hurt your feelings. We solve this problem by addressing it directly and by encouraging our test participants to be as critical as possible, because this is the only way we can actually learn something.
After you tell your test participants that there’s no way they can do anything wrong and to be as critical as possible, the actual tests can start. Just introduce the product to be tested and explain briefly what this product is intended for. Correct? No. In fact, you’re doing exactly the opposite. Instead of explaining your product, you should ask your test participants to give you a quick explanation of what they think the product is for.
For example we tell our test participants:
Please take a brief look at this page and say out aloud what you think. Please don’t click anything yet and answer the following questions: What’s the first thing you notice? What could this website be about? Who is this page for? What products/services are offered on this page?
The beautiful thing about this method is that it saves you having to explain your product at the beginning of the test.
In fact, you should never do this because you’re missing the chance to see if anybody outside your team actually understands your product. What is less beautiful about this method, however, is that you’ll probably be shocked by how confusing your product is for most people, that is, if you haven’t conducted any user tests yet. Apart from this, an excellent start for a user test is to ask your test participants to explain your product in their own words right from the beginning. Once they’ve done this, which usually takes no more than two to three minutes, you can present the test scenario to them and start the actual user test.
Of course, it won’t be always easy for you to watch people fail to understand your product, and you’ll feel the urge to explain it to them. But we insist that you shouldn’t do this; just watch what your test participants do and what exactly they think/say while doing it, because even though it hurts, the insights from these user tests are incredibly valuable when it comes to making your product simpler, clearer and easier to understand. The motto is therefore to just shut up and limit yourself to occasional nods and make appreciative utterances such as “mmm”, “ahh”, or “okay”. And if a test participant asks how this or that works, you shouldn’t even try to explain your solution – just ask how they imagine how this or that could work. In addition, you can point out that you can only answer questions after the test, because you want to find out how understandable or self-explanatory your product is.
So, as you can see, moderating these user tests is not that easy, and it will happen to you over and over again that you are “helping” your participants without actually noticing it by giving them instructions on how to use your product. This risk is particularly significant when testing your own products, without having internalised that the goal is to discover as many problems as possible and not to pass the tests. However, this isn’t a problem that can’t be overcome by always remembering that user testing is about discovering problems. It’s easier, of course, to test other people’s products, because your self-esteem in no way depends on passing the tests, but solely on finding as many problems as possible.
Another important part of these moderated user tests is documenting or recording the individual tests. Ideally, you should record each test on video to watch it again later, but of course, also to show the tests to your team. In the case of a digital product, you shoot the screen or use software to record the screen, and most importantly, record the voice of the test participants to capture their thoughts during the tests.
In our opinion, the best way to conduct moderated user tests is explained in chapters 7 and 8 of Steve Krug’s book, Rocket Surgery Made Easy. Steve Krug explains the whole process and provides a number of useful checklists to make sure that you make all the necessary arrangements to successfully run moderated user tests. These checklists cover several pages, which should make it clear how much there really is to consider if you want to conduct a moderated user test. And since it would be pointless to try to describe the whole process better than Steve Krug (because his description is actually sensationally simple, clear, and understandable and it would be a plagiarism to repeat his process here), we simply advise you to read this absolutely worthwhile and extremely useful book.
Advantages and Disadvantages of Moderated User Tests
You can ask questions during a test to better understand the test participant’s behaviour.
You can improvise during a test and, for example, give a test participant an additional task that has only arisen from the test.
– Planning, organising and executing individual tests is extremely time-consuming and inflexible, as you have to be present during each test to moderate it.
– Documenting test results (preferably in the form of video including audio recordings) requires preparation and technical equipment.
– Moderation errors will distort your test results (A classic one: false, positive test results because you disclosed the correct use of your product to your test participants without noticing it).
Unmoderated User Tests
Unmoderated user tests could be described in one sentence like this: You ask somebody to use a product (a website, an app, a can opener, whatever) and think out aloud while you watch them and listen carefully to what is crossing their mind.
If you’re an attentive reader, you’ll probably have noticed that this description of unmoderated user tests is exactly the same as the one we provided previously for moderated user tests. That’s because both moderated and unmoderated user tests are basically the same thing. The only difference lies in the way these user tests are carried out, because while in moderated user tests it’s essential that somebody who leads the participants through the tests is present during testing, in the case of unmoderated tests the participants themselves take over this task.
It only became possible to do unmoderated user testing when the first remote user testing platforms came onto the market about ten years ago. On the one hand, these platforms provide participants to product teams for their user tests and on the other hand, software solutions with which these participants can carry out their own tests and record them. The entire process, including payment to the test participants, is taken over by the platform, with the advantage that product teams don’t have to invest time in the recruitment of test participants, nor do they have to worry about organising and conducting individual tests.
Thus, instead of looking for test participants, planning and organising tests, and moderating every single test themselves, product teams have the opportunity to have this done by one of these platforms. We simply use Userbrain – our own remote user testing platform.
All we have to do is enter the URL of the website and the test scenario in Userbrain and choose how many test participants we would like to test with. Done! A few hours later, we receive the test videos (screen recordings including audio comments) and we are able to evaluate the individual tests.
Advantages and Disadvantages of Unmoderated User Tests
Test participants “moderate” themselves.
Test participants can test anytime, anywhere, on their own devices and in an environment familiar to them.
You save time because you don’t have to plan and organise the individual tests.
You enjoy flexibility and don’t have to be present while the tests are being conducted; you simply watch the test videos when you have the time to do it.
– There is no way for you to intervene during the test if something goes wrong.
– You can ask questions during a test to better understand a test participant’s behaviour.
Our Recommendation
In our opinion, user testing is all about one thing, i.e. actually doing it, preferably, of course, on a regular basis. Thus, it doesn’t really matter whether you decide to conduct moderated or unmoderated tests; as long as you do test, you’re doing everything right, so to speak.
The less experience you have with user testing, the sooner we advise you to conduct unmoderated tests. As mentioned earlier, this not only means much less preparation and effort in the implementation, but also that your test participants moderate themselves. And this isn’t only something practical, it also has the added benefit of not inadvertently telling test participants how to use your product during testing, and thus avoiding false positive results.
Another important point is the experience of the test participants that you can find on the previously mentioned platforms for unmoderated user tests. Most of these people have already participated in a variety of user tests, and while many product teams believe it would be better to test with inexperienced participants, our experience is that the more often a person has already participated in a user test, the more valuable and useful this feedback will usually be for you and your team. There is a simple reason for this: experienced test participants know exactly what is expected of them, i.e. to read the test scenario carefully, complete all tasks and think out aloud all the time along the way. And this thinking aloud is difficult for beginners, so that their testing is often less useful, because without understanding exactly what is going on in their minds, it’s usually impossible to understand their behaviour.
Finally, we have to make it clear that we are huge fans of moderated user tests and urge everyone to moderate their own tests. The reason for suggesting unmoderated tests to you is best summarised in the following sentence, with which we end this blog post: Unmoderated user tests are so simple and efficient that you will really be able to do them in practice.
https://www.userbrain.com/blog/moderated-vs-unmoderated-user-tests
若有收获，就点个赞吧
0 人点赞
