30. What’s the value of unmoderated testing?

Drew Freeman: Morning, Joe. How you doing this recording day?

Joe Marcantano: I am doing well, Drew. How about yourself?

Drew Freeman: I had such a hard time getting up this morning, so I told you before we started this call that, after we record these couple of episodes, I'm going back to bed, I'm taking a nap.

Joe Marcantano: Well, and I kind of figure with. I mean, folks don't know this, but we're recording this the weekend between or before the super bowl. So there's no football this weekend. So I kind of figured you'd have more of a lazy day plan.

Drew Freeman: That is accurate today is clean up the house day. Ca becausee I have been slacking on that.

Joe Marcantano: that's a good one. I am actually going toa do a little bit of the same because February is a heavy travel month for me, and this is one of the few weekends I'm in town this month.

Drew Freeman: All right, well, let's take advantage of it and we'll jump on into our first question. and this first question is, what is the value of unmoderated testing?

Joe Marcantano: This is a really cool question, and I'm glad we're talking about it, because I do feel like maybe not so much with researchers, but sometimes non researchers, folks who might be designers or whatever, might, end up relying on this a little too much, or maybe in the inappropriate times, whether that be for time constraints or just not knowing any better or whatever the case is. But I think it's really, a valuable tool as long as you know its limitations.

Drew Freeman: Exactly. It can be super, super powerful, but it can also be overly powerful and used in too many places. Everything starts to look like an unmoderated hammer.

Joe Marcantano: Exactly. why don't we start with just a very, very basic, like, what do we mean when we say unmoderated?

Drew Freeman: Hit us with it.

Joe Marcantano: So it kind of sounds intuitive, and it really is. It's really any sort of testing that's done without a person there to moderate. So, know, one could argue that quant testing fits into this if you're doing survey work. And certainly some forms of light survey work definitely fit into the unmoderated realm. But I think what most people think of as unmoderated is are things like, you know, if you're using one of the big platforms Right. Like user testing or whatever. And you're setting it up so that the participant does it on their own. Maybe it records their face and their screen and audio, and they kind of just talk through what they're doing without somebody there to poke and prod and ask questions.

Drew Freeman: Yeah, I would say the most common kind of testing style for unmoderated is typically your user testing, your usability testing, whatever you want to call it.

Joe Marcantano: Yeah. Which makes it really valuable, when it comes to, like, time for the researcher, because I don't think it takes any more time to set up than moderate it. In fact, I would argue it probably takes a tiny bit less. And it can run in the background while you, as the researcher, do other stuff and you can just kind of check in on it every day or so.

Drew Freeman: Yeah, that's one of the huge benefits of unmodated is that you can have, you know, tens or hundreds, even participants, all taking your study at the exact same time, which is obviously not possible if you need moderation for all of those people.

Joe Marcantano: Exactly.

Drew Freeman: So talk me through some of the other pros of unmoderated testing.

Joe Marcantano: So time is the big benefit, and that's the one we talked about. It's also pretty beneficial when it comes to your budget, actual dollars and cents. Usually the incentive is a little lower because the time commitment from the participants a little lower. It can also help you get a little bit more diverse participant base. And kind of the stereotypical person to think about or answer to think about here is like the single parent who can't reliably carve out an hour, two days, or three days ahead of time, but they might have 20 or 30 minutes suddenly available. That's the kind of participant that you're gonna be able to more easily recruit in an unmoderated study, as well as.

Drew Freeman: Folks who might not be available during the typical nine to five or even after work schedule that research typically happens in.

Joe Marcantano: Yeah, there are lots of researchers who, rightfully so, will adjust their working hours to fit their recruitment cohort. But there's also researchers who say, no, my hours are this, and this is what I'm gonna work. And I certainly understand that

00:05:00

Joe Marcantano: argument. And this gives them a chance to get some of those outside of the working hours participants.

Drew Freeman: And even when you are shifting your moderation hours, there's only so much you can shift. And we haven't even talked about international participants. And, you know, for me, in the US it would be almost impossible to find a good time for me to moderate a session with A participant in India, for example, with I think it's an eleven and a half hour time difference. So doing something like unmoderated testing can make it much easier to reach that audience if I need to.

Joe Marcantano: Yeah. I think we've talked before about how it's kind of best practice to have somebody from that culture or incredibly familiar with that culture to run sessions. So if you are, for example, running a cohort with, participants from India, it becomes a lot easier for you to find a contractor or an agency to review my unmoderated testing plan before I can let it run and then maybe have them give it a once over once you've done your analysis, rather than paying that contractor or that agency to run the entire test from you. So budget wise, it's a little easier there too.

Drew Freeman: O, and we talked about the time savings of running multiple participants at once, but you mentioned that these sessions are often less time for the participant. And beyond that even, it's possible that by the time when you launch an unmoderated session or an unmoderated test, you can have at least preliminary results, but often full quota results within hours of launching.

Joe Marcantano: Absolutely. I've definitely talked to folks who have said, especially when you're doing more gen pop studies, that, you know, an hour I might get my eight, ten people, which, you know, if you're looking to do very quick iterative testing, that's really beneficial.

Drew Freeman: Absolutely. So are there any other big pros that you want to talk about?

Joe Marcantano: The other thing that I would mention is that, you know, because of the time and money advantages, it does allow you to go a little higher in your sample size, which can give you a little bit higher degree of certainty. I would always weigh that against, you know, is that extra 2 or 3% of certainty really worth the money? And in some cases it certainly is, but it does, it does afford you that option.

Drew Freeman: Okay, so we've talked about unmoderated and all of the great things that it can do, but obviously no method is perfect. So talk to me about some of the limitations of unmoderated testing.

Joe Marcantano: The big one is you're not there. You can't ask why. So while unmoderated might be really good, for example, to say, can participants complete this user journey? You can certainly ask them, hey, did you understand it? But you may get a lot of, yeah, I understood it, or no, not really. I just got to the finish line. And you wouldn't get that really deep. Why?

Drew Freeman: Understanding that certainly is the biggest downside of unmoderated testing. another kind of downside that I like to call out for folks, especially developers or designers who might think that it is the hammer that they should be using, is that without a moderator, participants tend to take tests less seriously. They don't engage quite as fully. They don't maybe dive in as deep into the kind of imagination or role play aspect of a test as they would if there was a moderator directly observing them. And it can just be difficult with super early prototype stuff because often you need a human to be able to explain what is going on in the prototype or to help a participant recover from the almost inevitable errors that early prototypes have.

Joe Marcantano: Yeah, that first part you brought up is really, really important. When I'm, you know, either sitting across the table or next to somebody or I'm across a zoom and I see that person, there's kind of this implicit social contract. They are sitting there with me, they're talking with me. And you do get a, you know, quote unquote better effort in kind of a related vein. And a lot of this depends on the service you're using and the panel you're using. But you are gonna see higher instances of folks either kind of rushing through giving one word answers just to complete it and get the incentive. Cause it's not time based, it's task based. But you're also in my experiencen toa see a little bit higher degrees

00:10:00

Joe Marcantano: of fraud as well because you're not there. So the participant or the person who might be a professional survey taker is going toa know that they can kind of Bs their way through some answers and click submit and get that, you know, whatever that incentive is, that 20 bucks, 50 bucks, whatever.

Drew Freeman: Yeah, so I want to go back to the kind of unmoderated tends to lead to less engagement and less imagination and kind of give an example. So Joe, I'm going to set up a situation for you and then kind of have you talk me through what you might expect to see differently. If we were running this study moderated or unmoderated. So the study that we want to run is around, let's say it's a large e commerce like online shopping situation and we want to understand the process of how use, how consumers decide to research and decide to buy a particular product. What might you expect to see differently between a moderated session and an unmoderated session?

Joe Marcantano: The first thing I would expect to see differently is kind of the context that a participant is going to get. You know, if we're doing a moderated session and you set the context for me, you know, buying shoes online, whatever'you're there. And I can ask you, oh well, you know, do I know my size in this brand? And whatever. Right. Like I can ask the clarifying questions to get myself more fully kind of engrossed in the moment. Unmoderated. That doesn't exist. And so I kind of fill in the gaps on my own. And the big risk there, it's not that filling in the gaps is a bad thing. It's that I'm not going as the participant, I'm not going to verbalize how I fill them in. And so you, the researcher, have to make assumptions. Did this person assume that it's a brand they're familiar with and they know their shoe size? Did they guess? It just creates a little bit higher degree of uncertainty in the actual context of the scenario.

Drew Freeman: Yeah, for sure. And another thing that I was thinking of is that there's no like one right quote unquote, right way to research and comparison shop. So you know, you have to put, you have to put them into that mindset. And that right now anyway is really difficult to do without a human being there, like you mentioned. So you know, someone who is doing an unmoderated test, like you said, might just kind of rush through the steps and you pretend to do some comparison shopping. But actually comparison shopping is a very different experience. And we haven't even gotten into the kind of distinction between moderated lab testing where you're still acting and you're still, you know, playing a character versus a real life shop along where the participant actually needs to purchase this item and the incentive is maybe that the researcher will end up being the one paying for it, for example.

Joe Marcantano: Yeah, I think that. Not to the extent that quant can't answer why, but I think one of the big things to remember is that unmoderated testing can tell you what and to some degree can tell you why, but it's not going to tell you why. Like a moderated one on one interview will. And I think it's really well suited for, for the. Can folks do it? Do folks get it? It's not necessarily great or well suited for deeply understanding a problem space.

Drew Freeman: I think going back to a very early episode, the what is qual versus quant? I use the you know, narrow but deep or shallow but wide, metaphor. And I think that kind of applies here to a usability. An unmoderated usability test typically is going to be a lot just shallower than a moderated test can be.

Joe Marcantano: Agreed. And the advantage, you know, if somebody were listening to this and saying, well, if it doesn't have the advantages of moderated, why wouldn't I just do a quant? Right? Like, why wouldn't I just run a survey? The real advantage here is the speed. you will get faster results. They will not be as statistically reliable as a survey with an N of a thousand. Right. But you will get really good directional results quickly.

Drew Freeman: Another really big benefit of unmoderated testing over a quant survey is that a quant survey is all self reported data, whereas unmoderated testing. We can actually directly watch the behavior of a participant running through

00:15:00

Drew Freeman: our prototype or our website or what have you. And rather than basing our analysis solely on what the participant was saying, we can actually base our analysis on what the participant was doing. And as researchers, we should all know that behavior and doing should carry much more weight than saying, yeah.

Joe Marcantano: One of my favorite podcasts frequently says, self reported data is the least reliable form of data. And I struggle to come up with a form that might be less reliable than that. Doesn't mean it's all completely unreliable and you should ignore it, but, you know, give every piece of evidence its proper weight.

Drew Freeman: Okay, so we've talked around kind of some of the pros and some of the limitations using unmoderated testing. I know both of us are really big into. You need to let your research objective, your research question guide the method that you choose. So what are some types of research questions that unmodated is really good for?

Joe Marcantano: It's really good for. Can people do it? Is this immediately understandable? you know, if you have, going back to our shopping example, can people add the shoes to the cart and then add the promo code and do so easily and quickly? I can blast that test out to 20 people and have results in a couple of hours. And will it tell me deeply, do they get it? No, but it will tell me, can they make it work? Does it function? Does it get the job done? And maybe for the point I am in development, that's all I need. I just need to know directionally, are we on the right path?

Drew Freeman: Mm, A similar but related question that it can be good at answering not always is, how do people do it? Do they take path A, do they take path B? Do they go backwards and forwards a couple of times before they finally get to the finish line?

Joe Marcantano: Agreed'it's. Really good for how and Seeing the path you can get. Some of the. The tools out there are really great where you can set in your kind of a path that you want people to go through, but then it will tell you who succeeded and did the path, exactly who got to it in a more roundabout way and who failed. It'll give you click maps and heat maps. there are some really cool things that can. Or tools out there that can kind of help you ensure that you are on the right path and that participants can kind of intuitively get it done.

Drew Freeman: We've been talking mostly about user testing so far, but unmoderated can also encompass things like timed click testing or first click testing, where those kinds of heat maps and whatnot are even more valuable. What other types of questions is it.

Joe Marcantano: Good at answering the other types of questions? things like card sorts and tree tests, you can do those, in an unmoderated fashion, especially when you don't really need to be there when. And you can kind of set them up in such a way that you're giving the instruction to think aloud, to tell me why you're putting things in certain categories or why you're putting things under other things. It's really good for kind of understanding the way folks kind of categorize things and the way they're thinking about what data or what subjects fit with other subjects.

Drew Freeman: You do have to be a little bit careful there with those kinds of questions or objectives because you're starting to get close or maybe over that line of really needing to understand the why more deeply than unmoderated testing can often allow.

Joe Marcantano: Absolutely. Yeah.

Drew Freeman: Okay. So obviously answering why is in a deep way is the biggest limitation or the biggest question that you should be wary of trying to answer with unmoderated testing. But let's talk about how you might get some of the kind of shallower or more base level why understanding from an unmoderated test.

Joe Marcantano: Yeah, when you're writing your discussion guide and, I'm kind of using quotes when. When I say the word discussion.

Drew Freeman: well, and we can also use. We can also put quotes around the word guide because unlike, as we've talked about before, where a guide is not a script, in unmoderated testing, it is actually a script.

Joe Marcantano: Very true. And actually, let's rephrase it entirely. Let's say when you're writing your study plan, even more so than when you would in a discussion guide, you need to include those reminders to think out loud, to walk us through your thinking, walk us through why you're doing things you could not go wrong by putting that reminder in every single question.

00:20:00

Joe Marcantano: It's also even more important than in moderated, which it's already super important. But it's even more important to avoid things like double barreled questions because you're not there to provide that context or that cleanup of the question.

Drew Freeman: Yeah. When you're writing your study plan or your script for unmoderated testing, the same, the same best practices that apply to writing any discussion guide or any study plan, they all apply. But for unmoderated testing, you just need to ratchet that up, that scrutiny up to the next level. Because like you said, there's no way to clean it up and fix it after the fact.

Joe Marcantano: Yeah. I mean, let's be honest. You and I are senior enough and have been doing this long enough that we could probably run a moderated session with some notes on the back of a napkin and make it work. It wouldn't be the best, but we could make it work sometimes.

Drew Freeman: I mean, honestly, there are times when that's how I would like to run my sessions.

Joe Marcantano: Agreed. But that you can't really do that in an unmoderated. You have to carefully think it out. You have to be very intentional. It is much more meticulous when you're doing your planning. Then an unmoderated would be where, you know, I just put a note to myself in the margin. Be sure to ask why or be sure to probe.

Drew Freeman: If they say this in an unmoderated setting, you have to think very carefully about each eventuality and have contingencies built into your script for the likely ones.

Joe Marcantano: Which kind of goes to something we covered in an earlier episode. The importance of a soft launch.

Drew Freeman: Oh. It is a requirement for an unmoderated session to not only have a dry run with a Participant 0, that doesn't go into the, that doesn't count as part of the actual data set and the analysis. But to then after that, do a soft launch with real participants for 10%, maybe of, of the quota that you're looking for to make sure you that the questions you're asking are working for participants. They're being understood and they're being answered in ways that you expected them to be answered.

Joe Marcantano: Yeah. It's incredibly important, I think. Yeah. Like you said, it's required. It's not optional. you can sit there and think of every path, every option, every conceivable scenario, and people are people. Someone will find a way to interpret what you said differently and so having a soft launch is absolutely critical with moderated testing.

Drew Freeman: It is absolutely a best practice to run a dry run or do a soft launch. It is highly, highly recommended, but it is occasionally optional for unmoderated. Never optional, always required. Okay, so what are some other things that you have to pay more attention to or think about more carefully when you're writing a study plan for an unmoderated test?

Joe Marcantano: So one thing that you need to be really intentional about is the way you set up your screener and your whatever filters you're using to choose your participants for kind of the same reasons that we've talked about before. You're not there to ask that clarifying question or to provide clarification. So you need to be really, really intentional and really careful about how you're wording your screener questions. If they're multiple choice, what options you're choosing. The last thing you want to do is launch an unmoderated study with the N of 10. Tell your stakeholders you'll have answers in two days. You go to check on it a day and a half later and you find half the people you know didn't actually fit your criteria. Cause your screening question wasn't right. It's incredibly important.

Drew Freeman: Mm Anything else you want to add to that list?

Joe Marcantano: No, I think that pretty much covers it. What am I missing? What are you thinking of that I didn't.

Drew Freeman: Honestly don't, I don't think there's, there's nothing jumping out off the top of my head.

Joe Marcantano: Okay.

Drew Freeman: I think really it's just about you have to be extra intentional with your study plan. And this is the method. We thinking of it as a script rather than a guide is incredibly useful. Okay, so let's jump into, let's fast forward. We've got our data and now it's time for analysis. How do we do analysis? Obviously it's going to be different than moderated testing.

Joe Marcantano: Yeah. And it's going to be different depending on the tool you're using. Some of the tools have better analytical features than others. so, you know, one tool might automatically give you time to completion, deviation from the path. they

00:25:00

Joe Marcantano: might sort and tag things for you already the different answers, whereas other tools might not. They might just give you 10 transcripts and 10 videos. So the level of analysis you're doing there is going to depend based on the tool you're using. What I would very, very strongly encourage, and we've, we've talked about this before, is you need to, you need to watch Every video. Now maybe that means you're watching on one and a half or two times speed. But as of right now, even the best tools that are out there are still doing analysis based off of the things the computer can detect. So that's clicks and that's the transcript. They are not taking into account wide eyes, confused looks, 20 second pauses, you know, these kind of things that you need to be watching the video to see. So somebody might say oh yeah, that was totally clear and they're using a sarcastic tone. Well the computer, the transcript is going to reflect that as a positive, you know, attitudinal note and move on. Whereas you or I are. Any human watching would know that s sarcasm. They didn't really get this. This was not clear.

Drew Freeman: And some of the things that you've mentioned, there are things that you need like a video recording of a participant's face to notice. In my experience I typically don't get that when with the tools that I've been using for unmoderated testing. And I will still always recommend you need to watch every single video. And one of the things that is the most clear evidence for why you need to do that is the things like the participant is moving their mouse all over the screen because they can't figure out where to click. Or like you said, that long pause or the hemming and hawing that a transcript may or may not pick up on a human watching it even at 1.5 speed. Absolutely.

Joe Marcantano: Can you brought something up there that reminded me of something we should have talked a second ago. One of the downsides of unmoderated, some of the platforms out there will require some sort of download, some sort of extension on the participant side and that's so that they can record the screen, the participant face and audio. Especially if you are researching a product or a tool used in a work setting, you're going to run into a lot of hesitance participants who are not comfortable downloading something, particularly on their work computers. So just something to keep in mind there.

Drew Freeman: We talked earlier about how unmoderated testing can be more diverse and more more reflective and accurate of your potential target user group, which is fantastic. However, that need to have technology to be in order to be able to run. This can limit you to folks who have a laptop, folks who understand how to use a software tool like user testing or the other tools that are out there. Sometimes it can be easier to get assistance from maybe a local community support group or you know, what have you. For participants who it can be easier to get that support for moderated testing for someone who needs technological assistance, for example.

Joe Marcantano: For sure, absolutely. There's one more thing I want to hit on before we kind of wrap for this episode. and that is there are AI tools out there, AI research platforms that will claim to do moderated testing where an AI agent is the one doing the interview. And I want to throw out a word of caution. I've evaluated a few of them, not all of them, but several. And I would tell you that these tools are unmoderated testing. There is no human to human connection. They do not ask follow ups in the same way that a person would or as often as a person would. And I think that the most obvious example here is if you call a 1-800-support number and you talk to a chatbot, you are not going to interact with that chat bot, no matter how human like it is, in the same way you would interact with a person. So that is something to keep in mind. I'm not saying that these tools aren't useful or don't have their place, but they are unmoderated tools regardless of how they market them.

Drew Freeman: You might be able to make a case that they are unmoderated

00:30:00

Drew Freeman: plus or on the better end of the unmoderated spectrum because you can get some degree of follow up and asking why. But yeah, I think calling them AI moderated is a stretch at its most generous.

Joe Marcantano: Agreed. And there are some unmoderated tools out there where if they detect that somebody has not answered a question fully, they will insert a follow up question or a prompt to expand on that. And that's great. But again, that is not moderated. That is unmoderated.

Drew Freeman: Anything that you want to cover when it comes to best practices on communicating and reporting out what you've learned from an unmoderated test, I think that the.

Joe Marcantano: Thing that's most important to communicate out in addition to your story on what you learned, is making sure that folks have an understanding of the limitations. And that is not to say like undercut your work, but make sure your stakeholders understand this is what the test told us. These are the risks involved. These are the things that we are less certain of. You do not want to present something as 100% fact. If you did it, all you did was an unmoderated test and you don't have a deep understanding of why or you can't really provide that deeper insight, that a moderated could.

Drew Freeman: What you just said is essentially best practice for any communication and reporting it. It's just extra true when it comes to unmoderated.

Joe Marcantano: I think that's kind of the theme with unmoderated, because there's no person there. You just need to be extra careful and extra intentional about everything because you're not there to correct it in real time.

Drew Freeman: I think that's a really good place to leave this episode. Unmodated testing is really no different than any sort of testing you might be doing. It just has slight different tweaks and places where you need to spend extra energy. All right, thanks, Joe, for walking us through unmoderated testing. Hopefully we'll hear more about that in the future. But for today, thank you all for joining us and for listening. Please give us, a like and a subscribe or a review on your podcast platform of choice. That's really helpful for getting the word out and for others finding the show. If you have a friend who you think would be particularly interested or it would be beneficial for them to listen to an episode, please send the episode to them directly. We'd really appreciate that. you can also send us the questions that you would like to hear us answer, and you can do that at inside uxrmail.com. joe and I have great ideas on what we want to talk about, but we'd love to hear what you want us to talk about rather than us coming up with all the ideas and maybe missing the mark with that. I'm Drew Freeman.

Joe Marcantano: And I'm Joe Marintano, and we'll see you next time.

00:33:03