24.  How do I become a better moderator?

Joe Marcantano: Good morning, Drew. How's the weekend going for you?

Drew Freeman: It's going well so far. It's been a nice, quiet, sleepy weekend for me. Those are always the best kind, 100%. Especially as we come into the holiday season. There's notnna be a lot of them.

Joe Marcantano: No, no. We are entering the busy season. I have, we're doing something new this time. I pulled a question that I saw in a, Reddit subreddit that I follow, and I thought that this person's question and kind of the way they framed it would make a really good episode.

Drew Freeman: Hit me with it.

Joe Marcantano: Awesome. So I'm paraphrasing, but basically the person says, I'm a newer researcher and I find that I'm struggling with getting better at conducting interviews. I hear the typical advice, things like embracing silence and asking why, but it's hard for me to know when and how much. What are some things I can do to become a better moderator?

Drew Freeman: That's a great question and one that I'm sure both of us, as experienced folks have heard a lot.

Joe Marcantano: Yeah. Especially given the state of research right now, where, you know, 80ish percent, if not more of all research, is remote IDIs one on one. So this is definitely a skill that if you are in qualitative research, you need to sharpen and be pretty good at.

Drew Freeman: It's one of the bread and butter skills for sure. Okay. So I did a little bit of pre work on this and have come up with kind of a good, better, best, approach to this a little bit. And some categories, though, that I think moderation can kind of fall into. So let's start with, let's start with keeping track of time as a moderator. That might not be something that you think about as like, I need to improve this skill in terms of moderation, but it is so important in terms of running the actual session.

Joe Marcantano: Yeah, this is something that I see. One of the things that I see more junior researchers tripping up on the most is not watching the clock throughout the session.

Drew Freeman: And that can manifest in two different ways. That can be, I'm going too long and I'm not getting to all my questions. But it can also be, I'm ending with plenty of time left in My scheduled interview time when it comes to.

Joe Marcantano: Going long, and, I know that there are some folks out there who will ask up front. You know, do you have a hard stop? Can we go over a couple minutes if we need to? I am a believer of, because the participant is being compensated for their time, that the session ends at the end time. There is no extra couple minutes. I am a firm believer that that is the time 100%. You have to stop asking questions at that point and say your thank yous.

Drew Freeman: And I'm a believer in that. Even if the participant shows up a few minutes late.

Joe Marcantano: Agreed. If the participant shows up, you know, five or 10 minutes late and they offer to make it up, I may take them up on it. But I will never suggest going along to a participant.

Drew Freeman: Okay, so what does good keeping track of time look like? I mean, it'just what is just what you talked about. It means starting and ending the session on time. It also means, like you said, it also means getting through all of the pieces of your discussion guide. And an advanced moderator will appropriately elongate or shorten the sections where needed. So you might elongate a section where the participant is giving really valuable and really important feedback. You might shorten a section where the participant doesn't have as much value to give.

Joe Marcantano: When, I write discussion guides and when folks on my team and really most researchers when they write discussion guides, you should be putting a time estimate at the top of each section. Now, when you're a newer researcher or more junior researcher, those are pretty much going to be guesses. But as you get a little more advanced, as you get a little bit more experienced, those estimates are going to get closer and closer to

00:05:00

Joe Marcantano: the actual time. Using those as a benchmark is a really good way to keep track of time throughout the session. So if I know my intro is going to be, you know, five to 10 minutes, and then my next section is going to be 20 minutes, I know that half past I need to be through the first section M. And that's a good kind of midpoint to be like, all right, look at the highway signs here. Am I going in the right direction? Am I at the pace I expected to be?

Drew Freeman: Do I need to speed up or do I need to slow down?

Joe Marcantano: Exactly.

Drew Freeman: So another thing that I don't think a lot of people necessarily think about is how much time are your transitions between tasks or between sections taking. A really good moderator will have practiced those transitions so that they're seamless. You're not losing time, and you're not taking the participant out of whatever headspace that they're in.

Joe Marcantano: Yeah, there are lots of ways to do it. if you're running kind of a rapid research where your topics, you might have two topics or three topics and they're not completely related, you might just do a hard transition. You might say, all right, thank you so much for your help on that. I want to completely change gears now and just call it out and let them know we're turning here and that's it, then you're moving on. If you're doing stuff that's related, you can obviously be a lot more subtle and just say, okay, now let me show you something like this. We're gonna imagine we're doing this.

Drew Freeman: Yeah. When I say seamlessly transition, I mean more of cutting out the silence and cutting out that awkward pause as you're doing, you know, whatever transition work you need to. That's really what I'm talking about. I don't care if it's a hard, a hard transition, a, you know, ease into a transition. I don't care. Okay, so then the last kind of bullet point in this timekeeping bucket that I want to talk about is how do you deal with tangents? Because participants are going to bring up tangents and I think there's a good, better, best way to handle tangents.

Joe Marcantano: This is always a little tough because on the one hand, sometimes tangents give you the best insights. They're what's unexpected. But on the other, if you've got a discussion guide that you think time was already tied on, a tangent, even a 32nd or a 62nd one can really start to derail you and put you behind.

Drew Freeman: This is where you as a researcher have to use your brain and figure out is this tangent valuable or is this a, you know, is this a tangent that does not offer valuable insight or feedback for me.

Drew Freeman: Those valuable tangents, those ones that are still related, absolutely. You can let those ones go. The ones where they're not valuable, they're completely off topic. They'they're not gonna help you in your analysis. A good moderator will simply cut those off with something like, you know, thank you for that, but I'm gonna bring us back to this topic. A really like a Rockar moderator will ra than just say, rather than just like hard cut off the tangent will like steer the participant back to where they wanted to go.

Joe Marcantano: And you know, the easiest way to tell if a tangent's goingna be valuable or not.

Drew Freeman: What's that?

Joe Marcantano: Look at your research Questions which should be at the top of your discussion guide. Everything always comes back to your research questions.

Drew Freeman: Good callb back to an earlier episode. Okay, so then the second bucket that I kind of have in place is like your researcher presence, essentially. You need to make the participants feel welcome, safe, comfortable, giving feedback, basically, because for the participant, this is an awkward and potentially stressful experience for them as well.

Joe Marcantano: Yeah, I think of this bucket here as kind of like the rapport building.

Drew Freeman: Bucket, you know, setting the scene.

Joe Marcantano: Yeah, yeah, exactly. Know, making sure that they know all of the necessary things, you know, like, hey, this is the NDA. These are the kinds of things it's covering. We're going toa be recording this. If you're doing it in person, if you need to use the bathroom, just let me know. It's right down the hall. And we can pause, like all of the kind of like housekeeping things. The human to human part of this, rather than the researcher to participant part of this.

Drew Freeman: So I think those things that you just mentioned, that's the foundation. And then a rock star moderator takes that and builds upon it with the, like, presenting a calm and welcoming demeanor. you know, making it clear that we're not judging you. Like, this is a, for lack of a better word, this is a safe space for you as a participant. And that can even come with

00:10:00

Drew Freeman: the research. You as the researcher, helping the participant become less nervous and putting them at ease and drawing out that feedback. These are all things that kind of go into this rapport building and setting the scene.

Joe Marcantano: I think that it's important to call out here that nobody expects a junior researcher who's running their first, second, third project to not be nervous. Like, that is a natural part of the growth. Nobody, maybe somebody has, but nobody starts at the advanced level. No, it takes experience, it takes practice. You got to build there.

Drew Freeman: Yeah, 100%. So for me, the kind of technique and the overall tool that I use to try to make the participant feel welcome is humor. So, for example, in my introductory spiel, obviously we always have to say, you know, this session is being recorded, but it's not going to be shared outside of the research team, you know, all that sort of thing. So I'll add in. I almost always add in a little line like, if you're looking for your 15 minutes of fame, you're going toa have to look elsewhere. Little things like that can go a long way.

Joe Marcantano: Yep. One of the things that I always do is tell folks, you know, hey, I wasn't on the team that built this so you're not gonna hurt my feelings? Like, I have no skin in the game. You're not gonna flatter me, you're not gonna hurt my feelings. If you hate it, just tell me. I'll be the bad guy and I'll go tell them that there's something wrong with it.

Drew Freeman: And I think it's important to say that even if you were a part of the design team, and even if you did have a hand in creating it.

Joe Marcantano: Yes, for sure.

Drew Freeman: So then the next bucket is. I guess I think of this as like the craft of research. So this would be things like, how am I communicating the best and how do I use my. My everything, my words, my presence, my body language to get the best feedback that I can out of this participant?

Joe Marcantano: Yeah. If the last section was the human to human, this is much more of the researcher to human. A researcher, a participant. Excuse me, section.

Drew Freeman: so this is things like using follow up questions that are relevant to the research goals. this is things like avoiding bias in terms of just yes, no questions or leading questions or whatever kind of bias.

Joe Marcantano: Yeah. And to kind of tie back to what this person was posting about. when they asked this question, they were asking about how do I know when and where to ask why? And I've already said it once today. Look at your research questions. Uh-huh. It is important to know if the prototype is intuitive. Right. But it is also important to know why it is or is not intuitive. And that may not be explicitly stated in the research question, but your stakeholders are expecting an answer to that. And so if you get to a point where somebody gives an answer and they didn't explicitly explain how they got there or the reasoning behind it, that's the spot that you ask why.

Drew Freeman: I would say I'm struggling to think of a time when it is inappropriate to ask why at least once on like every section or every task or what have you.

Joe Marcantano: I agree. You may have participants that explain why naturally is a part of their answer, and that's great. but you should almost always be asking, tell me about what led you to believe that or tell me about how you arrived at that.

Drew Freeman: Something that you just incorporated. I think is a really good tangent for us to go on, which is you can ask why without using the word why, and you should.

Joe Marcantano: O. So I am a big believer in don't ask why. don't ask the question why. I think you should always be asking why, but phrasing it differently and for a couple of reasons. The first being nobody wants to we always talk about the five why's and kind of channeling your inner toddler, but you don't want to actually sound like a toddler.

Drew Freeman: It's so annoying.

Joe Marcantano: The backseat of the car going, why? Why? Why?

Drew Freeman: It's so annoying?

Joe Marcantano: It'not yeah. Also, some folks may interpret why as an adversarial or a challenge question.

Drew Freeman: Uh-huh.

Joe Marcantano: I do not trust your expertise. Show me your work. Whereas if you were to say, hey, can you help me understand that better? How did you arrive at that conclusion? Or can you just tell me more about that? That is much more inviting.

Drew Freeman: My favorite way to get at the why is to ask them. Hey, can you tell me what you were thinking here? Yeah,

00:15:00

Drew Freeman: I love that. Like, asking why is what we need to do. Asking why is not how we should be doing it.

Joe Marcantano: There are a couple of, other methods to do it. You know, that, that, I would say are more advanced methods. You could do something like echoing. So if somebody says, well, then I would click here. And you say click here. You kind of repeat the last couple words they say with a question in inflection.

Drew Freeman: So that echoing makes me a little bit uncomfortable because it'really easy to do poorly.

Joe Marcantano: Agreed. It is definitely a. A method that you need to be very careful about.

Drew Freeman: Because, like, even your example of click here as the question, I'm like, e, I don't love that.

Joe Marcantano: So I get your trepidation about echoing because I have it too, sometimes. One area where I've seen a lot of, success with it is if I am doing an interview about a fairly technical product or feature or someplace where I'm not really the expert. And so it allows me to kind of ask that question without running the risk of misusing jargon, but also to, like, kind of convey the. Hey, I don't quite get what you were saying there. And maybe my click here example wasn't the best, but I don't quite understand what you're saying here. Can you tell me more?

Drew Freeman: Yeah, I like echoing in more of a. Okay, so I hear you saying that this is confusing. Tell me more about that. You know, that kind of thing. Like, the participant says, oh, I don't like this button. I find it confusing. You know, I don't know what it's gonna do. Okay. You know, I hear you saying that you're not sure what this is gonna do. Tell me more about that. Or I hear you saying that, you're not sure what this is gonna do. What do you think it's going to do? What do you expect it to do that kind of echoing? I really like.

Joe Marcantano: There's another method, and I use this sometimes, although I'm not super fond of this method is the trailing off, where you'll just kind of start saying, okay, you've gone to the movie theater, you're looking to buy tickets, and then kind of like getting them to go to the next step. There are times that I like this method, but it also runs a fairly high risk of kind of, leading, which can be okay if you're trying to lead them in the right scenario, but not necessarily if you're trying to like, truly gauge what they might do next. What are your thoughts on the trail off?

Drew Freeman: I think with all of these, like, more advanced methods that we're saying, it really comes down to practice. And you just, with that practice, you just kind of get a sense of, this is the time to use this, this is not the time to use this. Or at least that's how it worked for me. And I't. I can't explain if this, then use echoing. If that, then use trail off. If something else, then use tell me more. Like I can't come up with a if A then B kind of scenario. It's just kind of a. I practiced a lot, I ran a lot of sessions, I made a lot of mistakes, and now I have this kind of, you know, I've built up this kind of just intuitive understanding and sense as to where I should be going. And it changes with each participant. That's the other hard thing, what one participant responds to really well. The next one that might make them close down and give you less information.

Joe Marcantano: I think that this is where we kind of fall into the art of moderating rather than the science of research.

Drew Freeman: I do feel like research as a whole is both art and science, but I do feel like moderation in particular leans more heavily on the art side than it does the science side. You have to be created, you have to be willing to change your approach, and you have to be comfortable with just kind of going with the flow.

Joe Marcantano: M.

Drew Freeman: And that's really hard to do. And the best way that I can suggest that people become more comfortable with going with the flow is by practicing. And that can be with actual participants, but that can also be with internal colleagues or mentors or friends. And just running sessions, whether their re practice sessions or real sessions is valuable. Hey, all this is Drew saying thanks for listening to the show. In order to help us know what you want to hear about, send us an email with your questions at inside uxrmail.com.

Joe Marcantano: So let's talk about some of the other things you have down here for the craftsmanship port portion.

00:20:00

Joe Marcantano: one thing that really strikes me that you have written is utilizes all forms of communications, and this could be body language, facial expressions, silence, communication, or lack thereof. Can you talk a little bit about that?

Drew Freeman: Yeah. Everything that we do is communication, Whether it is using words or using silence. That. That is communication. Our body language, our facial expressions, those are communications as well. So if I, as a researcher, you know, participant tells me something and I raise an eyebrow and cock my head to the side, that participant is probablynna understand.

Drew Freeman: M. I don't understand this. Tell me more.

Joe Marcantano: Conversely, if a participant does something unexpected and goes on to the you quote wrong path, don't make a disappointed face or a disgusted face or, you know, whatever. Don't have a reaction to that. Don't give them an opportunity to use your feedback to gauge how they're doing on the process.

Drew Freeman: Yeah, see, to me, that's actually like a slightly different bullet point here. That is the, like, you must be neutral, but you don't have to be a robot while being neutral. Something that a mentor told me when I was a, new researcher and a new moderator that was really helpful for me is that you are a happy neutral. You are happy that the participant is there. You are happy that the participant is giving you feedback, but you are neutral on whether they are doing it right or wrong. You are neutral in the feedback that they are giving you, but you are happy that they are there and happy that they are giving you feedback.

Joe Marcantano: I love that. Let's briefly talk about silence, because that was something they specifically brought up in this post, and you just talked about it a little bit. When is the right time to use silence again?

Drew Freeman: I don't know that I have a if a then you silence. But silence can be really helpful to draw out more information. People like to fill silence. That's just a human nature kind of thing. And I think that's especially true for participants because they feel like the value that they can provide is giving feedback. So if you are silent, they want to fill that by giving you more value, which is giving you more feedback. That is typically what I have seen.

Joe Marcantano: Drew, I'll tell you one of my favorite stories from when I was a detective.

Drew Freeman: I was going to say, I'm sure you actually have more insight into this than I do.

Joe Marcantano: Yeah. So I was at the time, I was investigating burglaryes, and we had a case where Somebody broke into a house, they stole some stuff. They were caught on the home surveillance system clearf face picture. And then they went and sold these items to a pawn shop where in the state that I was a detective. You have to show your ID to sell these items. So we had the guy like we knew it was him, that this was not a whounit. He gets picked up, he gets put in the interview room. And I walk into the interview room and I sit down with my notepad and I flip it to the open page. I take my pen out and I open up my pen and I set it down and I lean back and I just sit there. And I bet you less than 30 seconds went by. And he said, look man, I had to steal that stuff. I'm really sorry. The uncomfortableness of 30 seconds of silence was enough, without me even asking a single question to get this person to fill the void and admit to the crime they had committed.

Drew Freeman: So in a research moderation setting where you don't have that kind of adversarial relationship, the way that I think silence is most important is to give time for the participant to collect their thoughts.

Drew Freeman: So you know, especially if the participant kind of trails off their last sentence, give them five or 10 seconds. Maybe what they're doing is just thinking in their head, how do I want to phrase this? What am I actually thinking? Like, what do I actually feel? And then if they don't, if they don't give you anything after that, like five or 10 seconds, then you can go on to your next follow up question.

Joe Marcantano: One of my favorite ways to use silence in an interview, a, research interview, is I like to tell the participant, hey, I may pause after I ask you questions or after you answer. It's because I'm taking notes.

Drew Freeman: Yes.

Joe Marcantano: So number one, it gives me kind of this social like

00:25:00

Joe Marcantano: acceptance to be quiet and not respond to them immediately. But they very quickly forget that. And then they very quickly go into this, it's quiet. I have to fill the quiet mode. And so I'll be typing away or writing away and then they'll just dive back into their answer and tell me more.

Drew Freeman: I love that.

Joe Marcantano: And I will often do that even if I have a note taker.

Drew Freeman: 100%. 100%, yep. So, the last thing in this kind of craft, of research, craft of moderating area that I use to help me identify rockstar moderators is how well does the moderator kind of redirect or reflect questions that the participant asks you as the moderator. So a, Participant might ask, like, is that the right thing? What does this button do? You know, questions like that. I think we all have gotten those questions. A really good moderator will very tactfully kind of turn that question, not answer it, and use that to get even more feedback. So, Joe, what's your preferred method of, like, reflecting or redirecting a question that the participant asks you?

Joe Marcantano: So I have two, and it depends on how the session is going. This is, again, much more of the art and the rapport that I feel that I've built. But if somebody says, hey, know, am I doing this right? Or what happens if I click this? My question might be simply, will, what do you think would happen if we. If we click that or if you were to click that, what's your expectation?

Drew Freeman: That is. That is my go to for. That is my go to suggestion for moderators when they're struggling with this.

Joe Marcantano: Or if somebody says, you know, am I on the right path? I might say something like, what. What clues do you. What sparked the thought that you're not on the right path? Or even, do you think you''right picking up the other thing that I'll do. And again, depending on the session and the participant is it might explicitly call out, I'm actually notnna answer that. I'm curious to hear your thoughts of what that button does. There's nothing wrong with being direct and saying, I actually wantna hear your thoughts here. And at the end, I'll pull the curtain back and I'll tell you everything younna know. But for right now, I'm not gonna answer your questions.

Drew Freeman: Yeah, for me, the good is what you said second, which is, I'm notnna answer that. We'll cover it later in the debrief. That's a good way of handling a question to me the best. Like, the advanced is the first thing you said, which I call turning the question, which is turning the question back on the participant. So, you know, Joe, as the participant asks me, what does this button do? I'm not sure. I, as the moderator, say, okay, what. What do you think that button's gonna do? What do you expect that button to do? What do you think is gonna happen when you press it? Not only have I not given, like, not only have I not told Joe an answer, not only have I not let him, I have also used that as an opportunity to get more feedback from Joe.

Joe Marcantano: Exactly.

Drew Freeman: Okay, so let's move on to kind of the last bucket, which is more like the technical logistics side of things. And really, what I'M talking about here is being familiar with the tools that you're using, whether that's Zoom, Google Meet, WebEx, you know, whatever it is. Being familiar with the prototype or the product or the mock up that you're using and being ready and able to handle and recover from errors that are inevitable and will come up.

Joe Marcantano: This is something that I see one of the biggest jumps I see between junior and admin. M researcher is a junior researcher will kind of get the walkthugh of the prototype from the designer and then go right into writing the discussion guide. A mid researcher and hire will get that prototype. They'll do the walkthrough with the designer and then they'll spend 30 minutes clicking around in it. They'll play with it, they'll look for things, they'll read all the text. They will put in the extra time and effort to really familiarize themselves with.

Drew Freeman: The prototypes and what that that does two things. One, it allows you to ask better questions as you're writing your discussion guide. But two, it allows you to be more prepared and more confident in handling an error. Like if a participant gets themselves into a dead end, you're then better able to get them out of that dead end and back to where the, the starting point that they need to be.

Joe Marcantano: And it's little things. You talked about familiarization with Google Meet or Zoom, or whatever your conference software is, but also in the prototype tools know if you end up in a dead end or the prototype glitches or whatever goes wrong, knowing that you can push R to go right back to the start to restart or that

00:30:00

Joe Marcantano: your designer has built in a reset button that's on every screen but hidden like knowing these little things. So you can, if you reach a dead end or if the prototype glitches, you can go, you know what, I'm really sorry, there seems to be a tech issue. Let's restart. I'mnna click here and then you were on this screen. Let's take it from here.

Drew Freeman: Absolutely. And what you did there, I'm not even sure if you did this consciously, but is another thing that I love the errors that come up, they're not participant hers, they are errors on your side. As the researcher that is a oh, I'm sorry I made a mistake or oh, I'm sorry there seems to be a bug in the prototype even if that's not the case and the participant just messed something up.

Joe Marcantano: Yeah, the participant is never wrong. The user is never wrong.

Drew Freeman: The system is simply not understandable enough or easy enough. Or intuitive enough or whatever.

Joe Marcantano: Exactly. Anything else you want to cover in this technical part?

Drew Freeman: I think again, this one is really down to being prepared to just go with the flow and react to things as it comes. And again, I think that mostly comes with practice and that comes with time.

Joe Marcantano: Yeah, I think this is an area where it doesnt take a lot to get to advanced. This is probably of the buckets weve talked about, the easiest one to get too advanced in. And it's just a matter of putting in that little bit of extra effort so that in your mind you have a plan if something goes wrong.

Drew Freeman: I agree. Yeah, I agree.

Joe Marcantano: Awesome. Well, I think this is a good place to call it for this week then.

Drew Freeman: Sounds great. you know, I love talking about moderation and how to be a Rocktar.

Joe Marcantano: Moderator, so I want to thank everybody for joining us today. give us a like and a subscribe on your podcast platform wherever you listen to the show. I also have a special ask this week if you are really enjoying this episode or other episodes. If you really like the show, share it with one person who you think might like it. The easiest way and the best way for us to grow is that kind of organic growth where you just tell somebody else about a cool episode you heard. So please share the episode, share the podcast with somebody else. Give us a tag on LinkedIn when you do that. And we'd love to continue to grow. If you have a question that you want to hear us talk about, you can send that over to insideuxr@gmail.com and if you want to support the show, there's a link in the show notes with that. I'm Joe Marantano.

Drew Freeman: I'm Drew Freeman.

Joe Marcantano: We'll see you next time.

00:32:35