26.  What’s the best way to analyze my data?

Drew Freeman: Hey, Joe, how's it going?

Joe Marcantano: I am well, Drew. How are you?

Drew Freeman: Feels good to be back on this normal recording schedule for us.

Joe Marcantano: Yeah, it does. It's great to be back in the swing after we took a break. And now we're on our second episode back, which is really cool.

Drew Freeman: And with our second episode, were'going to get into something a little bit more kind of traditional to what we do. So the question for this week doesn't come from anyone in particular, but it is definitely one that you and I have both asked and answered a lot in our careers. So our question for this week is, what's the best way to analyze my data?

Joe Marcantano: I love this question because it's so huge, but also so specific. You know, it's what do I do after I'm fielding? But then also, like, literally the minutiae, how do I do it?

Drew Freeman: And for me, it always feels like I have to mentally gear myself up for it, because on the one hand, this is where we actually learn stuff, and that's really fun, and that's, like, the great part of our job. But also, I just feel like I've run a sprint after moderating or putting a survey into the field, whatever, and now it's like, shoot, I'm not done. I have more to do.

Joe Marcantano: Yeah, I agree. It's always a little hard right when analysis starts for me because I always feel like I need a little bit of a recharge after fielding for three, four, five days, whatever it is.

Drew Freeman: Yeah, it definitely takes me a minute or a day or, you know, whatever to kind of really fully ramp into it. So let's start there. Joe, how do you spend those ramp in hours or days? What are the first steps that you take?

Joe Marcantano: So the first things that I do are I go back to, you know, the headlines that I have been writing out every day during fielding, and I look over those, and, I think back to all the sessions, and I. I just kind of give myself a gut check. Are the things that I wrote in the middle of fielding still the general direction of where I think the story is? Did things change drastically after the first couple of sessions? I like to look and just kind of give myself a gut check and say, all right, am I on the path that I suspected we were going to be on.

Drew Freeman: Yeah, I do something very similar, which is go back to my research questions or my research objectives and I make sure that I have an answer to those. And then I can pick out a couple of, whether it's clips or quotes that really illustrate those answers. And that's very similar to what you were just talking about. Because if you're doing your moderation process well, that's what your headlines are all about. Your headlines are going to be answering those research questions. The next thing I do is write a very minimal outline of like the story that I want to tell. Because that's really the whole purpose of doing research in the first place, is to be able to convey the information we learned. And what is conveying information if not a story? And stories are really powerful ways to get folks to remember and kind of absorb what we have to say. So really trying to think about what story the data is telling me is, is an, important part of my process.

Joe Marcantano: I want to kind of call out something that you said but didn't say there because you said that you wanted to kind of outline your story to storyboard it, so to speak. You didn't say though that you start going through the data. And that's because I think there's this perception out there that it's a very kind of regimented process. Right. We plan, then we recruit, then we field, then we analyze than we present. But in reality these steps all kind of overlap a little bit. And even if I've got a dedicated time, in my plan to do only analysis were human beings. And because I did all of the sessions, I've already kind of started to pre do a little bit of the analysis in my head. And sometimes that's even despite my best efforts not

00:05:00

Joe Marcantano: to.

Drew Freeman: Yeah, I think that is something that, I cannot stop my brain from doing and that is analyzing the data or thinking about the story before it's actually quote, unquote, analysis time. And I think that's okay. I think that's perfectly natural and in fact can be a really big time saver. The key there is to continuously reorient yourself, even during moderation, to what are my research questions, what am I trying to accomplish with this research? To make sure that you're the kind of pre analysis or the mental analysis that you're doing is actually staying on track and moving us towards your objective.

Joe Marcantano: I do the same thing. I definitely come into analysis with a, with an outline, a general idea of what my story is probably not quite 75%. But one of the things, and you know, part of this is just experience and knowing myself is that I will tend to fall victim to, you know, either the first or the last sessions, leaving stronger impressions for me. So for me, when I'm going back to do the actual analysis, when I'm sitting down to start reviewing the data, I'll start in the middle. I'll start with those sessions that will kind of have gotten lost, know it's the Tuesday session that's right before lunch, those kind of things. And I will bring those back up to the forefront and I'll start there before I then start watching the sessions that were first and last or reviewing the transcripts that were first and last.

Drew Freeman: Okay, so what you're saying is hinting at kind of two different approaches to analysis that it sounds like you and I kind of both naturally fall on different sides of. So tell me if I'm wrong. What I'm hearing you say is that you tend to go session by session or transcript by transcript. Whereas what I do is actually go across sessions question by question or topic area by topic area. Am I understanding you correctly?

Joe Marcantano: Yeah, so I definitely do that. So I will go back and if I have a question or I'm noticing a link and then if I'm going back, I will go back question my research question, my research question. But my first pass, I definitely e either videos or transcripts, I will go session by session to go through them that way.

Drew Freeman: That's interesting. I don't think that there's a right or a wrong answer there. It's just those are two different approaches that you can take and obviously you and I both use both of them, but we kind of have the one that we prefer, or the one that we tend to more often.

Joe Marcantano: Yeah, the I, will say that the way that I do analysis is probably a little bit unique. You know, im, Im sitting in my office right now and on my office wall is a six foot whiteboard and I do all my analysis, almost all of it on the whiteboard. Ill use different marker colors for the different research questions and Ill divide up my whiteboard the by the participants and I'TAKE my notes directly on the whiteboard or sometimes I'EVEN print up transcripts and I'USE a piece of tape to attach a really good quote that I want to do later or use later and I'll tape it right to the board.

Drew Freeman: Yeah, I almost tend to use a super basic, super kind of a skeleton version of My readout deck for that kind of planning and creating the body of my report.

Joe Marcantano: And there are lots of other ways that folks do it. You, we can talk about affinity mapping, coding. There's lots of ways. I don't think that there's a wrong way. I think that people should experiment with different ways to do analysis, see what way works the best with their brain and how they like to think, but also to see which analysis method works best with which research method.

Drew Freeman: The other thing to think about is that these analysis methods or these, these methods of looking at your data, they're not mutually exclusive. You can combine as many as you want in order to help you get to the best outcome. Coding and whiteboarding are, you know, they work really well together and you can absolutely use both in your analysis process. That's absolutely something that I'm doing as I'm, you know, making that skeleton deck that I was talking about. For example, I'm going through encoding things and making counts and, you know, like you said, making all those kind of connections that are often more visual. So that kind of visual aspect, whether it's pen and paper, whether it's something like Miro, whether it's, a physical whiteboard, something visual, I find is really helpful for all of that.

Joe Marcantano: And the final piece that folks should consider is, you know, what tools do you have available to you? You know, not everybody has a MIRR subscription. even though they have free tiers, don't put company data into stuff that is not approved by your company. you know, some of the research platforms out there have built in automatic coding and that might work really great for you. maybe you're working at a place that doesn't have a dedicated research tool. So you're using Google Meeter Zoom and your notepad, that's okay too. But maybe that means that finding transcripts is going to be a little tougher for you, so you may not lean towards using a transcript for analysis.

Drew Freeman: Yeah, I took notes, did analysis, I did everything manually with pen and paper for the first, I don't know, five years of my UX career. And it was really challenging for me to move from paper to electronic.

Joe Marcantano: Same for me. The first time somebody showed me an electronic notes system, this was in, Google Sheets. And I, remember thinking, you know, I've been taking notes with pen and paper on legal pad for, you know, five, six years and it works. Why would I change now? And, the first couple studies were rough, but once I started doing that and having that ability to scroll across the Question. that certainly made double checking things or going back and being like, you know, something doesn't feel right about how this story is framing. I want to check this. It made it so much easier to go across participants that way.

Drew Freeman: Absolutely. Okay, so let's jump to a different analysis topic, which is analyzing qual versus quant. Hope. Hopefully most of our listeners know kind of the broad strokes, the basics of this, but let's jump in. How do you analyze qual versus quant data differently or the same when you're.

Joe Marcantano: Doing qual versus quant? You know, we've talked about this before. Quant really can't give you great answers to why, so you're looking for different things. The qual, to me, is a little more, I don't want to say forgiving, but maybe there's a lot more leeway to find the analysis method that works for you.

Drew Freeman: There's also a lot more room for researcher interpretation, for good and for bad.

Joe Marcantano: Absolutely. And when you're doing quant, you know, you're either you've got a sheet that has the formulaas in it, or you're using a calculator, and ultimately it's up to you to interpret that data. But the finding the what is a little more rigid and structured that way.

Drew Freeman: Usually what you have to do, and usually the challenge with analyzing quant data is one, cleaning the data so that it's in a state where you can actually see it and run calculations and whatnot. But two, figuring out which calculations are the. Are the right ones to run.

Joe Marcantano: You said something there we should probably dive into just a little bit and maybe not too deep, because maybe this will be its own episode. But when you say clean the data, what do you mean by that? talk to me a little bit about cleaning data when it comes to quant studies.

Drew Freeman: Okay, so quant in particular, because it's a little bit different between quala and quant. Quant in particular, cleaning the data typically looks like one, getting rid of obvious nonsense data, whether that is someone who just typed in gibberish or, you know, answered the wrong question, whatever, it might be just getting rid of that. Secondly, it might be transforming your data into a slightly different format. For example, a multiple choice. A multiple choice question on a survey might have words for the participants to select from, but when you're actually running analysis, you might need those word categories transferred into a numbered list, 1, 2, 3, 4, 5. To be able to run your calculations. So doing a transformation of Apple equals 1, banana equals 2, that's also data Cleaning. Those are the majority of cases where you're doing data cleaning.

Joe Marcantano: Yeah. And depending on what platform you're using, there's certainly some degree of automation there, particularly when it comes to things like straight ling survey answers or the folks who complete the entire survey in, you know, 9 seconds, 10 seconds. You can clean very quickly that way. But

00:15:00

Joe Marcantano: on some level, you, as the researcher do need to, to go through and look at the responses and make sure everything kind of jives with how you would expect responses to look.

Drew Freeman: And honestly, this can be a really good use case for working with a, partner who specializes in surveys, because those companies, those people typically have some sort of algorithm or tool or process that they can automatically run data through and it will callull out most of the bad data for you. Anything else? Anything I missed, Joe?

Joe Marcantano: No, not that I can think of. I think it would be interesting to do another episode later on on how do I clean quant data? but I think for what we're talking about today, that covers the high points.

Drew Freeman: Okay. So that's kind of the main differences between analyzing qual and quant data. I do want to make sure that we hit on, I think, the most important similarity between analyzing the two different types, which is no matter which one you're analyzing, you need to let your research questions, your research objectives guide you. That's number one. Number two, you still need to tell a story. Whether that is a quant data focused story or a qualitative words, feelings, behaviors, kinds of kind of story. You're still telling a story either way. You're still letting your research questions guide you. The process is really pretty similar at the core level.

Joe Marcantano: Drew, I'm wondering if you've seen the same thing I have, particularly among more junior researchers who will, will take what you just said to heart, as they should, but to the point where they are sticking to the research questions but sacrificing the story. And what I mean by that is maybe when they wrote out their research plan, they had four research questions, five research questions, whatever, and they feel the urge to do the presentation, answering question one, answering question two, answering question three, and so on, when really you might be able to tell a more compelling story. If you do something like change the order around or hit one several times, or do whatever, that kind of creates a more engaging, memorable story, but still answers all the questions.

Drew Freeman: I love that. I'll use an example from a job interview kind of mock study that I did where when I was doing the analysis, yes, of course I had my Research questions for, you know, that I was going to answer. But it struck me as I was listening to my mock participant that, hm. This company might not be targeting the right users. That wasn't a research question that they asked, but that it became clear to me that that was the most important potential learning. So I built everything around that. I still answered all the research questions, but I built it around the. This person that we're talking to might not be the right thing, might not be the right person, and we might not be asking them the right questions or questions that they know the answer to.

Joe Marcantano: Yeah, that's awesome. That's such a great call out that, you know, you don't want to not answer your research questions, but sometimes your research questions may not be the most important thing you discover. And there's not really, a hard and fast rule for that. Like you need to kind of learn your industry, learn your vertical, learn your company ##n and kind of make that call to say, hey, I answered questions 1, 2 and 3. Here they are. But this is the big thing that I saw that actually I think is the main takeaway here.

Drew Freeman: We didn't even know that we needed to ask question for, but it turns out that's actually the most important one.

Joe Marcantano: Yeah, exactly.

Drew Freeman: So that leads us nicely into kind of the next thing that I look for when I'm doing analysis. And put really bluntly, these are just things that make me tilt my head and go, m. You know, the visual that I have in my head is a dog when it tilts its head, when it's like, I don't understand what you just said. So I have a mental image of a dog tilting its head in that cute, questioning manner every time I think about this. And things that make you go, are just things that, you know, trip your researcher spidey senses, your researcher intuition. Often those are things like patterns that emerge, outliers. So if most people are saying X, Y and Z, but one person says abc, why did that person say abc? Are they doing something differently? Are they using the product differently than everybody else? And maybe is there

00:20:00

Drew Freeman: an important learning that we can take from that?

Joe Marcantano: Yeah. It is almost hard to describe what it is specifically that you're looking for when you're kind of, when you talk about things that make you go home, it's a I'll know it when I see it kind of thing. And oftentimes it's those perfect little nuggets that are great, for kind of conveying the story or the points across that you want to tell.

Drew Freeman: This is absolutely one of those places where research is an art and not a science.

Joe Marcantano: Agreed. I want to cover one more thing before we kind of start to wrap this episode. I want to talk about moderated versus unmoderated. Talking specifically about qualitative research here. I'm curious, Drew, do you analyze moderated data versus unmoderated data differently?

Drew Freeman: It depends on how much time I have for the analysis process. If I have enough time and the videos that we get from the unmoderated, you know, participants are short enough, I'll just go through and watch everything and take notes like I. As if I was moderating. So if I have the time, then no, my process is pretty similar. You know, I do the same. I'm coming up with the story as I'm. As I'm watching it, as I'm watching the sessions. It's just that those sessions are crammed into a smaller space then. Then the moderated ones usually are. But if. If there isn't time m. Then I will. I will focus and kind of skip ahead and focus more on answering those research questions. And then really, just as I'm answering those research questions, listening to that inner dog in my head that's tilting its head and going,

Joe Marcantano: I am a firm believer that if you run unmodated, you need to watch every session after the fact. Now, you can be watching it on one and a half or two times speed because you already have a transcript, presumably because most of these tools will do transcripting automatically. But those transcripts won't pick up things like long pauses. They won't pick up things like confused looks. And if somebody says something that is contradictory to their facial expression or their actions, I want to know about it. So when I am planning out a study and creating the timeline, if it's unmoderated, that's a deal breaker for me. I will ensure that I have enough time to watch every video, even if it's on one and a half or two times speed.

Drew Freeman: So I agree with you that watching the videos is absolutely the best practice. I'm going toa play devil's advocate a little bit and say, what about those times when someone else moderated the sessions and you just get handed the data to do analysis? That is. That feels pretty similar to me to what I just said, when it's unmoderated and we need to go fast.

Joe Marcantano: Yeah. And I've had instances like that. And you know, the typical research answer, right. It depends, like, how much do I know about this person's moderation skill? What kind of notes did they take? Is this you handing me the data and the transcripts, or is this somebody really junior who I would expect to miss Things, you know, those kind of things play into it. If you or someone of your experience level is handing me notes in an outline, then I'm, comfortable going with that.

Drew Freeman: As almost always in research, the answer is it depends. But I think we're both in agreement that the best way to do analysis for unmoderated is very similar to moderated. And that starts with watching all of the sessions, even if it's on 1.5 speed.

Joe Marcantano: Drew, any other points you want to hit before we, call this second episode of the New year?

Drew Freeman: No, I think we covered it pretty well. So thanks everyone for listening. Thanks for again for sticking with us through our end of year holiday break. Please give us a like and a subscribe on your podcast platform of choice. If you enjoy the show and want to help us grow, which we hope you do, please leave a review that helps us with the various algorithms that these tools all use. Also, something that would really help us. Send us questions that you would like us to talk about at insideuxr@gmail.com and if you'd like to support us, there's a link in the show Notes where you can do that.

00:25:00

Drew Freeman: I'm Drew Freeman.

Joe Marcantano: And I'm Joe Marantano and we'll see you next time.

00:25:04