42. How do I maximize non-traditional data sources?

Joe Marcantano: Drew, it's time for another episode.

Drew Freeman: Let's hit it. I'm excited.

Joe Marcantano: All right. No small talk on this when we're jumping right into it.

Drew Freeman: Dive right in.

Joe Marcantano: All right, so this week's question is how do I maximize using air quotes, non traditional sources? So when we wrote this question, what we had in mind here is, you know, social media posts, direct customer feedback, call center reports, the things that are not surveys and interviews, things that are.

Drew Freeman: Coming from avenues that aren't developed or derived by researchers.

Joe Marcantano: Yeah, so I, know like you and I, as we were writing this, kind of struggled on what was the right word for these sources. We had secondary. We had non traditional. We had some other stuff. And I don't know that any of the words fully encompass what we're talking about, but, yeah, basically any information that comes not from a researcher.

Drew Freeman: Yeah. And I think it can be easy as a researcher to downplay this kind of information, but I think you do so at your own risk.

Joe Marcantano: Yeah, this is an area where we're probably gonna say like tiny grain of salt with everything, because there's a lot of. It depends. But there are some real gems you can tease out of these non traditional sources.

Drew Freeman: Yeah, absolutely. So where do you want to start?

Joe Marcantano: Let's start with are, there scenarios where a non traditional source is the quote unquote right place to pull information from? Are there questions or times where you would rather go there?

Drew Freeman: That's a good question. I don't know that there are times where I would say it's the best place to go, but there are certainly times when it might be the first place I go. So, for example, if I want to know what are some very common customer pain points with our product? Social media posts or bad reviews, Those kinds of things. Those are excellent places to go to try to pull a list of common pain points.

Joe Marcantano: Yeah, the thing I was thinking there is, you know what these probably lack in, you know, unbiased, like true data source, it makes up for in speed. Like I can download customer reviews and throw them into a word cloud generator and get an idea of the sentiment really, really fast. You know, rather than scheduling eight interviews or setting up a survey and waiting for it to run like these are the answers that I can get in an afternoon or less.

Drew Freeman: Speed is absolutely a big benefit to this for sure. I think the way that I would use this, like I said, this might be the first place I go because of the speed. But I'm going to use it primarily as a way to then hone in on areas where I should spend time, spend more time, dig deeper in primary research.

Joe Marcantano: Yeah, this, this can really give you like these non traditional sources can form the basis of your hypotheses. They can kind of act like a benchmark of. This is what we think we're testing that, you know, we think that because of the reviews that people are doing this. So let's number one, see if that's true and number two, figure out why.

Drew Freeman: I think it's also really important, to note these kinds of channels are gold mines when it comes to road mapping and planning out areas that need attention in the future.

Joe Marcantano: For sure. I mean, if we're looking at call center reports or support line calls, we know that that's not an unbiased source. Right. Like people are only calling if they have a problem. But if we see that 20% of the calls are about one thing, that's a really good red flag.

Drew Freeman: Exactly.

Joe Marcantano: For every person that called, there's probably two or three or more people who got frustrated and gave up.

Drew Freeman: M Totally.

Joe Marcantano: The other thing I like to do is

00:05:00

Joe Marcantano: a lot of specifically software companies and software as a service companies will talk about Churn and they're always looking. How do we reduce churn? How do we identify the warning signs of CHURN early? This is a really good place that you can start. Because Churn customers are notoriously difficult to recruites. They left for a reason. They're unhappy with you. The hundred bucks likely isn't going to do it for them. But if I can go back and say, okay, these 10 customers churned and I look at the helpline data and I see that they were calling about these things. Mostly that is very, it's corollary data, it's not causal data. But that's a really good place to start diving in and start looking and to treat as a yellow flag.

Drew Freeman: Totally, absolutely.

Joe Marcantano: Drew talk. Let's talk nuts and bolts. You know how you get a research question from a stakeholder? You meet with a stakeholder, you've sat down with them. How might you start looking at these non traditional sources? What's the actual process for this?

Drew Freeman: So one of the first things that I want to do is understand kind of the background or the context in which this research project is happening. And a piece of that can be these non traditional sources. So I might do some keyword searching in blues Sky, Twitter, LinkedIn, Facebook, whatever, and just get a sense of what are people saying on the Internet about this topic, about this product and this interaction with the product. I might ask, some customer service reps, hey, do you have a list of, you know, we're going toa be doing some research in this area. Do you have a list of top issues that you get calls about or you get tickets submitted that are in this area? And then it just kind of gives me more flavor and a better picture.

Joe Marcantano: And we've kind of hinted at this, but when we're talking about this kind of data, which is frequently, if not always anonymous data and almost always self reported, what kind of weight are you giving it? What kind of, you know, when we're thinking about like the scale of most reliable to least reliable evidence, where are you putting this?

Drew Freeman: On the one hand, I don't give it very much weight because it's less reliable. It has that anonymity, it's self reported like you talked about. On the other hand, these are especially like help tickets or call center reports. These are things that actual users experienced a problem with and that problem was great enough that they could not solve it on their own. So that lends it more to the other side of giving it more reliability and giving it more weight. So there really is no right answer here. It's one of those. It depends. In general, I look at this mostly as flavor, extra background, extra context. You know, maybe it points, it points me in directions rather than being a, a set of cookbook instructions.

Joe Marcantano: Yeah, I think of it the same way. I go back to the old adage, you know, where there's smoke, there's fire. And just because I see smoke doesn't mean I know where the fire is. exactly. But I, but have a general idea. I know a fire exists and I know it's probably that way.

Drew Freeman: Exactly. That's a really good, that's a really good explanation for it. So the other way that I will use this data that we haven't touched on yet is I will use this data sometimes in reports actually. So I might take a quote of an upset user on social media and I might put that in my background or context section of my report so that my stakeholders can kind of understand the, or get a hint at the depth of emotions that this area can cause when it's not done well.

Joe Marcantano: Yeah, I can already hear the stakeholder pushing back and saying, but we don't even know if that person's a real user of our product and blah, blah, blah. But the fact of the matter is like, that thought that sentiment is out there in the world now, and so real or not, we have to deal with it, we have to address it.

Drew Freeman: Right, exactly. I'm never going to use a quote like this, you

00:10:00

Drew Freeman: know, to back up, an insight that I'm presenting during my report, because that would be. That would not be good research. But I will use it to help illustrate the background problem, the context of the problem space that we're working in.

Joe Marcantano: Yeah, a really good way to think about this is like, I might use this information to justify why I'm doing the research in the first place, but I would not use it to justify the conclusions of that research.

Drew Freeman: Yes, yeah, that you need to save for your primary sources, your participants in a study, your survey responses, et cetera, et cetera.

Joe Marcantano: And we should probably talk a little bit about. We're kind of lumping all of these things together. Right. We're calling them all non traditional sources, but they also get different weights attached to them. I'm not going to give a social media post as much credence as I might give a scholarly article written by a reputable source or a reputable research or research agency or university or whatever. Right. Like, I'm going to consider the source when I'm assigning, how quot unquot true. Something is.

Drew Freeman: Yes. And the only kind of pushback that I would give to that is that if it is emotional context that I'm looking for, then I might weigh the social media post over that scholarly article, simply because the scholarly article is much less likely to get into or have the same emotional weight that a social media post can have.

Joe Marcantano: Yeah, that's a really good point and a great call out. Drew, why don't we talk about, the elephant in the room here? AI and LLMs. You know, I'm talking about like ChatBTS, Geminis, all of these, these products.

Drew Freeman: Yeah. This is definitely an area where there can be a lot of controversy, but I do think that there is a way that you can use those well and responsibly, but there's also a way that you should never, ever, ever use them, in my opinion.

Joe Marcantano: I agree. I'll be curious to see if our thoughts are the same here. But let's talk about, you know, talking about these large language models. What's the right way. What's the way a researcher can or should use these kind of tools?

Drew Freeman: So I think there's a couple of ways. The first is I think it's entirely reasonable to ask Chat, JPT or Gemini or whichever one you're using. Something along the lines of what are the top three pain points users report in X product? That, again, I'm looking for which direction the smoke is. I'm not that. And that's all I'm doing. I think it also could make sense if you have something that is a little bit more within the garden wall, that's a little bit more proprietary. It isn't going to be searching the entire Internet. You could give it your, you know, you could give it a database of your call center reports, for example, and say, what are the top three issues that are reported in these call center reports? I think that's another way that you could use it.

Joe Marcantano: Well, yeah, I don't necessarily disagree. I think I would just be. Or maybe I'm just vocalizing more caution than you did. Chat, GPT and a lot of these other tools have gotten a lot better about hallucinating a little less. They've gotten a lot better about providing sources. So when I'm doing that, I'm always clicking on the links to see the sources, to see where it's pulling this information from and where it's drawing these conclusions. We've all heard the old phrase garbage in, garbage out. If it's pulling in nonsense, it's going to spew out nonsense. So I'm always looking at where it's kind of drawing the information that it made these conclusions from.

Drew Freeman: Yeah, we're totally on the same page there. So to keep using your smoke and fire analogy, I might use it to you maybe there's a subreddit that I had no idea about and was unlikely to find on my own, whereas this AI might be able to point me to that subreddit. And now I'm no longer using the AI. Now I'm just reading through this subreddit.

Joe Marcantano: Yeah, you're kind of treating it as the. To continue my smoke fire. You're kind of treating chatptt as a lookout that reports smoke. You're stillnna go investigate. You're stillnna go look at it. You're not just going to deploy everybody because you heard that there's smoke over there, right?

Drew Freeman: Yes, yes, yes. Okay, so let's talk about the way that I would, Me personally would never use AI at least the way that it is now. I can't envision a world where I get there, but I couldn't envision a world where large language models existed 10 years ago. So who knows? That world

00:15:00

Drew Freeman: is what's being called synthetic data, and that is data that a large language model pulls together and then you treat it as if it came from an actual human participant.

Joe Marcantano: Yeah. The way folks should think about this is if you told JAT GPT, pretend you're, a user of X product, what do you think? You know, where might you trip up or where do you struggle in the user journey, whatever, when you are asking the large language model to be your participant.

Drew Freeman: Yes. Yep. That. That is what synthetic data is. And I know. I feel very strongly that you should not use it. Joe, I think you're the same way.

Joe Marcantano: Yeah. This falls into the just no category for me. even if we ever got to a point where it could be more accurate, you. The fact remains, I am a big believer of it is not user research without the user. if you're not talking to an actual person who might get distracted by the dog barking or interrupted by a phone call or, like, have all these human things happen, then I really wouldn't trust any of that data.

Drew Freeman: Yeah. If we ever get to the point where I would be comfortable using synthetic data, we are having much more existential conversations around what is consciousness, what is a human? And my brain will just explode by the time we get there.

Joe Marcantano: Well, and if we ever get there, I think we're also having the conversations about, does do user researchers have a role anymore?

Drew Freeman: My answer to that question, which we don't need to get into because that's a wholeher topic, is there will always be a role, but that role will continue to change.

Joe Marcantano: Yeah, I agree. Drew, how do you deal with. Because it's becoming more and more common that, you know, PMs or designers or whoever are kind of starting to scrape these secondary sources on their own. They're kind of starting to do a little bit of that background research on their own.

Drew Freeman: Sure.

Joe Marcantano: How might you deal? Or how might you address. Let's say there's a, In this fictional world, there's a PM who, quote unquote, has done their own research. But that research you consisted of. They pulled support tickets about a specific thing, and then maybe they even talked to one or two of those users who reported that specific thing, and they're using that as the basis of this is the problem we need to solve. This is 100% a problem people are having. How might you address that with that Stakeholder.

Drew Freeman: I think the, the way that I would think about that is a twofold one. That is a valid. That is a valid opinion. That is a valid thought to have after doing the work that they did. So I don't want to discount it because I want my PMs to do, you know, to do that kind of legwork and help me figure out where that smoke is coming from. But what that PM probably doesn't know and can't know from this kind of research is why that's a problem or, you know, how much of a problem is it. That's the kind of thing where I think you need more of that primary research.

Joe Marcantano: Yeah. The thing I think about when I hear scenarios like this is, And we talked in a previous episode about kind of like picking your battles. Because on the one hand, there's a part of me that wants to say that's not rigorous research. We can't be building features based just off that. But on the other hand, if a PM or senior leadership has decided that is the direction, period, full stop, then it is no longer our role to convince them that that is not the direction. It is now our role to advise them in the way to make that direction as successful as possible.

Drew Freeman: Yeah, totally agree.

Joe Marcantano: So why don't we wrap up here. What kind of final parting thoughts would you leave for folks as we're thinking about these non traditional data sources?

Drew Freeman: Yeah, my lasting thought is that this kind of information can be very valuable when used in the right way. And when I say the right way, I mean your preliminary fact, you know, fact gathering exercise or opinion gathering exercise where you're trying to understand the problem space better, you're trying to understand the background, you're trying to understand the context.

Joe Marcantano: Yeah, I think my kind of closing thought here would be, you know, always remember that especially when you're talking

00:20:00

Joe Marcantano: about like social media, posts, subreddits, whatever. It really does tend to pull people from both extremes. The people who are super fans and the people who are super haters. So like, don't look at a subreddit and see that there's 20 people talking bad about your company or your product and think that that might mean that the company or the product is failing and that I need to jump ship and woe is me s. That's not necessarily the case. The overwhelming majority of people fall somewhere in the middle and they don't feel strongly enough one way or the other to post something. So just keep that in mind. Remember that even if a bunch of people are complaining about something, it does mean that it could be an issue, but it may not necessarily be an existential threat.

Drew Freeman: Yeah. I think the way that I summarize that idea is that again, it can tell us what, but it can't tell us how much, how many we should not use it, for that kind of purpose.

Joe Marcantano: Yep. Completely agree. This has been a really good talk and a really interesting topic that we've touched on here.

Drew Freeman: I'm glad you enjoyed it.

Joe Marcantano: Yeah. So thank you, everybody, for tuning in, for listening to this episode. If you like what you hear, give us a rate, a subscribe, a follow, tell a friend, help spread the word about inside uxr. And if you have a topic you want to hear us talk about, send that question in to inside uxrmail.com. there's a link in the show notes if you want to support the show. And with that, I'm Joe Marantano.

Drew Freeman: And I'm Drew Freeman.

Joe Marcantano: M. We'll see you next time.

00:21:41