33. How do I run card sorts and tree tests?

Drew Freeman: Hey, Joe. How are you doing today?

Joe Marcantano: I am well. How are you?

Drew Freeman: Here in the Madison area, it has gotten brutally cold the last couple of days. But on the bright side, we finally have some snow, so that's pretty.

Joe Marcantano: Yeah, we picked up some snow a couple of days ago, but it's mostly melted now.

Drew Freeman: Ours is still very much with us. It's like zero degrees today.

Joe Marcantano: O. that's the thing about the Midwest. You get snow, it hangs out for.

Drew Freeman: Three weeks, if not more. Yes. Okay, so jumping into our topic for today's episode, our question is, how do I run card sorts and tree tests?

Joe Marcantano: This is a really great question, and I'm glad that we've kind of got them intertwined here, because I was gonna.

Drew Freeman: Say I'm glad we're. These are two different methods, but I'm glad we're grouping them.

Joe Marcantano: Yeah, they're distinctly different, but they are related. And so it's good that we've kind of paired them together.

Drew Freeman: They're also really good partners. Like, when you're creating a study plan, should. If you're thinking about one, you should probably think about the other one, too.

Joe Marcantano: Yeah, I think that there's real value in making them kind of part one and part two, or phase one and phase two of your studies. they certainly can build on each other.

Drew Freeman: Okay, so let's start at the beginning. Ah. What is a card sort? What is a tree test?

Joe Marcantano: Yeah, so a card sort is a way to understand how your users or your customers group, content or information. And it is used best when you don't already have an established hierarchy, an established information architecture, or maybe you do and you know it's wrong and you're ready to abandon it. So it's best to think of it as kind of starting from a clean slate.

Drew Freeman: The only thing that I would add to that is that you can also kind of expand the way that you think about grouping to be. How do people prioritize information? A card sort can be really useful in prioritization as well.

Joe Marcantano: Yeah, exactly. a tree test, on the other hand, is when you already have an architecture in place and you want to test it, you, want to see how folks do in kind of predefined navigational structures.

Drew Freeman: Well, and the only challenge that I would give to that is that the architecture needs to exist, but it doesn't need to be in place and live. You can absolutely run a very valid and useful tree test on a proposed architecture.

Joe Marcantano: Yes, for sure. In fact, some of the best tree tests are often done. You can do them live, in person, but they're often done, using a third party software where you program in the architecture. You're not necessarily doing it on a prototype or even on the live site. You can, but it's actually better to do it in these tools because then they will record the exact path, they'll tell you how many times people clicked back or gave up or how much time they spent on the screens. So you're not working with like a full prototype here.

Drew Freeman: Yeah, those third party softwares or tools for tree testing can really give you a lot of valuable, a lot of valuable information that will be helpful in both data analysis, but also reporting.

Joe Marcantano: The one last thing I want to hit on kind of the basics of card sorts and tree testing and what they are and what they're for, is that there's also two different types of card sorts. There's a closed card sort and an open card sort. So if we're thinking about like categorizing or prioritizing, an open card sort does not have the prenamed categories, whereas a closed card sort does. So which one is right for you just kind of depends on where you are on the architecture you're building and what your other research has already told you and what you want to test.

Drew Freeman: And of course you can always kind of combine the two and make a

00:05:00

Drew Freeman: hybrid where you give some pre filled answers but also leave room for people to create their own groups or their own elements.

Joe Marcantano: Yes, for sure.

Drew Freeman: Okay, so I think, I think we should probably take a step back here. We've been talking about and using the word architecture and we should probably define that in this context. So what does architecture mean in this research context?

Joe Marcantano: It's the structure, the literal layout.

Drew Freeman: So the full phrase that gets used is information architecture or ia. So I think that helps for folks. It's the structure of how your information is laid out.

Joe Marcantano: A really simplified version of this might be if I want to save a new document or I want to save an existing document with a new name, I would first go to file, I would then go to save, and then I then would go to save as that's kind of this hierarch layout that we're kind of talking about in this case.

Drew Freeman: Yeah, That's a good example of information architecture. The way that I kind of think about it, it's like the, it's like the skeleton. Everything else kind of hangs off of that. Or maybe scaffolding is another good, another good metaphor. All of the rest of your website or your tool kind of hangs off of this information architecture. And the architecture is what, is what allows it to stand up, so to speak.

Joe Marcantano: Yeah. If you were going toa draw out the architecture, it might even look a little bit like a family tree or an org chart. It'snna be, hence the name tree top down. Yeah, exactly. And you'll be able to see where things fall under other things.

Drew Freeman: Exactly. Okay, so let's start with card sorting, both the open variety and the closed variety. When would you recommend that people really think about using card sorts? What questions, what research questions are they best at answering?

Joe Marcantano: So this is when you want to understand maybe from a point of zero or from a theory you're testing, how is, how are our users or our customers categorizing the information? Where do they see X piece of information falling within these categories? Whether it be a, ah, closed card sort where you're giving them the categories or an open one. You know, what groupings do they, do your users create in their mind? Which things do people think are related to one another closely enough to be in that same group? And then which things kind of stand out and go on their own and are separate?

Drew Freeman: Absolutely. The other place that I will recommend that folks think about card sorting is in that prioritization mode. And really you're just grouping things slightly differently there. Like your groupings might be, must see every time, need to see some of the time, rarely need to see, that is prioritization. But really you're just grouping still. Okay, so let's talk about maybe some of the limitations or some of the cons of card sorting as a method.

Joe Marcantano: Yep, there's a couple of, a couple of things. So card sorts are frequently, but not always done unmodated. So while that has the plus of being able to run a little quicker and you can run a little bit higher sample size, it does mean, as we talked about in our unmoderated episode, you are not there to ask follow up, questions, you're not there to ask why.

Drew Freeman: It's interesting you say that because the vast majority of my experience with card sorting is that they've been done with a moderator present to ask follow up questions.

Joe Marcantano: O that's really interesting.

Drew Freeman: Yeah, so we've got different experience with how we've used card sorting.

Joe Marcantano: Yeah. I could definitely see tremendous advantage to doing it that way. your trade off then though is.

Drew Freeman: Like how much time you have to spend.

Joe Marcantano: Yeah, yeah. likely reducing sample size. I would bet as well.

Drew Freeman: Yeah. Typically. I mean I almost think of the way that I've typically run card sorts Is like their IDIs and I'm just using a card sort to help me structure the interview is essentially how I've run card sorts in the past.

Joe Marcantano: That's super interesting. Yeah. I have never run them moderated. I've actually only ever run them unmodated.

Drew Freeman: And here's the beauty. There isn't a right

00:10:00

Drew Freeman: or a wrong answer.

Joe Marcantano: No. It's pros and cons and trade offs and what best fits the researcher trying for sure do.

Drew Freeman: Okay, so any other kind of limitations that maybe that aren't dealing with the live or unmoderated question?

Joe Marcantano: One thing that I would, that I would throw out like a caution flag on is whatever your items are that you're sorting or prioritizing or whatever it is you're organizing here. And this kind of goes for tree testing too. You need to make sure that your labels are intuitive and appropriate. you don't want folks not really understanding what an item is. You don't want them assuming it's something that it isn't. That can really kind of skew your results. So just make sure that you have very clear, very easy labels for things.

Drew Freeman: See, I actually think this is one of those areas where if you run a card sort with a live moderator, you need to worry a little bit less about that. Or like it's not as big of a pot, potential derailment point because the moderator can step in and say, oh, can you tell me, can you tell me more about why that is your interpretation or can you tell me what gave you that and why you're grouping it that way?

Joe Marcantano: Yeah, absolutely.

Drew Freeman: Again, it really just goes back to the pros and cons of live moderation versus unmoderated.

Joe Marcantano: Yeah. I remember one time I ran a card sort for a company that sold a health service. But their website was designed to serve several specific audiences. They wanted to serve employers who might by the service to offer to their employees, the patients and then the doctors. And they all had different informational needs and it was a matter of understanding how people, how do people group these needs?

Joe Marcantano: How do people expect it? And different audiences gave different answers, of course.

Drew Freeman: And that's something that you can get pretty easily from a card Sort where you could get it from doing IDIs with all three of those groups. But it would be more challenging to pull out those themes from IDIs. M okay, let's move to tree testing. What questions is a tree test really good at answering?

Joe Marcantano: A tree test is really good at answering. Do folks intuitively know where to go to find whatever piece of information? So let's use my health service website that I just talked about as an example. If my recruitment cohort was doctors, for example, or medical coders or somebody in the doctor's office and I told them, here's the architecture, where would you go to find help with billing codes?

Joe Marcantano: And now we can see that this person might say, oh, well, I'll go here where it says doctors. I'll go into this side where it says administrative help, and then into this menu where it says FAQ or whatever. It's really good at kind of testing existing or proposed architecture.

Drew Freeman: A buzzword or a, a keyword that makes me think that a tree test is an appropriate measure to think about is if someone says, how do people use the menus, can they use the menus or can people use our navigation? Those are kind of the two big buckets that I think about that a designer or a product manager might say that will get me to start thinking about running a tree test.

Joe Marcantano: Yeah, a really good place for a tree test is kind of testing that journey at a high level when you don't have a prototype yet. You know, if we want to test can people find specific pieces of information on the website? But I don't have a website or a prototype built yet. I don't have to wait for my designer to build something. I can, you know, in 30 minutes get a tree test set up on one of those software platforms and I can launch it to maybe 10 or 15 people and I can get directionally. Hey, these are the things that we may need to watch out for when we start doing IDIs with participants on a prototype or the live website.

00:15:00

Drew Freeman: Yeah, it's really important to emphasize that when participants are interacting with a tree test, they're essentially just seeing lists of words. That's all they're seeing. There's no images, there's no website. There's usually not even any like explanatory text. It is just kind of those navigation or menu headers. That's all they see.

Joe Marcantano: Yeah, if you were to think about like an org chart and you know, you were to. The question might be like, you know, where do I find X piece of Information, and they see the first four categories, and they might click on one category, and that opens up the menu below that category. Then they click on the next one, and they can go back if they think they're in the wrong spot, but it shows the layer that they are looking at. And then they can just continue to dig deeper until they think they have found the right spot.

Drew Freeman: Yeah, thinking about that. Branching is a really. That's where tree testing gets its name. you know, you start down a big limb and then you branch out into all of the smaller, more discrete pieces of information. All right, so what are some of the limitations of a tree test or what are some of the pitfalls of a tree test?

Joe Marcantano: I hesitate to say that it is done often unmoderated, because I feel like you're going to come back and say that you've run them all moderated.

Drew Freeman: But no, tree testing is one that I would definitely say unmoderated is the kind of default.

Joe Marcantano: Yeah. So you're going to fall at the mercy of your participants and, like, you'll give instructions. You'll say, hey, explain what you're thinking, what you're looking for. tell me where you think things would be, why it might not be here, whatever. But you're relying on the participant then to kind of actually do that and do it in a way that you get and you understand.

Drew Freeman: And you're also relying on yourself to make good. To make a good script, essentially. All of the things that we talked about that are extra important when you're running on moderated testing, go back and listen to that episode. They all apply here.

Joe Marcantano: The other thing is, you know, I keep using the website example. We're testing a website, Right. A tree test only has the text. It only has the text headers. That means that your participants are not necessarily going to get the other contextual clues from the website, which could help them on the right path or take them down wrong paths, depending on what those pictures and videos and animations and whatever are on your website. This is just testing the architecture in a vacuum.

Drew Freeman: Yeah, that's a really important point. You have to remember what you are testing and what you are not. Okay, Joe, So the next kind of thing that I want us to talk about will not be surprising to anyone who's listened to our episodes before. It's something that we talk about all the time. But how should people think about card sorts and tree tests when it comes to running them solo or fitting into a, I don't know, a program or a bouquet of Various testing types.

Joe Marcantano: The thing that I would stress is that very rarely if ever is the card sort or tree test the last test you will do.

Joe Marcantano: It will help you understand mental models for grouping and test the navigation. But on a super high level, at some point you do need to talk to somebody who is in front of and using the thing, whether that be the prototype or the website or whatever it is like we talked about in the previous example here. You know, if there's a picture that's very distracting or misleading, you're not going to know that. You should run tests after this to ensure that as you get closer to that higher fidelity finished product, it continues to perform in the way that you expect it will.

Drew Freeman: It's a really good point. both a card sort and a tree test can be good early or mid kind of low fidelity validations that were headed down the right path. But you should, I hesitate to say always, but you should certainly consider running. Maybe it's a usability test to validate that

00:20:00

Drew Freeman: people can accomplish the tasks that we envision them using our website or our tool for. That's a question that a card sort or a tree test simply can't answer.

Joe Marcantano: Yeah, and something like a card sort or a tree test can be really good for, you know, if you've already got a launched product or website and you're not seeing conversion or sales or whatever like you expected, this might be a really good first stop to say, okay, have we organized it in a way that is intuitive for folks? Do we need to, Is the architecture the problem or is the copy on the website a problem? And so it can be a really quick, easy way to either shine the spotline on it and say, yes, this is the problem or you know, maybe your tree trst results come back and everybody completes things pretty easily and they find things pretty easily now, you know, okay, likely not an architecture problem. Let's start doing some other testing. And because especially tree testing is done unmodated so often, you know, if it is a simpler tree test, I can get it up and running and launched in a day and have very basic high level results depending on your recruitment cohort in 24 hours, 48 hours, where I can just say thumbs up or thumbs down, people got it or they didn't and then I could do analysis and kind of get into the minutiae.

Drew Freeman: Yeah, absolutely.

Joe Marcantano: Drew, I want to ask you because my experience is doing both of these unmoderated with kind of specially designed softwareah. I'm curious just the logistics of running a card sort sitting there with the person or maybe remotely, you know, how did you do that? What software suites or, you know, physical items did you use?

Drew Freeman: Yeah, so if I was doing this live, in person, physically with the participant, I would use laminated cards. I say laminated because we were often running these as part of a big, almost promotional kind of user testing experience. So we were running, you know, potentially these studies and these cards could get used 10 times, 50 times, 80 times. So lamination was important there. But if we were doing it, if we were doing it remotely, what worked really well for me in the past was making, making my cards in a slide deck software like Google Slides or PowerPoint. And what was really helpful about that is at the end of each session you could just click File, save as and you could say P1 Emily, P2 Joe. And then you could go back and do the analysis and see exactly how each individual grouped things. So it was almost like one, it offered the cards and you could use that to make the cards. But two, it then became a note taking tool as well.

Joe Marcantano: I love that. Now when you were running these live, was this one on one, were you doing it kind of focus group style or maybe dyads, triads, how are you? What was the interaction kind of ratio there?

Drew Freeman: My experience is that this was 90% one on one. And the only time that it was not, it was we wanted it to be one on one. But like I said, because this was happening at a, essentially a user conference type of event, there might be coworkers who came together and we couldn't talk them into doing it individually or maybe it was the customer and then their account manager who was just kind of sitting back and observing. But I would recommend doing any, like any idi, I would recommend doing this one on one whenever possible.

Joe Marcantano: Yeah, I could see a lot of things that could happen if you were doing this in a group setting that would, I don't want to say invalidate, but make your results less reliable.

Drew Freeman: I mean the same things that happen, in a focus group. It just makes it harder to do analysis because you get people who start to, you know, you start to get that groupink or you start to get one person who dominates the conversation and pushes the other person to agree with them, that kind of thing. Yeah, I would say one of the biggest benefit in my mind to running a card sort moderated versus unmoderated is that you can ask those follow up questions and you can ask, why did you group that that way? Why did you Group this one with these cards,

00:25:00

Drew Freeman: why did you decide it didn't belong in this group? Being able to ask that I think is really valuable. I think it's especially valuable if you are doing either an open or a hybrid card sort where you allow participants to create their own groups or their own cards, because then you can get that explanation for why they feel like they need a new card rather than using one of the existing ones that you gave them.

Joe Marcantano: Yeah, the other big benefit I see to doing this either online live or with the person live is this is, you know, being a participant in usability study is already a little bit awkward and a little bit weird. But this is also kind of a, a more specific task that might feel even more awkward. especially like if your instructions aren't clear and maybe you're meaning to run a hybrid card sort. But if your instructions aren't clear, they don't know that they can create new characters. So they don't understand how to do it on the computer. so being live and being there to explain it to them and kind of remind folks of that, seems to me like a real advantage.

Drew Freeman: Yeah, that's a really good point. Running a card sort or participating in a card sort is certainly less common than a remote interview or a think aloud usability test. So it can take participants longer to kind of get into it and get into the full swing of things. And honestly, that kind of not rapport building, but using those early parts of your guide and your discussion when you're live to ease the participant into it and really understand. Okay, I can't quite push them into this yet. We need to ease in a little bit more. That's really valuable. It doesn't mean that running them live is always the right answer though.

Joe Marcantano: Yeah, if your time allowed, you could even have kind of an example card sort, and talk through it with them. Maybe you've got cards with Marvel superheroes on them and you say, I'm gonna put Captain America and Iron man together. and then I'm going toa put Thor over here, whatever's on your cards, but you could literally do one in front of them and be like. And that's how this is done for sure.

Drew Freeman: That would be an excellent way to not only build rapport, but to also get them in the mindset that you will need them to be in. That's a great idea. All right, as we wrap up here, what do you want to leave our listeners with when it comes to card sorts and to tree tests?

Joe Marcantano: The thing I would leave for researchers. Here is, you know, as we kind of, you and I discovered as we were talking about this, there's no one solid right way to do this. And if you are somebody who is more junior in your career and you've never done one of these, don't sweat it. First of all, know as you kind of do them, they are not particularly difficult on the researcher side to do. They are not overly complex. It's also, as you kind of said, Drew, it's not something that's super commonly done. I would wager that even a lot of senior and lead researchers have probably only ever done it a handful of times. I, was going to say 10. Yeah. In their entire career. This is not something. It's not like the one on one IDI that you need to have in your back pocket and ready to go and say, I know how to do these. If this ever comes up, you're going to have time to Google some articles or pull up your favorite podcast and kind of refresh yourself.

Drew Freeman: It is a good tool to have available to you, but it's not one that you'll be using every day or every project.

Joe Marcantano: Totally.

Drew Freeman: I think that's a really good place to leave our conversation. I'm super happy that we learned mid episode that we had both run card sorts in very different ways. That's always fun. And I want to thank everyone for listening to this episode today. Hopefully you learned something new and maybe are excited to try to find, a place where using a card sort or a tree test is appropriate for you in an upcoming research project that you have. Some ways that you can really help us out as a podcast is to give us a like or a subscribe or a review on whatever podcast platform you use. Something that would be really helpful is for you to send us the questions that you have and that you want to hear us talk about. Whether that is asking about a particular UX method or, you know, maybe you want to get slightly more personal when

00:30:00

Drew Freeman: it comes to that. We're not opposed to that either. You can send us those Questions at inside uxrmail.com. with that, I'll wish everyone a good day. And I'm Drew Freeman.

Joe Marcantano: I'm Joe Marantano, and we'll see you next time.

00:30:15