36. How do I run a diary study?

Drew Freeman:  Hey, Joe. How are you doing today?

Joe Marcantano: I am well, Drew, how are you?

Drew Freeman: I am doing well. It is in that very strange time in the Midwest where it's not sure if it's winter, if it's spring, if it's first spring, second spring. It's that time of year.

Joe Marcantano: Yeah, you and I were talking about this earlier. I'm training for the London Marathon, and so I was planning on going out for my long run today, and then I texted you last night and said, well, we can record more than one episode because we just got three inches of snow, and I'm not running outside in the snow.

Drew Freeman: Yeah, I mean, I wouldn't run 15 miles like you were going to do anyway in any conditions, but I certainly wouldn't do it with three inches of snow.

Joe Marcantano: You know, it's, I think I've been doing distance running for a decade now, and I've kind of decided after this race, I'm gonna take a little break. But, it's definitely an acquired hobby. It's an acquired taste.

Drew Freeman: Yes, yes, absolutely. All right, so for today's episode, we are going to be talking about diary studies. So our question is, what are diary studies, and when should I use them?

Joe Marcantano: This is such a cool question. It is a study I've done a fair amount, and I think depending on kind of the timeline and where you're working, there are researchers out there who probably do two or three a year, and there are researchers who probably do one every five years.

Drew Freeman: Yeah, I'm definitely more on that one every couple of years kind of schedule. So I'm glad that you've got more experience here.

Joe Marcantano: Yeah, I think I did five or six of them. Not last year, but the year before. really was in an area that very heavily relied on them.

Drew Freeman: Okay, so let's start really basic. What is a diary study?

Joe Marcantano: So, like the name kind of implies, it is. It is a study that involves filling out a, you know, quote, diary or journal. So a good example might be, we want to learn how folks are going to use XYZ product, and we expect that as they become more proficient with the product or as they learn about the capabilities of the product, they might start to act and behave differently or wenna measure that onboarding how fast they're learning. Where they struggle. So what we'll do is we'll do a diary study. And so we'll give access to folks, whatever the feature or prototype or product is. And you may give them prompts every day over a certain number of days. You may just do more free form and say, hey, have at it, use it. But we want you to fill out every day, every other day, every three days, depending on your cadence. You know, answer these questions, tell me about what you did, why you did it. And so by the end of the diary study, you know, and this, a lot of it depends on how in depth, how long you're doing it. You have 5, 6, 10 entries per participant and you can kind of track how things change over time for those.

Drew Freeman: Participants you mentioned my favorite use case for diary studies there. So much of the time we are doing research on brand new users or we're doing research on people who have been using this product for a long time and are experienced. A diary study allows you to try to get a glimpse into that learning curve from a new user to an experienced user, which is something we don't do a lot of research on typically.

Joe Marcantano: I think that a lot of times the entire goal, especially when we're talking about like software based products, is we want something that someone can pick up and start using fairly proficiently right away. But there are products with a level of complexity that it just takes time to learn. So like, you know, you and I drew, have been driving cars for 20 plus years, but somebody who's 14, 15, 16 years old and is never driven, they are not going to go from novice driver to essentially professional driver in one day. Right? That is a week, months,

00:05:00

Joe Marcantano: years learning process. And in that learning curve, we want to see where they struggle, where they do really well, why they did well, and how can we take those lessons from where they did well and apply those lessons to where they struggled. And the learning curve was a little slower.

Drew Freeman: Absolutely. This might be a little bit of a tangent, but in my work in electronic medical record software, we very clearly had two different swimlane or two different Persona types when it came to experience. The first was a brand new user who's using this software on day one, week, one, month one. But then we would have users, we would have doctors, nurses, billers who would be using our software for years on end and would become incredibly efficient, incredibly good at, using the software. And we wanted to make sure that the software could scale with them and give them as much speed, as much flexibility as they needed to perform their job as well as they could. So we not only needed to think about the discoverability, we also needed to think about mastery. And while we didn't use diary studies to try to do both of those things, diary studies are kind of how you try to measure the middle state that in between state.

Joe Marcantano: Yeah, another really good. I'm glad you brought up like the person who's mastered the software. Right. Because another really good way to use a diary study is let's say there's a feature in your programming, in your medical software program, that only you have the analytics data. It only gets used once a week. And when you talk to folks, they say, oh, I use it here when this. But as you look at the analytics, it doesn't really line up. A diary study is a great way if you know they're using it once a week. Maybe you do a two week or a ten day diary study and you can look for things that might trigger that use. Because while participants don't lie to you, they misremember. Or maybe they lie to themselves. Anyone who's follows me on LinkedIn has seen me talk about this example a couple of times. It's the ask a participant how many times they go to the gym and then I tell you five days a week because that's my aspirational goal. But I got busy last week and I was sick the week before. And it's really like three and a half times a week on average. So when you're talking to somebody who you're talking to them about something they use less often, the risk of them misremembering or maybe just not being truthful with themselves is a little higher. And so by creating this framework where you're getting the reporting in the situation rather than in the hour that they've carved out and they're not really in that mindset can give you better answers.

Drew Freeman: 100%. Okay, so we've talked about, what a diary study is and we've danced a little bit around what it's good for, what kinds of questions it can answer. Let's hit that a little bit more explicitly. What kinds of questions, what kinds of research questions get brought up that lead you to think diary study is a good, a good method in this case.

Joe Marcantano: So one of the first ones is kind of one that we've both talked about and that's just how quickly can somebody move from novice to master? How good is our in house tutorials? Where do people stumble? Like, what is the journey like when the journey is more than an hour of learning this software or this product, what else? The other one that I really like is when you have kind of this, we'll call it a newer product or feature and you, you aren't sure how folks are going to use it. You're not sure you've designed it and built it and it's really cool. But m. Maybe it was designed without a problem in mind, looking to see how folks apply it as they become more familiar with it and as they learn it over those few days. Where do they see it fitting in? What problems do they see it solving?

Drew Freeman: This is a related use case to that, but I worked, with a company who was. They had what they thought was a really cool feature, but it wasn't super well known, it wasn't super popular. So they did a diary study and gave the participants tasks. And they were tasks like, I want you to look up information about this product, do a diary entry on that. Then I want you to do a diary entry about signing up online to be able to use it. Then I'm

00:10:00

Drew Freeman: goingna ask you to do another diary entry where you actually go into the physical location and use the product there. And then I'm going to ask you to do one last diary study that kind of walks through the whole process of discovery, exploration and use.

Joe Marcantano: Yeah, I love that. The last example I'm thinking of is where an in lab, in person or remote interview simply isn't sufficient. And so I'm thinking about a product that requires extensive use because you want to see like, I don't know, we were talking about running. You want to see how running shoes wear down over time. But I don't want to test it in the lab. I don't want them coming into the lab every day. Or I don't want to have a robot on a treadmill just doing the foot strikes every day because how they store the shoes might affect it, you know, if they take the laces and undo them every time versus just sliding their shoes out. So I need real world conditions over a prolonged period of time to get my insights or my findings.

Drew Freeman: So this makes me think of something else and leads to a different question which is kind of what is the difference between a diary study and longitudinal testing?

Joe Marcantano: So a peek behind the curtain for folks. Drew had such a great question that we actually had to do some googling to figure it out here. Drew, I'm seeing a couple of answers that both kind of fit in with my framework. why don't you talk about the answer you had in mind, which is one of the ones I saw and then I'll talk about what I'm seeing.

Drew Freeman: So the way that I typically think about it, and this very likely comes from my like collegiate educational background in statistics and economics, is that longitudinal testing tends to be more quantitative and can often be survey based. But it can also be. We are looking at. Again I'm going to go back to my economics educational background. We are going to look at the gross domestic product for a country across 10 years or we're going to look at the unemployment rate in the pre, the K through 12 education system for five years. We're going to look at childhood early childhood development achievement scores. Are those are actually real examples of papers that I wrote when I was in college. That's why they come up.

Joe Marcantano: So the other definition that I'm seeing is that a diary study is actually a type of longitudinal study and the other type is more of an omniannel study. So this might be something that requires multiple steps in multiple products or platforms. so maybe it's the example in this article that I'm seeing is they wanted to study cooking for whatever reason. And so step one might be looking on the Internet for a recipe. Step two is making the grocery list. Step three is going to the grocery store. And so all of those different things involve different products, different experiences, different modes of behavior. And that is another type of longitudinal study.

Drew Freeman: That's interesting. I wouldn't have thought of that as like a concrete difference.

Joe Marcantano: Me neither. But it totally makes sense because if you are trying to. If I'm building a grocery list app and I want to know how it fits in with how do you find your recipes and the step after, how do you go and shop for your groceries? It completely makes sense.

Drew Freeman: Totally. I just hadn't thought of it.

Joe Marcantano: Same.

Drew Freeman: Another you, you speaking about that did make me think of something else which is I typically think of diary studies as yes, they happen over a period of time, but that's typically still a relatively short period of time. We're talking days, weeks. Whereas when I think about longitudinal research I think more months, years.

Joe Marcantano: I agree. I think the longest diary study I've ever done was two weeks. They just simply are not typically done ah, on a longer timeframe than that.

Drew Freeman: It starts getting really expensive really fast. If you do that, it gets really expensive.

Joe Marcantano: It also gets really time consuming.

Drew Freeman: It also just beyond being really time consuming it gets really logistically challenging. Like any diary study that you run, you're already going to have people who start the process but Then drop out before they complete the end of the diary study. And the longer you stretch out that time, the more people you'renna have that drop in the middle for sure.

Joe Marcantano: Yeah.

Drew Freeman: Okay, so we've talked about

00:15:00

Drew Freeman: kind of what a diary study is and when it's generally helpful. Let's talk a little bit about the details of how you run one.

Joe Marcantano: This is actually pretty easy. if you are using an unmoderated software, odds are that most of them have kind of a diary study function. And so you can just program it in and run it that way. If you're not using an unmoderated software, and I've done it this way before, it works completely fine. Just create some Google Docs.

Joe Marcantano: For each participant, maybe you make a folder and there's day one, day two or entry one, entry two, docs in there, however you want to organize it. But you don't need to specifically pay for a software to run a diary study if you already have a Google subscription, a Google workspace, or even if you don't, I think you can get away with it with the free version of that. Just make a doc and share it with just the participant and make sure like the logistics there. Make sure you're sharing just Participant 1's document with Participant 1 and just Participant 2 with 2 and so on.

Drew Freeman: Agreed. It's totally doable with completely free tools and you don t need to, you don't need to pay for a tailor made diary study tool.

Joe Marcantano: There are also, it's rare these days, but if you are going to be meeting with this person in person and you want them to handwrite it, you can run up to Target and grab a couple of composition notebooks and you can give them an actual notebook to fill out. Give them a, an actual diary.

Drew Freeman: Going old school with it.

Joe Marcantano: Yeah, I have done one of those.

Drew Freeman: Okay, so let's jump into analysis, which I think is probably the most different for people.

Joe Marcantano: There are two ways to do analysis in my mind at least, and which one I do depends partly on the research question, but it also just depends on like how much time I have. And so the two ways are you can kind of analyze as the entries come in or you can do it all at once. Now even if you're doing it all at once, you should still be checking in periodically to make sure nobody's run into roadblocks, nobody's run into things that like cause them to drop out. But those are kind of the two ways, you know, do the analysis as you go or do it all at once.

Drew Freeman: So what are the pros and cons of each method?

Joe Marcantano: The pros of doing it as it comes in are that it becomes easier to kind of understand themes and how themes change over time. If I'm not going to get burnt, hopefully not going to get burnt out doing, let's say our sample size is 10 and we're going toa have five entries per person. That's 50 entries. That's a lot to analyze. But if I'm doing five entries every third day or so, that becomes a little more manageable just on my sanity. But by doing that, I also create a little bit of separation between entry one and entry two and it becomes easier for me to evaluate different themes across different days.

Drew Freeman: Right. So how do you try, if you decide to go that analyze it as it comes in approach, how do you try to balance that and do the best job that you can.

Joe Marcantano: So if I'm doing the analyze as it comes in approach, I try to treat each set of entries. So my day one entries, for example, across all my participants, as I'm analyzing them in my head, I'm thinking about those as a set of IDIs and I'm trying to pull themes, trying to pull, you know, shared insights or shared findings from that group on its own. And then I will take those themes and insights. And I've talked about this before, I'm old school. I have a whiteboard in my office. I'll draw a line down the whiteboard, I'll write entry one at the top and I'll write all my themes there and all of my insights there. And then that kind of gets just put in a box and set aside until I start looking at day two or entry two.

Drew Freeman: So then when do you ever take time, once you've gotten all the data in, to then kind of go back maybe question by question or topic by topic to try to evaluate how did things change from entry one to entry two to entry three?

Joe Marcantano: Absolutely. So by the end, by, you know, if we're doing five entries on entry five or sometimes I like to give it a day to just let my brain reset after I do the analysis for entry five, that's when I look across and if I'm doing

00:20:00

Joe Marcantano: the same questions, maybe I've got a ratings question and I want to see how that's changed. I'll look across the days, just question two across the days. Right. But if I don't have similar questions in my entry, maybe I just have themes and I just gave them an open Ended question. I'm analyzing those themes across days and looking for, as you've put it in the past, things that make me go, And that might cause me to say there was a big change in the sentiment between days three and four. I'm going to look at this participant, stay three and four, and I want to see what caused that change or what might have caused that change.

Drew Freeman: Yeah, I think that's absolutely the right way to approach kind of the. The balance of the analyzing the data as it comes in while also keeping the hole in mind. Okay, so let's talk about then the pros and cons of not analyzing the data as it comes in, but instead analyzing it all at once once you have all of the entries.

Joe Marcantano: So in my mind, there's a lot more cons to this way. The big pro is that it allows me to work on other stuff while this is running in the background. So maybe my diary study is running and I have another stakeholder who needs IDIs done. I can essentially knock out these two projects at the same time because the diary sty'running in the background. And all I'm doing is checking in periodically to make sure nobody's had a technical glitch or gotten derailed in any way.

Drew Freeman: I think another pro that I think about is that it potentially keeps you from forming conclusions too early and then falling into confirmation bias with the later entries and looking for places where your earlier thoughts are, have stayed the same or have changed or whatever. So it gives you a more holistic approach. But that's only potentially.

Joe Marcantano: I totally agree. And the. I guess the con to that is that, you know, when we're doing the analysis entry by entry, I can kind of have this if I'm doing it right and I'm not letting the previous days'bias me. I can kind of have this nuanced. People were feeling pretty good on entry one. It becomes more amorphous and conjoined. If you're trying to do it all at once, that's just like a human being thing. Right. It becomes hard to put those walls up between the entries.

Drew Freeman: Are there any other big pros or cons about analyzing all of the data at once that you want to call out?

Joe Marcantano: The other big thing that I would call out is the mental sanity burnout. Right. If I've got the example we used here, like were 50 entries. But I've done diarye studies that are bigger, and it can become really overwhelming to analyze that many data points at once, because it's not like analyzing 50 surveys. Where I just plug them into a calculator and I look at the data. This is qualitative research that I need to read each entry for. So that can become really overwhelming.

Drew Freeman: Like you said, each individual entry is.

Joe Marcantano: Essentially its own idi, even if it's aote short idi. Right. Like, because it's not a super long entry, it's just the number. It's looking at that mountain of entries that can feel overwhelming. So, like, especially if you're doing the all at once and analysis, take breaks, give yourself time to get outside, go for a walk, play with a dog, whatever. Get away from your desk every hour or so.

Drew Freeman: Okay, so wrapping this topic up, what are the most important things that you want people to understand and remember about diary studies?

Joe Marcantano: The things that are most important that I would tell folks to kind of take away and remember, number one, is that timeline wise, this is a longer study. Do not suggest this to your stakeholder who has to make a decision next week. Right. Like understand the time and place to use a diast study. And that's both the research question and the time that you have. And then the other piece is understand the amount of work that you're giving yourself. In our example, we had 50 entries that needed to be analyzed. Make sure that you build in the appropriate amount of time for analysis. We didn't even talk about if you were goingna do midpoint or follow up IDIs on these diary studies. This was kind of assuming it was just a diary study. So just be aware of the amount of work.

Drew Freeman: Yeah, we only really talked about a simple scenario.

Joe Marcantano: Yeah. If you were doing like, which I think is much more common, a multi method diary study with, you know, either midpoint

00:25:00

Joe Marcantano: or midpoint end after the fact IDIs, those are more data points that you need to look at. So just be aware of the amount of work that you're giving yourself and make sure you give yourself the appropriate amount of time.

Drew Freeman: Okay. Anything else that you want to leave us with?

Joe Marcantano: No, I think that covers it. I hope it was useful for folks. I really hope that especially for folks who don't do diary studies all that often, this was a good primer for them.

Drew Freeman: I. I have every confidence that it will be all right. So thank you for listening to the show today. We really appreciate it if you could give us a like or a subscribe or a review on whatever podcast platform you were hearing us on. That would be awesome. That really helps us be shown and be found by more, by more people. If you have questions that you want to hear us discuss and talk about. You can send those to us@inside uxrmail.com and if you'd like to support the show directly, there's a link in the show notes where you can do that. I'm Drew Freeman.

Joe Marcantano: And I'm Joe Marcantano.

Drew Freeman: and we'll see you next time.

00:26:11