Speaker 1: Hey there product lovers, welcome to the Product Love Podcast, hosted by Eric Boduch, co- founder and chief evangelist of Pendo, and super fan of all things product. Product Love is the place for real insights into the world of crafting products as Eric interviews founders, product leaders, venture capitalists, authors and more. So let's dive in now with today's Product Love Podcast.
Eric Boduch: Welcome lovers of product. Today, I am here with Kristen Berman, who's the co- founder and CEO of Irrational Labs. Welcome, Kristen.
Speaker 1: Great to be here.
Eric Boduch: Why don't we kick this off with a little overview of your background?
Kristen Berman: For sure. I have a product management background. I started out of college at Intuit as a product manager for QuickBooks Online and Quicken, and then transitioned into behavioral science close to 10 years ago or a decade ago with a collaboration with Dan Ariely, who's the famed behavioral economist who wrote Predictably Irrational, and have since been using behavioral insights to help product managers and marketers design and grow their products.
Eric Boduch: That's awesome. That's awesome. Tell me about your time at Intuit. Jumping into product right out of school I think is generally unusual, or at least it used to be, so tell me how you got into product and obviously Intuit, especially back then, was well- known as a place to grow product leaders.
Kristen Berman: Yeah. I think Intuit actually took a different perspective than at Google, who was purely computer science based hiring for PMs, and they took the customer- driven approach, which is, " Look, we should put smart people who care about the customer in product positions." They actually did something quite clever, in the moment it was terrible, but in hindsight was lovely, where they put us in a call center for three months and we took QuickBooks Online support calls in order to get to know the product. And so they did have a philosophy of deep diving into customer empathy that I think helped onboard new PMs without the CS experience. My general, though, transition from product to behavioral science was interesting. I was leading customer insights for QuickBooks Online and was hitting my head against the wall just trying to design first use. So QuickBooks is hard to use. It's still hard to use if anyone's out there using it. Sorry about that. And it was tough to figure out what to do. And I noticed that as much as we would ask customers what they wanted, how do you design an accounting product for small businesses who have no idea what accounting is? And then I heard Dan Ariely speak, and it was like, " There is a science to human behavior." At that point, he had just released Predictably Irrational, and continued to, in my Intuit career, basically get really confronted with how PMs could benefit from the science of decision making. I was transitioned then to be a PM for Quicken Online. This was prior to when... before Intuit bought Mint, there was a Quicken Online. It was a free product. They tried to give it away. And we were developing a new feature. And I think this was the hardest as a young career person that I'd worked ever, trying to figure out how to design and build a product to compete with Mint. And we brought people into the lab. At that point Intuit had a really, really lovely user experience lab where you could watch people in the product, and the whole PM and design team were watching as we saw people for the first time use this product or use this new feature. And if you know personal finance, you know you have to sync your bank account, and a bank account, you see your money. And so people would do this-
Eric Boduch: Yeah, yeah.
Kristen Berman: ...Yeah. And so people would do this and they would just look, and this is our new feature that we just slaved over, and they would cover their eyes. And our PM team was behind the wall being like, " Oh God, what have we done? What wasted effort if people..." And we just kept watching people, time and again, come in and just look at their finances and cover their eyes. In hindsight, as a behavioral scientist, you know exactly what's going on. As a PM and design team at that point, we didn't. So what was going on is basically you're showing people all the money that they're losing in a very clear visual way. You basically give someone a pie chart of how much they've spent, and this is almost the definition of loss aversion. This is the exact definition of loss aversion, you're showing me exactly how much I'm losing, there's an emotional effect and we don't like it. And so, if we had known that as a PM and design team about loss aversion, information aversion, we inaudible very clearly not built that feature that shows you exactly how much you're losing. I think if you reflect on FinTech products now, there's low engagement. I would philosophize that low engagement is due to this really strong, emotional reaction to our money that we haven't figured out how to help people with.
Eric Boduch: So would you have done it differently, or would you have not built that feature at all?
Kristen Berman: Yeah. My reflection in general on FinTech, and I have spent a fair amount of time working with research with companies like Chime and Digit and gamut of the fun challenger banks, is it generally, we have this idea that if you just give people information about their money, that they will make a different decision. If I just tell you how much you're spending in a very technical and accurate way, then I can empower you. And by the way, you know about your money, so if you know about your money, you should know how to spend less. And I think from behavioral science perspective, this is just incorrect. There's actually no findings that show budgeting works. And so from a behavioral science perspective, we'd likely show people a few things: we'd show them their income. This is very exciting. We once worked with the company and they sent emails that said, " Look, you got paid. Guess how much?" They opened right there, it was over 80%. Amazing! Just tell people that they get paid. And I'm not saying that this is the only feature, but it helps. We'd give people social coordination about what other people are doing, not because you want to be better than them, just because we don't talk about our finances and you really have no idea what others are doing. And so the norm is to spend money. It's a visible norm. Can the norm be to save money? In order for that to happen, we have to help people understand that others are saving money. So these are basic behavioral science principles. And I think from a PM or a marketing perspective, we take a more rational approach, at least a FinTech, versus the irrational one.
Eric Boduch: Awesome. Awesome. That's really interesting. Maybe we should step back a little bit and talk about what is behavioral science?
Kristen Berman: Great question. Behavioral science is basically an approach to decision making. It's in classic reaction to economics, which is basically humans are pretty rational and we react to the supply and demand curve. If I was waiting in line for a cup of coffee, I should figure out if that$ 3 I'm about to spend is worth it, and I would do things like compare it to if I should be spending$ 3 at lunch, or if I should be spending$ 3 in investing in the market. That we should be looking at most decisions in a rational way, thinking about the opportunity cost of our time, behavioral economics is the reaction to that. It says, " Look, that's just not how humans behave. We look at other people for cues is what to do. We're loss averse." We make decisions based on a bunch of heuristics and the behavioral economics field has studied and categorized those. And so behavioral science and behavioral design, which is basically the application of behavioral economics. And what I do is apply these insights from psychology to product decisions to build products that hopefully change behavior for good.
Eric Boduch: Yeah. Talk to me about this journey from product management to behavioral science from Intuit to where you are today at Irrational Labs.
Kristen Berman: Yeah. A little bit of luck, but meeting Dan was great. I think if you want to get into behavioral economics, typically the path is a PhD. Typically, PhDs don't go back in the industry. That's just not the path. And so by meeting Dan, I got to... He's an academic who wants to apply the work, and Google hired him, and consequently me, to start their behavioral economics group. So this was around 2012, 2013, and we were as close to internal as you can get at Google, where we had a bus pass and we would go, and we helped over 25 teams at Google think about how you design products that change behavior for good. And so I think they were one of the leaders, and still are, in using behavioral science to change behaviors. I got a lot of at bats working with product teams and marketing teams to try to see what worked and how, and take basically the theory and put it into product decisions. The one that we can talk about, obviously not a lot we can talk about from examples, is one that we did with Google AdWords. Google AdWords is, like most ad products, a very, very difficult product to you use. If you try Facebook Ads, it's also difficult. So no harm to AdWords. The problem with difficult products is that people are easily... they're going to give up. The idea that you basically have to try really hard to learn something means that you're going to lose a lot of customers. And these ad products, they have a low retention rate because of that. And so the question is, how would you think about changing the mental model of a product so that people would push through onboarding, that they would just keep going? And the realization here is that most products are framed like a one and done; that you immediately get a benefit from it; that if you do something, you should to immediately get a benefit. That's just not how Google AdWords is. And so our team, we took over the call center and said, " What if AdWords reps actually said, 'It's not,'Welcome to AdWords. How can I help you?' immediately." It's, " Welcome to AdWords. Let's put you on a three- month AdWords expert program and we'll spend 90 days. We're going to put something on your calendar. It's not about figuring out right away, it's about testing keywords, it's about trying new things." And this increased retention by 14%. And so these are the types of things behavioral science can do, where you basically look at a problem and you're trying to apply some psychology insight, and then you test it and you see what happens.
Eric Boduch: There's a lot of threads I want to go down here, but we had Dan Ariely as a guest speaker at one of the pandemonium events, right?
Kristen Berman: Nice.
Eric Boduch: So amazing, amazing talk. Amazing guy. What was it like to work with Dan?
Kristen Berman: Dan's incredible. I think if listeners have not dove deep into Dan Ariely, I would suggest going down that rabbit hole. So many things. I'll give you one personal example, and this is a little bit revealing, but shows his character and his creativity. So he actually once asked him, " What do you think you're best at, Dan?" And he's like, " I'm really good at the science, but I'm really creative." And I think that's the beauty of maybe a lot of professions, is you can be really good at what you do, but there's some art to it as well. Look, I started Intuit, I was young. Before I started behavioral science, I smoked one cigarette a day as I drove from Intuit to San Francisco. The drive, for listeners who don't know, it's like an hour drive, gets longer. And he's like, " Look, we can't work together if you're going to smoke. This is really terrible." I said, " It's an addiction. I'll try. I'll do my best. I'm not..." He said, " Okay, here's the deal. Let's split a car. Let's figure out what car we want. We'll split a car. The deal is you can't smoke in it." And so think about this now, you have somebody who's working with Dan Ariely and he's paying for half the car, " Am I really going to smoke a cigarette now on the way back from work?" Think, " Every time I would do that, I'd have to think of Dan? Obviously not." And so that was the last cigarette I had. And so I think he looks at solutions from a fairly creative perspective and also really nails some of the behavioral science juice, which is some core motivations of human behavior.
Eric Boduch: Interesting.
Kristen Berman: By the way, I'll give you one more. The first time I was on the phone with him, I was like, " Okay, I'm ready. I've done a lot of... I was new to the field. They said,'Okay, let's get started.' And I thought we were going to talk about the strategy of the project we were working on. And for an hour and a half, we copyedited something. And I was like, "When are we going to get to the bigger strategy?" He's like, " Look, life is about the details. Behavior change is about the details." And so I think that's a general appreciation that behavioral scientists have, is the details matter. And Dan can go really high level but also really cares about those details.
Eric Boduch: Yeah. No, absolutely. We're talking about this interconnection, you're doing a lot of time and product with psychology and economics. Talk to me why it's important to connect those three things together.
Kristen Berman: Yeah. I think the question in itself is a funny one, because basically, product managers have to understand their customer. And if you think about what understanding a customer is, it's really about understanding the psychology of the customer, what actually drives them. So I think this is deeply embedded in the product management core, is figuring out what drives your customer. And the reality is, there's just a lot of research area. Any time you start a new feature, you should be spending a day. If you can't spend a day, spend an hour looking up what other research has been done about what drives your customer. I actually was just talking to a team yesterday. They said, " We're going to do financial education for kids." I said, " That sounds very nice. Have you thought about, does financial education for kids work?" " Yes, it does? We think it does. We've talked to parents. They feel like they need to educate their kids. There's this obviously low financial literacy in the U. S." Financial education for kids does not work. Across meta- analysis and meta- analysis, there's been no consistent result. Now, of course they could try hard and figure it out. I'm not saying that teams aren't smart, but the likelihood that they would do it is very low. And so I think we have to think about what we know about psychology and then also try to figure out and be humble about what we don't know. I think this is really what behavioral science brings to the discipline of product management, is an appreciation for how complex humans are as well as a rigor to figuring out what works.
Eric Boduch: Yeah. Do you think PMs do a good job of this?
Kristen Berman: Obviously, there are PMs who do a good job at many things. I think by and large, the field could do better. So I think the idea of what drives somebody is lacking. We haven't met a PM or a designer who does a lit review before they start something. I wish that was part of the process. Part of our behavioral design process is an extremely rigorous analysis of every step that somebody takes to get to the behavior that you want to drive. We've seen journey maps. We've never seen a behavioral diagnosis like how we would do one.
Eric Boduch: Yeah. Going back to something you mentioned earlier, people that have degrees in behavioral science tend to not go back into industry. Why is that? And should we as product leaders recruit more of those people onto our teams?
Kristen Berman: I definitely think you should recruit more behavioral scientists onto teams. I'm conflicted on if you should take PhDs and pick them out of academic work. I think self- selection's real. If you're going to get a PhD, you may not have self- selected to be the action- oriented executor, and at least in the Valley it's required. Our team, we tend not to hire PhDs because of that. There's just a self- selection difference, so we train people who've had experience doing field experiments, RCTs and testing just doesn't have to come from a PhD. So I do think more PMs should look at behavioral scientists. The other approach, by the way, is to just become a behavioral PM. The idea that you could actually really look at yourself like you're a behavioral scientist. We started a bootcamp to help PM's, marketers and designers skill up on this. And I think in one world, it'd be perfect if we had more headcount for behavioral scientists. I was told by one HR manager, he said, " It's much easier just to change your title." And so getting new head count is great. And that if you can do that, great. And if you can't, I would scale up.
Eric Boduch: Yeah. I was actually even thinking about just giving him a product management title and just saying, " Hey, but you're going to be a cross team and you're going to look at behavioral issues with our customers and how we can make our products more engaging at the customer base," because that's a major part of what a product manager should be doing.
Kristen Berman: Totally. To your credit, this is what Pendo does, it's trying to help people understand what people actually do. I think it's helpful to understand what they do. And then the question is, how do you get them to either do more of it, do less of it, do it at the right time? And this is, I think, where the art and the science come in and behavioral scientists really offer a nice lens to the actually secret sauce of getting someone to be an engaged user.
Eric Boduch: Absolutely. Let's go back to Irrational a little bit. Tell me more about Irrational, tell me about the company, tell me why you started the company. And then I would love to hear about some of the interesting projects you've worked on. I know I threw a lot at you right there, but-
Kristen Berman: Yeah. I'll take all five questions at once. Irrational Labs, we are a behavioral science consulting and design group. We're pretty small and nimble. We hire, I would say, world's best behavioral scientists. I'm obviously biased, but we were really good at helping PMs and marketers assess what's going on and then designing interventions that change behavior. And so we usually work with the company from the diagnosis, the design and the data analysis piece. I'll give you a few examples so folks know what I'm talking about here. We worked at the company called Steady. I'll go back to the FinTech area as well, just to keep it consistent. But we worked at the company called Steady. They are basically helping gig economy workers find jobs, and they're actually doing something I mentioned before, which is showing you your income. It's one of the reasons we wanted to work with them; to design the future. And so you have to do something in all these FinTech apps I mentioned before as well, which is sync your bank account. It's extremely difficult to do. You have to figure out your... Extremely difficult in the behavioral science world, by the way. We once removed one field from a conversion flow and got a 40% a page over page increase in conversion. When you have somebody do something extra, we should assume the drop off may increase. Obviously, there are times when that's incorrect, but for the most part, a rule of thumb is we do things that are easy. And if a sign of flow is hard, we do it less. And so Steady had the same problem that any kind of onboarding flow that's hard has, is people were dropping off. And so what we did is basically said, " Look, what's the psychology that would drive someone to continue?" And really, there's a high cost and zero benefit. There's a high cost and a future benefit at least, to actually doing something. And so we said, " What could be an immediate benefit?" And I think this is generally an approach... We have something in Irrational Labs called the three Bs, where it's we identify the key behavior, we reduce barriers and we increase the immediate benefit. This is a simplified view of a human, but this tends to work at least for our experiments. And so for this case, we basically said, " Let's increase the immediate benefit of what they're doing." And so we did a couple. We tried a few things, both of which tended to work, which was, we basically just added a forced choice. We said, " Instead of saying,'Maybe later,' we'll say,'Accept or decline.'" A very, very simple change, you could say yes or no, but we actually made people say, " Decline." What a hard choice, you actually have to say no to something! And so by adding the decline, I think it was 63% we increased the amount of people who went through with the bank account sync. You're basically saying, " Look, if you really don't want this, great, but if you do, now's the time." The other condition we did, which worked even better, was something we called the goal grading effect, which is you basically want to make people feel like they're close to something, feel like they're actually about to get to it. And we said, " Look, complete your flow." We made them feel a little bit like there was an error message. And this worked even better than the prior one, which was the forced choice. These are very, very simple, and I'm not saying revolutionary ideas, but what we're basically doing is applying an insight about how you get somebody over some barrier to complete something that they've already said they want to do. I think that's key. We don't want to be manipulating people to do things they don't want to do. But look, they're halfway through the flow, they're giving up with the bank account details, let's get them over that hump by giving them some small motivation so that they can see their income a little bit faster. So long- winded way to basically say they have irrationalized, we look at these problems and we're designing these interventions to drive either conversion or basic product features.
Eric Boduch: I like that. What else can you talk about? Give me some more projects. I love hearing about results.
Kristen Berman: Yeah. Maybe I'll give you a few. We worked at the company called Livongo and they do... now more than diabetes management, but they offer a diabetes management program, and they were basically having a problem that a lot of companies have, which is you want to get users to use your product. You have to actually get them to start. It's a cold start problem. And so for Livongo, they were actually wanting to ship people a free diabetes kit. This is for people who are already identified that were either at risk or about to be at risk or needed monitoring, and that are extremely low... At this point, this is early in their career. So I'm not saying early in their life cycle. They had low uptake or lower than they wanted. And in their emails, and this is basically in a healthcare company, in a perfect world, we have more touch points than just email. But the reality is many times you're regulated to email as a way to get people to use a product. And so they basically said, " Look, we've got this kit. Get the kit." And that was their control email. We said, " Look, that's interesting, but why?" And we could have gone off and explained everything that was in the kit. The thing that we decided to do was basically a little bit different. We said, " Claim. Claim your Livongo kit. This is yours to lose." And we didn't say lose, " But this is yours." So we changed the mental model. Again, this goes back to how people are viewing your product. These are bigger than just saying, " Get to claim." This is about an insight about how people view the benefit that you're giving them. And imagine your healthcare company says, " Look, you've already paid for something," which is pretty true in our insurance world today; we're paying premiums and they give us benefits, " you've already paid for something. If you don't use it, you're wasting your money." And this is basically what this email looked like to folks. We increase the enrollment rate by 120% by just making this small change in the onboarding flow. I think basically onboarding in general is actually really a really lovely place to start. Many times we have people focused on retention at companies and they're a little bit later in the flow than they should be because you're not going to fix a retention problem after somebody's been using your product incorrectly for six months. We need to figure out in the very beginning of onboarding how we change the mental model and drive basically people to understand what the value is of the product. I'll give you one more example if you're okay.
Eric Boduch: Yeah. I'm intrigued.
Kristen Berman: Great. We worked at a bank, this is actually with Common Cents Lab. The study example was also with Common Cents Lab. Common Cents Lab is a lab I started with Dan out of Duke University. It's focused just on financial decision making for low to moderate income Americans and it's grant funded by MetLife Foundation and BlackRock philanthropy. And so we get to do all this fund research a lot because of the grant funding. And so this study was actually with a bank. And the question really was, look, someone's getting a car loan out. What is the number one reason that people default on their car loan?
Eric Boduch: I have no idea.
Kristen Berman: Yeah. I-
Eric Boduch: I'd assume they get a car they can't pay for.
Kristen Berman: ...So this is a problem. People are overbuying cars. And this is a growing problem now that used cars especially are increasing in price. In general, what it is is that if your car actually breaks down and you're actually on the border, let's say that you're paycheck to paycheck, your car breaks down, you're not going to be able to go to your job. And so you're losing money. So now you're paycheck to paycheck, but you don't have as much money as you did because either you're taking less hours or something bad happened. And so you have these emergency time events that happen. People aren't bad people, they're not actually trying not to pay off their car loan, there's just things that happen in life that make it so that you may miss a payment every now and again. In a perfect role, people would have a savings account built up for this. Like an average car loan, maybe a thousand dollars or something. It's a lot for some folks, but it's not an unachievable amount to build up and have in a savings account. And so we worked with them and said, " Look, how do we help people actually build up this savings account?" And in one world we could basically just say, " Okay, great. Open a savings account and put some money into it." This is difficult as you have that lump sum money right there, and it's just hard for people to understand that in the future, they may need it. And so instead, what we said is, " Look, can you just round up your loan payment?" You're already making a loan payment. And by the way, when people make loan payments, it's not like they think about their loan payment like$ 282, they're thinking about their loan payment like$ 300. We do mental math and we just round up. So it's like, " Look, let's just put that extra money that you're already mentally accounting for into a savings account every month. And by the way, at the end, when you do that, you're going to have around a thousand dollars left over. Do you want to do this?" And so by asking that question upfront in the onboarding, around 38% of people said yes, and at the end, they had... forgetting the numbers here, but around close to a thousand dollars in savings. We also did another experiment that asked people to round up and around 10% of people opted in. The difference between those two experiments was just when we asked the question. It was a different company too, so we could say that there's fair amount of variables there, but in the former one or the latter one, we basically asked the question after they had been already using this... at this point, it was a mortgage loan account. They'd already been using it for a while, and that had car loan account. We'd ask them to round up. 10% of people said, " Yes, it's actually really lovely, but it's not what you'll get if you focus on onboarding. If you focus on onboarding, you're basically helping people set the mental model of what's expected. And I think from a behavioral science perspective, we would get way more people to focus on onboarding and add more questions, even though you'll probably have more friction, to help people set the mental model of how to use the product in the future."
Eric Boduch: Yeah. I'm curious too, I don't know if you have any insights into this, but just triggered a thought because I've always heard that there's a lot of people out there, especially with certain personality types, that no matter their income, they'll spend to their income level. So they could be making$ 20,000 a year or$200, 000 a year, but they're likely to spend to that level so that they do have the issues that you just talked about. Like something goes wrong and there's not that savings associated with it. Is that a human characteristic and a good chunk of people that they're likely spend based upon what they have? They see income as a number they should or can spend to every month? Is that something you typically see as how humans approach income and money?
Kristen Berman: Yeah. There's a percentage of people... Basically, if you have money in your checking account, it acts as a target, " That this is how much..." By the way, this is also why I don't love that credit cards always give you your credit limit. If you call a credit card company, they'll crosstalk mention your credit card limit. It's like, " Oh, that's my target." That's not your target. And so we have these tendencies to say, " Look, I'm getting paid. I can spend all the way down to a certain amount." And usually, we don't spend to zero, we have an aversion to zero. We leave a little bit in there. And each person's a little bit to different, but they leave a little bit in there. If we're redesigning the budgeting process, we'd say, " This is..." We don't need to actually count and add things up, we just need to, at the point you get paid, put a little bit of money into savings automatically, maybe put a little bit of money... there's now companies popping up to, you put money into your rent account, you put money into your disposable... like your food card or your daily expenses card. We don't want to make it too complex, but really this is the essence of set and forget budgeting. We're just saying, " Look, you can spend everything on your card. It's okay if you go all the way down, because you've got these other cushions." I think the idea that we would help make people calorie count, which is actually what a budget is; is like you're asking people to do a lot of work, a lot of mental math, is a little ridiculous. It's like we've sent a man to the moon and we can't help them budget. We're asking the consumer to do the budgeting. I think in general, the behavioral science lens on most things is to make it wildly easier than it is for people. And if it's not easy, we'll do the easy thing, which is you look at a number, you make one heuristic and say, " Can I spend it or can I not spend it? Do I have enough money in my account or not?"
Eric Boduch: One of the things too as we're talking about human behavior that was triggered in my head was this whole concept of misinformation, which obviously is on a lot of people's minds today. I know you guys had a blog post about it a little while ago. Talk to me about misinformation, why we're susceptible to it and how we fight it.
Kristen Berman: Yeah. Hard questions. In general, I think we should have listeners know this is not a new problem. Conspiracy theories have been consistently with us for their age old's conspiracy theories out there. The problem is really the spread of them at this point. There's never been really a moment in time by which you can spread something so quickly so fast.
Eric Boduch: Yeah. crosstalk Way back when it spread through word of mouth, that's relatively slow. Then we had the printing press. Again, only a certain amount of people can read. But now the internet and our addiction to social media and never ending scrolling means things can spread really fast.
Kristen Berman: They can. Yeah. And so there's basically two problems here. Can we stop the creation of misinformation? And I'm a little skeptical that we can completely just take away misinformation. I think there's some historical relevance that humans have put on it. It's a problem to be solved. I think the urgent problem to be solved is to reduce the spread of it. And this is where the platforms come in: Facebook and TikTok and Twitter. And so we had a partnership with TikTok to do just that, where we said, " Look, let's try to figure out... There's medical misinformation, there's political misinformation. How do we help people avoid spreading something that either they know is false or they don't know is false and still want to spread it?" And so I think that's really where the problem started, or the definition of the problem started. As behavioral scientists, what we then did is what I was talking about before, is we just... we read, we read papers. We said, " What other research has been done around stopping the spread and misinformation?" It turns out that we're not the first person to think of this. And so in the literature review, you have a few things come up. One of the primary researchers here is David Rand and Gordon Pennycook from MIT. And they've done some nice stuff on accuracy. So the question with misinformation is basically, are people spreading something that they know is inaccurate? And that would mean, and you'd infer from that, people don't value accuracy. And so they said, " Look, do people value accuracy?" And they did just a normal descriptive survey and said, " No, actually people say they value accuracy. 73% of people say that they value accuracy and the information that they're reading and the news that they're sharing, and yet when faced with something..." By the way, they also said, " Look, can people determine if something is accurate?" And they did some nice studies to show that no, actually people can determine by and large what is accurate or not. The problem comes at the point of sharing, where even though people can determine if it's accurate or not, they still were sharing it. They said, " How likely would you be to share this news information?" And even though they were saying it was accurate, they would still say that they would be likely to share it.
Eric Boduch: Even though they said... They'll share inaccurate data.
Kristen Berman: Right. And so the question there is, it's not that people don't care about accuracy, it's that they don't care about accuracy in the moment of sharing. In the moment of sharing, there are other things that we're thinking about, either because we're distracted and there's an attention issue, or because we're thinking about confirmation bias and seeing it, what our followers may think or friends may think, or the future likes that we may get. Regardless of what it is, we're just not thinking about the value of accuracy there. And so one intervention, I'm not saying this is the only intervention, there's lots of things we could do, I'll say the one that we did, which is basically remind people that it may not be accurate at the point that they're sharing. The other thing we did is slow them down. And so it's like, " Look, we can tell you that it may or may not be accurate, or you should be caution, and we're going to make you click another button that says, " Are you sure?" And so we designed this label, " Caution: The information may not be accurate." TikTok changes to Unverfied, but... And then before they shared it again, we said, " Look, are you sure?" So the TikTok finding is that we reduce the percentage of people who are sharing by 24%. Just the label reduced views. Actually, if you've used TikTok, you know that it's a really speedy platform. And so if you're just flipping with your thumb through the videos and you see a label on it, it's possible that... By the way, it's possible that you may click in because you see a label. It's also possible that you don't care because you now see this label. Actually, one condition that we did add that didn't work out, but that we were nervous about, he said, " Look, we're going to label all these videos. If you see a video label, aren't you curious what it is?" And so he said, let's actually delay when the prompt shows up. So we can't get people who are just on the video and then click in because they see the label. So we said, " We're going to delay it by two seconds." Two seconds at TikTok is a lot. If you haven't used the platform, this is ages. And it actually had no difference. So this delay actually didn't change the percent that people shared. But this is what we test, because I think us in the TikTok team were expecting it to actually work even better because we wouldn't be attracting this curiosity effect. So in general, our team, when we designed this, we went through a lot of steps. I make it seem like it was just figuring out this accuracy thing and then designing the label. We did qualitative research, where we mocked up different types of labels and compared almost 16 labels where people had to look at and it was more of an understandability sort of UX/ UI investigation. And then we did a quantitative study with around a thousand people where we showed people different labels and then there is a likelihood to share. In these types of studies, we don't actually care about the absolute answer. We don't say, " They said 7. 1- 10, they said seven." We actually only care relative to the other designs. So did one design do better or worse than another design? That increases our intuition about what would work or not work. And so in these quantitative pretests that we do out of product, the rapid tests, we can do one in less than a week, and just gain a better intuition about what people would do so that what we do in product testing, we're more confident in our hypothesis.
Eric Boduch: Now, let's turn this back to product a little bit, since you mentioned TikTok, obviously a very popular product. What do you see as the top issues in product management today? And how does looking at customers' behaviors impact those?
Kristen Berman: Yeah. I think this is going to sound so obvious as I say it, so I apologize in advance. Behavioral science focuses on the behavior that we want to change. We get uncomfortably specific about the behavior. And we find that when we're generally talking to teams, and maybe the PM knows it in their head but the team doesn't, we're generally talking to teams and we say, " Look, what behavior do you want to change? What uncomfortable civic behavior?" And sometimes if you replace the word behavior for action, what action do you want the user or to do? This may sound more intuitive to folks. And you ask everyone on the team, more likely than not, they say something different. We actually did this once where we videoed a bunch of execs and says, " What behavior does engagement... What does engagement look like? Tell us the behavior." And then we presented to them at the end of the day with their sliced together answers, which were all different. And then you look at their marketing landing page and you're like, " Yeah, you guys disagree. The marketing landing page is a mess. You want people to do very different things." And so if a PM were to go through every single page of their app or their experience and say, " Is it clear what behavior somebody is supposed to do, an uncomfortably specific behavior, on this page?" I think actually marketing landing pages, it is very clear. You have big buttons, but in product pages, we forget that actually people still need a very clear direction of the behavior. And so you can do this at the micro level at each page, and then we want to do it at the macro level. What behavior do you want to change at the macro level for your product? And so we think if PMs could sharpen a skill, it would be thinking in terms of behaviors, in addition to outcomes. Folks are very good at outcomes: active use, engagement, retention. Sure, lovely. These things are great. It's really getting everyone to agree on the behavior that we think... By the way, when you do this, it's just so much easier to design products, when you're like, " Look, this is the behavior we're going for." By the way, it's a hypothesis, so you may get it wrong. But when teams agree about the behavior, we find that all of a sudden light bulbs go off and things get much easier.
Eric Boduch: So how should startups engage with behavioral science? How should they approach that? If you were starting a company to do X, maybe you're starting a delivery service, like the next DoorDash, how should we engage with behavioral science when we're starting companies?
Kristen Berman: Great question. I'll say, in general, this is not to be a self- plug. This is actually just super genuine. We created this bootcamp because it was really hard to engage with behavioral science as an applied person who's very busy. And so the bootcamp is just eight weeks of lectures and videos and articles that walk you through the behavioral design process; what I talked about with the key behavioral diagnosis and design, but at the fundamental level, I'll say I'll... This is framework, so just helpful to simplify the world. I would say, I think the easiest place to start is to say, " What is the behavior I want to change? What are the barriers to that? And how do I reduce the barriers? And how do I increase the immediate benefits?" And this is a simplified view of the world, because you basically say, " I want to increase the..." We're- very- present- biased. And so these benefits basically can be anything, but they need to be immediate. And if you have any kind of barriers in your way, cognitive or logistical, people just aren't going to take the action. And so it would be pretty ruthless about this framework of saying, " What is the behavior our team is looking to do? And then how do we reduce barriers and increase benefits as a framework for design, for prioritization, for thinking about..." And I think what this represents is basically you have to have a model of the user. We're making a lot of assumptions about why people do what they do. And so this model basically says, especially for the benefits, is like, " What is going to motivate somebody to take an action?" And I think especially teams that are starting are pretty idealistic about behavior or user behavior or they think people will do things. And we need a better model that says a little bit more about why that'll actually take the action. And then probably hiring a behavioral scientist. I think there's something to basically getting a lens in there that says, " Here's what we know about the world."
Eric Boduch: Yeah. If you had a team, at what point should product leadership think about hiring a behavioral scientist? What size makes sense? Obviously, if you had 1: 00 PM, you probably don't want a full- time person that's just doing behavioral science, but at what point does it make sense?
Kristen Berman: We've set up three behavioral science teams now and helped them... We helped Indeed with theirs, helped them hire and set up their group. We're helping Intuit now set up theirs. I guess I'm forgetting Google because we started theirs, and then Vault, which is a UK energy company. All of these are probably too late. The teams are already set up and ingrained. And so I think it comes to the point where basically you're working with a couple design and researchers and you're like, " We need a shortcut. We need to figure out basically, what is the model driving our users?" And that behavioral scientist supports the PM, they support the researcher and they support the designer, because they're basically... all of our behavioral scientists, by the way, are good at Figma. We get in there and we do the design. Now, we shouldn't be designers, but if life is about the details and behavior change is about the details, we really have to think about what is a choice architecture? What is the decoys? We talked to one company that had an open text field and we said, " What are you doing? You can't have somebody enter a number there. Of course you have high drop off." And they said, " Look, this is just in our design guidelines. We don't even have a radio button guideline." At some level this is just not going to work. And so behavioral scientists, I think, can basically support... I don't have an exact number, but I think as you get the research, the design and the PM team as the triad, this is really when the behavioral scientists can come in and offer the guidance. crosstalk By the way, I have an article called The Behavioral PM. This is a riff off of How to be a Good Product Manager, Ben Horowitz's post. And it really thinks about... I guess the question would be, is, do you need to hire a behavioral scientist if you're a small team? I would say, get better; get better and curious about what's driving your user. I don't want to say that you have to... I'm biased if I were to say that you have to hire behavioral scientists. I think the skill set is out there and available and people can be learning. And I think the caveat would be, do it for good. As we think about behavioral science, we always have to think about what are the incentives that you're doing and are you designing for the consumer's well- being?
Eric Boduch: Yeah. I knew we're running a little out of time, which tends to happen, but it started when you said, " Do it for good." I started thinking about behavioral science and engagement patterns for bad, because there's a lot of things out there that I look at, and I was like, " Ooh," you know?
Kristen Berman: Yeah.
Eric Boduch: They are completely grumous, especially when you get into mobile gaming and crosstalk and things like that, where you're get people to pay for things that you get ingrained these habits. You form these habits for people and then you start saying, " Now these habits cost you money, but they're already ingrained."
Kristen Berman: Yeah. And I think this is... I'll step back and say for us, the problem in most... and this is most of life, it comes down to an incentive one, how are we measuring PMs and marketers? And if we're measuring them by how much active use you could have, you're going to get people designing products that have red buttons and notifying me and making it hard to turn off notifications. This is an incentive problem. No one is a bad person at these companies. It's an incentive problem. And so as a PM, the opportunity really is to figure out how you're incentivizing your designers yourself, and is that for the customer well- being? And the job of the PM is to measure the customer well- being and report that back to the company. And so I think, if we turn up the dial on KPI, it would say, " How do you measure well- being and how do we report on it so that we can actually try to drive that versus maybe easier to drive but very measurable outcomes, like engagement and active use, which are nice to drive maybe for the business, but also we have to start measuring the well- being.
Eric Boduch: Yeah, yeah. This could be a whole nother podcast, I think, about habits; how they're formed, how we change them.
Kristen Berman: Yeah.
Eric Boduch: But alas, that'll be for a future day. I'd love to hear, what's your favorite product?
Kristen Berman: I am smitten for Peloton right now. And I say that knowing that it's so cliche, but it is... I just talked to crosstalk
Eric Boduch: I'll ask why, why are you smitten? What makes you smitten for Peloton? Because it's an answer I hear a lot and I have one myself and I do love it, but what about it-
Kristen Berman: I actually only use the... I only use the app. I talk to David Packles, who's their group PM, and the way he thinks about the world is very nice, where he is leaning into behaviors that are already happening on the app. If there's a new thing called Stacking, where people are putting together workouts, that and the Here Now feature, a lot of this came from existing behavior where you're looking and observing. In behavioral science, we basically don't ask people questions as much as most researchers do. We observe behavior and then try to build behavior change products that, we call it mistake proof, that just help the user do what they want to do. And so I think Peloton has a nice combination of that. Tactically, they all say, " Look, their streaks are very good. In the app, gives you a variety of ways to engage." So there's a problem in exercise called explore/ exploit. Actually, there's a problem in life called explore/ exploit, where basically imagine you move to a new city, you have the opportunity to see a lot of restaurants, go to a lot of restaurants. How quickly do you settle on the restaurant or a group of restaurants that you really like? You may settle too quickly, by which you're missing out on the great restaurants, but you avoid the search costs. And if you optimize for too long, you're going to have a handful of bad restaurants. And I think Peloton does pretty nicely, is basically expose that there are a variety in the app; a variety of types of workouts you could do with them. There's running, there's yoga, there's stretching, there's strength. I think they're helping people explore a little bit more on exercise than your typical, " I go to a yoga studio in San Francisco and this is the one that I go to." I worry that if people get too consistent and they're working out, the working out will go down. The other thing I'll say is I think one big thing in behavioral science and idea is that we're not just individuals. We are part of a system, a social norm. We are accountable to other people, this is what drives me to prepare for a meeting. I'm not going to just show up to a meeting unprepared. Why? Because other people would think I'm silly. Peloton, with the instructors, really they're this soft accountability. People really feel engaged with the instructors. There's Jess, there's Alex. I think they use the instructors as a way to hold people to the types of people that they want to be in a pretty clever way. And I think they could probably do more of it with the instructors asking for commitment and things. But accountability in general is something where, if you're acting without anybody knowing what you're doing, you're going to act different than if other people see you. And by the way, if we go back to misinformation, we act differently when we're sharing political information if other people know that we're sharing it versus not, and who we're sharing it to. So there's something really big to play with with anonymous versus non- anonymous behavior. And I think Peloton makes you a little bit less anonymous than other apps.
Eric Boduch: Got it. Now, if I'm a PM out there and I want to learn more about behavioral science, what would you point me to?
Kristen Berman: Again, this is biased, but we have a bootcamp, it's eight weeks. We're also starting and we're going to launch a cohort. Cohorts are all the rage now, and we're just fast followers, but we're going to launch a cohort. Our first one will be in October. And that'll be more intense where it'll be a four- week program, three days a week. The current one is eight week online, self- paced. We have a Slack group, we have an HR call, we have office hours. So there's some touch points, but a little bit more self- paced. It's behavioraleconomicsbootcamp. com, but also just find it in Irrational Labs. That'd be the formal way to do it. The informal way to do it would be to read a Dan Ariely book, Predictably Irrational, watch a TED Talk, start reading papers. So if you're thinking about solving a problem in education and health and finance, somebody else has already thought about the motivations of what's driving your customers. And if you don't have a day, devote an hour to a quick scan of the literature.
Eric Boduch: Awesome. One final question for you today, three words to describe yourself.
Kristen Berman: Ooh, three words to describe myself, I would say action- oriented. I have an urgency bias. I am an optimist. Optimism bias is very present for me, where I think things will go very well. And then just funny. Just kidding, it's a joke. I would say I try to take a unique path. I live with 14 people in a community house. We have apartments, we have a home, two of our friends just bought two houses next door. I'm trying to design life for happiness. And so really live the principles of behavioral science of we are who we surround ourselves with, the environment drives our decisions. And so I think that means that sometimes you have to get off the beaten path on what the norm may be that you're in and crosstalk
Eric Boduch: I feel like there's another podcast here too, right?
Kristen Berman: Yeah. Yeah, yeah.
Eric Boduch: I wish I knew that earlier. There's so much I would like to hear about this, but crosstalk
Kristen Berman: Unless the name crosstalk
Eric Boduch: Unless they're future days, that's the great thing, you know?
Kristen Berman: Yeah. The name of the crosstalk is inaudible, if people want to Google it, though. We've written a lot about how to do this stuff.
Eric Boduch: Awesome. Cool. Thanks. crosstalk This is super pleasant. I really enjoyed it.
Kristen Berman: Yeah. This is great. Super fun. I agree.