Speaker 1: Hey there, product lovers. Welcome to the Product Love Podcast, hosted by Eric Boduch, co- founder and chief evangelist of Pendo and super fan of all things product. Product Love is the place for real insights into the world of crafting products, as Eric interviews, founders, product leaders, venture capitalists, authors, and more. So let's dive in now with today's Product Love Podcast.

Eric Boduch: Well welcome, lovers of product. Today I am here with Scott, the director of product management, and Russel, the SVP of innovation and product management from WebPT. Scott, why don't you start us off by giving us a little overview of your personal background, and then we can have Russell go next?

Scott: Yeah, for sure. So hi, everyone. My name's Scott. I'm a director of product management at WebPT. Before WebPT, going way back, I'm actually a doctor of physical therapy by trade. Went to PT school, spent all that time and money getting that PT education and then stopped clinical care pretty much immediately after graduating and started a technology company. Ran that for a number of years in Boston and joined WebPT almost four years ago now and joined the WebPT product team after the acquisition with WebPT. So I've been looking at patient engagement and how we better connect providers and patients through technology for a better part of the last 10 years.

Russell Olsen: Hey, everybody, I'm Russell Olsen. I'm head of product at WebPT. I started my career actually in electrical engineering, and then I went over and did financial audit for a while. Then about 15 years ago, I got into a startup in healthcare and then spent the last 15 years doing everything from quality reporting to complex care management for diabetics, spent some time at Watson Health doing some really interesting things there and part of the formation of the Watson Health Business Unit, and then about three years ago came over to WebPT and lead the product portfolio here.

Eric Boduch: So talk to me about WebPT. Can you give us a little overview of what WebPT does and the problems you're solving?

Russell Olsen: Yeah. Yeah. So the history of WebPT, I was co- founded by a therapist who looked at all that she was doing on paper and thought that there had to be a better way. There was. So it was one of those really awesome stories where she was looking to solve her problems, solve them, and it just resonated and just had that perfect market fit, the documentation, the system, kind of had the mental model of a therapist, and so it resonated really well and took off. So our mission statement is around helping rehab therapists achieve greatness in their practice. So giving them the tools, the solutions to put into play so that they can better engage their patients and ultimately help people recover and get back to living better lives. So it's been a really great story. So we do everything from scheduling, to billing, to documentation. We have, I think about 5, 000 home exercise videos that we allow patients to engage with their care on. We have a whole suite, as Scott mentioned, the tools that he helped create around engaging patients and re- engaging them in their care, and we even have an e- commerce place where the physical therapist can buy physical goods for their practice. So just kind of pretty much anything and everything that a rehab therapist would need to deliver care.

Eric Boduch: Awesome. Thanks. Now, talk to me about this last year, because it had to be an interesting year for WebPT, right? The world's taken a big turn. It's definitely shown them the software space. How has COVID affected your product roadmap in the direction of WebPT over these last little, now we're working on what, nine months.

Russell Olsen: It's weird to think it's been nine months. Yeah. The data tells a pretty interesting story, and we were climbing up into the right and took a hard cut down. It's hard to get physical therapy when nobody's coming into the practice. So we saw a really significant drop in users, and we have a large portion of the market using our software, and pretty much people stopped seeing patients, which is the lifeblood of their practice. So we saw a pretty nasty dip and strike down internally. So as you start looking at that happening and looking at our revenue and how we sustain our business, we started looking at how we're supporting our users and our employees and really started looking at, we did not want to lay anyone off. We didn't want to do any furloughs. We had no idea how long this thing was going to last. So one of our first jobs was to make sure that we didn't have any layoffs and didn't do any furloughs, and we were able to accomplish that obviously by cutting back. Basically inaudible anything that was a non- essential for us. So that was what happened, I guess, when we hit that dip. Scott, well, you talk a little bit more about kind of from the roadmap perspective and perhaps-

Scott: Yeah. So to set the stage, this year's roadmap was a really important year for our products. Our company for the 10- plus years now, and we're really about on the precipice of launching some major new advancements to our tools and some really big product new launches. I think every one of our paths is undergoing some form of a relaunch in 2020. So in some respects, it was the worst possible time to have to" pivot" our product roadmap, because we had some very well articulated goals. The product vision, and the product direction was directly tied to helping support those goals and objectives. So for us, what was a really big part of this was figuring out, how do we open up the capacity and make smart decisions about opening up capacity to address the new evolving market need and while at the same time, not allowing that to cut into the product roadmap and the product vision itself? I think we were successful in doing that. So a big part of kind of what changed overnight, as Russell alluded to you, and we all know is the delivery mechanism for care changed again, essentially overnight. Everyone went into various forms of lockdown. People stopped going to PT. so the sort of core delivery model changed, and we needed ways to support it. As a product manager, it was one of the most fascinating situations ever because we all know how long organizational change takes, and we have been surveying sort of digital care delivery methods for a long time, and we had kind of had the plans to work on them in our back pocket for a while, but they never were common, and they were never common enough for us to kind of go down and make a big development investment in these areas. But yet. Overnight, almost every practice switched to having some form of telehealth, which is just a fascinating, cool, cool wrap the shift. So we knew we were going to continue to be successful and meet our mission statement. Again, as Russell said, to help rehab therapists achieve greatness in practice. We had to help them support these new delivery methods. So a lot of what our roadmap focus ultimately ended up being was figuring out what is truly important, what is not important right now, what can we sort of cut from and shift around so that we can continue to execute on our key priorities, keep those going, while at the same time still open up some capacity to explore these new tools.

Eric Boduch: Yeah. That's really interesting big shift to telehealth. I imagine it was a slow shift before, but then all of a sudden, you could maybe see like in four or five years that there's going to be a big shift to doing more and more remote. But now, all of a sudden, it just jumps up, right? Instead of there's two telehealth visits a quarter, there's now 2, 000 telehealth visits a quarter. Talk to me about how nimble you had to be to Justin and to help your customers deliver on that?

Russell Olsen: Well, I think it was interesting. I mean, personal story, I think last it was October or last December, like Scott said, we're always kind of around and playing around exploring telehealth, and I did a virtual visit with a doctor, and it was horrible. It was a really bad experience. I think I got the wrong diagnosis. I ended up having to go to an in- person doctor, and that's... So there's all sorts of challenges, and then you throw in the fact that physical therapy, most of it's hands on, right? So part of the fundamental of the delivery mechanism was in person. So as I started to look at that, as COVID hit, and we all went home and hunkered down at our houses, and everyone is remote, we started looking at this, and we didn't know how long this going to last. Is this a one month thing? Is this a forever thing? Then the other thing going through my mind was, well, is this a fundamental pivot from the roadmap? Are we shutting down these projects that we've been working on for years in some cases that we know long- term are the right things to do? So we started kind of going through this. Well, we started out by saying, " Well, here's some recommended people that were already doing telehealth and things, and just you should go sign up with them." Then that kind of evolved that like, " Well, what if this thing doesn't go away, and what if this is a fundamental shift that we want to enable our users have the best experience possible and integrating kind of sending them off to some third party is maybe not the right way to do it long term?" Then the other thing kind of not in the back of my head, when you launch a product, especially when you might've thrown together quickly, there's issues. Things don't work. Especially if you add the component that you are touching the consumer, right? So it's one thing if I am delivering this to a business, and it's kind of within this confined constraints. They're already using the soccer, where we already have minimum screen size resolutions, technology. But now, when you open up consumers, who knows what technology they're using. I think one of my biggest fears was, yeah, we slammed something out. We get something out there, and we just create a support nightmare for the organization, and it's worse than if we hadn't done anything at all. But at the same time, I didn't want to sit there and do anything at all. So these were all the things kind of rolling around in our heads as we kind of went through this process. Scott, jump-

Scott: Yeah. Well, yeah. So when we surveyed our members, as Russell said, it was exactly at first, we were like, " Hold on, we need to figure out how we can shuffle things around. So here are a couple of partners that we think would work." But the more and more we talked to our end users, we realized they had major workflow problems. They were using, in some cases, four or five different tools to handle things like patient communication about the visits, patient scheduling, actual video visit, communication, and then a followup. Then they had to use our tools to document those sessions, perform billing tasks, collect payments, all those things that they were already doing. While it was amazing to watch everybody adapt and quickly find solutions, there was a huge dissatisfaction in the market. On top of that, as Russell said, they couldn't treat with their hands. They're trying to learn how to change their whole method of what they went to school for years to do and then practicing for years. It was a major change. So what we realized pretty quickly was if we built an integrated workflow, one that tied into the tool set that they were already using with WebPT team, but we did it in an intelligent way that, to Russell's point, allowed us to potentially partner more effectively to de- risk some of the technical risks, we felt really confident that we could deliver a strong product to the market. We could deliver it quickly and provide a lot of value. So that's sort of what we set out to do.

Eric Boduch: Yeah. There's a lot of challenges in that, right? I mean, you're moving, and it's a whole new motion. I mean, you had a normal 10-10- 10 approach, right? Which this is probably now you're moving to more shipping small in an instance like that. Is that right? Can you explain what your normal approach is and if I'm right on the shipping small side?

Russell Olsen: Yeah. So I mean, just back up, our normal approach is we ship software. I mean, we have over 90, 000 users on the site daily. We're releasing code weekly. So there's a lot of change. Then people don't want to wake up every morning and have something different in their system. So our normal process is what we call the 10-10- 10. So the first 10 users or 10 physical therapy clinics data get a piece of soccer or a feature. We make sure it's working. We iterate. We get a feedback, then we go to the 10%. So might go to 1,000, 1, 500 clinics, make sure operationally it scales, that the operations enablement is there with training materials and all that. Then once you don't have any hiccups in the infrastructure, and then we go to the last 100% as the last 10. So that's our process, and Scott can walk you through it, but we didn't change. We just did it faster.

Scott: Yeah, exactly. So we had a couple of things that worked to our advantage. So one, moments of crisis, and I will say kind of when this initial first wave happened, it really was definitely rally people together. So one of the hardest things about shipping products is organizational alignment. This is probably one of the rare instances where we had immediate 100% organizational alignment, from our CEO, down to the support rep that had been there for a day. We all knew what we had to build and what the stakes were about getting it out. So that helped streamline things and allow us to go faster. But what we also were able to do was utilize high- fidelity mock- up, because again, the product surface area was relatively small in how it worked. So we were able to build both a quick prototype, as well as some of it was code. Some of it was just design only, circulate with that with the company over a couple of days, period, getting meaningful feedback about approach and direction and meet with the individual stakeholders across our organizations. So within about a two to three- day period of kind of that launch week that we ultimately started doing this, we're able to take activities that normally might have been staged out and overlap them on top of each other. So we probably spent two to three weeks ahead of the initial like, yes, let's go do this really making sure we understood the technology, knew the approach, had the alignment from the high level of the organization on down. Then we basically hit the seven- day window, where what we did is we kind of met with each of the cross- functional groups around the company. We said, " Here's what we're building." We knew that from the engineering projections, we could get that first milestone shipped within a seven- day period. So essentially, what we did is over that seven days, we got alignment of the entire company while we were building the actual product. A few changes came in based off of those kind of early feedback, and this also included reaching out to some of those first-hand members that would get the product. We were able to iterate a few things in terms of workflow while the engineers were still working and then progressively work up to make sure that we had sort of the cross- functional collaboration necessary to ensure that all stakeholders were informed, understood what was happening and what the timeline was going to be. By the end of that first week, we had completed all of our development work, and we had trained the entire company about what was coming, and everyone kind of had an understanding and agreement that basically then what we were able to do is the following week, we started our 10-10-10, except instead of taking maybe a month or two months sometimes to go through it, we went the first 10 on day one and day two, 10% on day three and day four, and by Friday of that following week, we were at 100% rollout. So again, I think strong organizational alignment, strong understanding of sort of the mission behind the product and what it was going to be allowed us to take our normal process and just condense it really tight.

Eric Boduch: Now, does this have ramifications for your normal process timeframes moving forward, or is this an exception?

Russell Olsen: It's interesting. One thing Scott didn't mention was we also changed the sales model.

Scott: You're right. Yes. I did.

Russell Olsen: But not only that, but we knew that we wanted to get out to many people, so we enabled the self signup model, which was our first time doing that. So to your question, I've pondered a lot since that seven- day sprint, if you will. How can we do this more often? There's a lot of factors that have to come in to galvanize the team and to have a clear mission to accomplish something. But it's absolutely something that we've thought through and looking at what lessons we can take and apply and what lessons probably only worked in maybe that scenario and might not work so well if we're trying to apply that again. I wouldn't want to spend seven days and rewrite the billing app and try to mess with payments with payers. But definitely some process. Really, when we have title alignment, a lot of these things go faster and easier.

Scott: Yeah. Russel and I both joked at the time where I was like, " Man, our CEO is never going to let us do a slow rollout again." We showed what was possible to the whole company. But in many ways, I agree, we had extrinsic factors that made a lot of the really hard things that go along with launching a product not hard in this one instance. So I think it highlighted the importance of various areas of the product development and release process. It highlighted the importance of alignment and what it takes to get there. I think it highlighted the importance of a shared understanding of requirements and what the final deliverable is going to look like, and it definitely shared the importance of various parts of the process, such as the cross- functional collaboration to ensure all stakeholders feel heard and have weighed in. So I think it's allowed us to refine our internal processes so that there's better success in how we launch and roll out products. But yeah. Getting to that timeline is going to be challenging, I think, of the future. But hopefully, we've seen some productivity gains, for sure.

Eric Boduch: Yeah. I want to dig into a couple of things you said there, one of which was a feedback, right? Because customer needs are changing immensely during this time. I'm sure you're getting a ton of feedback, and you're fitting them into a really tight process, and you've probably got conflicting feedback. Right? So how did you navigate through all that in this tight process of rolling this out?

Scott: Well, Russell, I've got a couple ideas, and I'll let you chat on this. So for us, what I tend to find is in areas where you get a lot of feedback, the first step to try to prevent them from becoming just complete bottlenecks that block anything moving forward is to make the scope smaller. Because on a smaller install on a project, you can tend to get to consensus on what that should be, and then you roll those conflicting points into sort of future iterations and where you look to sand. So for the case of this product, I know I presented Russell and our CTO a flow chart of sort of holistically what this process should be. We all laughed. We all knew it was no shot when we were building all of it. But it was important to say, " Okay. Here are kind of all the various touch points where we think this could go in the future. Now, here's what we're going to actually do." A very, very narrow slice of that. So for us, I think that the priority is make it smaller so that you can get something that you can ship, and that at the end of the day, the products can speak for itself, whether it works or whether it doesn't, and those things are going to rise to the surface. So ship small, iterate, gets some data to support those conflicting points, and it usually is the best way to win those arguments. But yeah, Russell, I'm curious where you're inside of with it.

Russell Olsen: Yeah. I think if we bookend it, it was an incredibly difficult decision to kind of disrupt our plans and to pull people off and go tackle this, not knowing all the uncertainty that was out there. Once we got the initial delivery, it was almost just as a difficult decision of, well, how far do we go with it? Is this now a new team, a new product line that is going to live on forever, because certainly, there was an argument that there's a new norm, and we need to have a whole different skill set of tools, or have we delivered what we needed to, and do we get back to our other projects? That was a really difficult decision of, how far do we go? I think the feedback loops were important to let us know that, " Okay, we had done enough. There were some operational cleanup that we did a little bit after." In fact, we just did another little round just recently. But we decided that it was working. People were using it. It was delivering the value. We didn't see any issues with NPS scores or feedback from people that were using the product, and we decided that it was good enough. It solved the problem, and now we can get back to some of the other projects that we knew we had to solve that were more important. So yeah. You kind of bookend to very difficult decisions. Do we start, and do we stop?

Eric Boduch: Yeah. It brings up an interesting question. Do you see telehealth for PTs becoming the norm or at least the much more significant piece of their business than it was prior to this? Is this a change in how your target customers deliver their services?

Russell Olsen: So I'll let Scott dig into it, as the physical therapist in the background in any way. I think my perspective is there's definitely a digital component that has changed. So one of the other things that we haven't talked about is we also launched a digital intake product that fortunately, we've been working on for almost a year prior. But it enabled it, and almost not entirely touchless, but definitely facilitated getting rid of some of the clipboards and paper and pens that now people are looking at really strangely. Nobody liked them, to begin with, but now, they've got an extra factor of what's touched that thing before I touched it. So we were able to also bundle in not only the virtual visit and the telehealth experience. We had some digital intake tools. Then also, on the billing side, we haven't talked about it, but there's a lot of changes on how you bill insurance companies and which ones were allowing it and which ones weren't and how you had to bill it. So we're able to kind of adapt and adjust. I think some of those will live on. Some of those won't, and Scott you're going to find on that.

Scott: Yeah. As people in technology, we tend to believe that technology is going to solve every problem. So I think that that would be a potential fallacy here as well. I don't think you're going to see 100% digital therapy being the norm.

Eric Boduch: No, no, definitely not 100%. But before, I imagine it was less than 1%, right?

Scott: Yeah. It's between two and five. So what you're going to see in my belief is a continued move towards hybrid care models. So if you think about a traditional course of care and therapy, I may have acute onset of some issue that acute onset might require highly skilled manual intervention, and it's advantageous for me to receive that early at the beginning of my course of care. In that case, we're not going digital. We're going directly into clinic, and we're going to manage that care well there. The other side of it though is there's many conditions that don't require hands- on care at the beginning, and they can be delivered remotely. So at the same time, I know I've talked about this a lot. There's been almost a half a billion dollars invested into the digital therapeutic space around musculoskeletal conditions. Those are changing the standard of care. Those are 100% digital. So if you talk to those companies, they'll tell you, yeah, I think that's where the future is going to be. I think we'll land somewhere in the middle. I think it's going to all come down to consumer preference. When we survey patients, we're hearing a lot of patients who want to go back to the clinic, interestingly enough. That could be because the digital tools are starting up to par, and they aren't good enough yet. But it could also mean that people do prefer. Physical therapy is a fascinating industry, because while it's the most common and most costly health condition in the United States, not just the US, actually across the globe. One in two adults every year will develop a musculoskeletal condition that could be treated by a therapist. Only 10% of those patients actually show up and not making it into care with a PT. So it's woefully underutilized on top of all of it. So I think that digital tools have the potential to build conduits to tap into that broader market and provide care in more meaningful ways. I think that you'll see traditional therapy groups start to adopt some of these tools as they become available and more prevalent. But right now, what we're seeing, and we saw with our usage patterns on the telehealth tool was when lockdowns are occurring, telehealth use rose. When lockdowns were coming down or people were allowed to go to the clinic, usage dropped. So I think the way that the traditional therapy industry will... they will continue to use this somewhat. But it'll be largely an alternative traditional care, right? They'll probably go back to traditional care inaudible.

Eric Boduch: So just curious, from two to five, to what? From 2% to 5% to what? What do you think?

Scott: Yeah. So it depends on who you ask? I think the APTA survey, and it may have these numbers slightly off, but it went to... Basically, they surveyed our professional organization for all the therapists. They survey the organization and found that two to five were actively practicing any form of telehealth. Then in the midst of COVID, it was up North of 60%. So it got very, very high. A number of smaller practices unfortunately closed down because they just didn't have the means or capacity to manage those swings. But again, when we survey our member base, I know when I made calls to sort of as many members as we have close relationships to, every single one of them, it was closer to 100% in some form of a telehealth program in place. So it really was a radical shift. Everyone I talked to now tells me that they're going to continue to support it. But I think they had a preference towards in- person care when possible.

Eric Boduch: Got it, got it. So we're going to see a big bump from two to five, at least for part of the services delivered and based upon consumer preference, but we're not going to get to that 100% digital any chance, forever.

Scott: Probably never. But we'll see how that-

Eric Boduch: So another thing you mentioned I wanted to get back to, in this case, you had top- to- bottom alignment really quick, right? So that wasn't a hindering factor at all. So there wasn't as much work to do inside your organization getting alignment around the shift of product direction. But that's not always the case. So tell me what a normal case is. How do you ensure that alignment to accelerate product development normally, and what did you learn about the process you went through with this big shift with COVID that you're going to apply moving forward to enabling better organizational alignment?

Russell Olsen: Yeah. A couple of things. It'd be what did make it difficult. So we had some existing things in place that enabled that we leveraged. So I'll talk about those. But I think what was difficult is normally you would have thrown everyone into a room and gotten on the whiteboard and hammered it out. Instead, everyone's calling in on zoom and kids in the background and personal fears going on. So it was just a really interesting time to even drive the alignment. One of the things we put into place early on was this concept of a shared group of people that are collaborating on the product. So it might have different names. We'd call it a product council. But the concept is that you have a cross- functional group of people that are actively engaged in the product, and it's often facilitated and coordinated by the product manager, but it's not a top- down. It's really a collaboration so you'll have somebody from maybe your onboarding team, your support team, your sales team, marketing, all in there talking about the product. So we had that kind of concept and that structure in place in the organization. So when this thing came about, we spun up a new product group around this problem that we're solving. Really, the concept there is that 360 feedback. Not only does it drive alignment, but it also surfaces, hey, somebody is going to ask the question about, " Well, how is this getting into Salesforce and invoicing?" Things you might not be thinking about as you start to launch the product, or how are we going to turn it on, or how are we going to message it? So having that cross- functional team all present really helped catalyze that alignment, and that is our normal process. Again, a lot of this wasn't... We didn't do a whole lot of things abnormal. We just did them a lot faster, and it was in an accelerated manner.

Eric Boduch: Yeah. I mean, it sounds like it was easier to reach consensus in this case too because of the dramatic impact of the pandemic.

Russell Olsen: Yeah. Yeah. I would say there certainly was a lot of debate, right? So there was debate of whether we build this all natively ourselves, whether we partner, whether we white label a solution. There's all sorts of debates on whether we should even do this at all. So there's lots of debate and a lot of discussion and a lot of passion I think into it. I think usually, if I were to kind of compare this to maybe a normal project, I think a lot of that debate and passion sometimes happens behind the scenes or happens offline. This one, I think a lot of it happened real time all on Zoom. I think it was actually more healthy than where you might've had a lot of pre- meeting conversations or a lot of discussion around the meetings or after the meetings. I think the fact that we were very passionate kind of in the meeting and in the moment, I think drove alignment faster than maybe traditionally how it's done.

Scott: Yeah. Yeah. I completely agree, Russell. It was because of the stakes and the timeframe, the disagreement was very productive. It happened early inaudible and allowed us to sort of... In any case, where we disagreed and moved up, we were all able to move on. I think the biggest disagreements, whereas you touched on the course, how long do you keep working on this? How far did you take it? The benefit year, as I said before, it was, we were able to show data, right? We're able to say, " Hey, this product is resonating. We're accomplishing what we want to do, accomplish from this product." It is okay where we are. Remember, we do have bigger organizational goals that we don't want to lose sight of. So yeah. Sure.

Russell Olsen: Yeah. One of the metrics we looked at was, I was personally really fearful was we've got patients using this solution now. Right? What happens when the patient tries to get on the virtual meeting with their therapist and can't? It doesn't work. They got an old phone or an old browser, or something's not working right. Are they going to call us? Are we going to, as our support team, going to now have to open up to every patient that calls in. So one of the things we tracked early on and kept really close tabs on was, " Well, what's the support? Are the therapists calling us because they can't get their meeting started?" They've got a patient waiting for them? Or can the patient not get online. So one of the factors for me in feeling better about this was we really didn't have a lot of support issues come in, and that was a sign for me that it was working really well, and people were buying and growing. The product was growing. So I think looking at the data and understanding how we can leverage the data to help us validate maybe our feelings and our thoughts.

Eric Boduch: So talk to me about success. How do you measure product success in this case? In general, how do you decide on your North Star metrics? How do you iterate on them?

Russell Olsen: Yeah. So I'll start high level and then Scott can jump in on maybe this one specifically. I say in general, we have our North Star around helping rehab therapists achieve greatness. So then under that, we have beehag goals about, how do we get our solution in the hands of every therapist and every patient that they treat. Then from there, it gets into metrics around growth and profitability and other areas. So one of the ways we look at is looking at, there's long- term metrics around growth and products and sales and support, insurance. So those metrics are kind of there and in the background, but they don't move unless something's really wrong. They don't move drastically. So in the kind of short term, we look at initiatives. So we're going to improve this experience, or we're going to reduce these costs or fix these errors. So we'll go through a process where we say, " Well, here's what we're going to do over the next X period of time and the metrics that we're going to look at to see if that works, which are proxy metrics for maybe the long- term customer churn or growth metrics that take a little bit longer to understand if they're moving in the right direction."

Scott: Yeah. Yeah, totally. As Russell said, that once we get down to the product level, when working with product managers on our team, we are trying to figure out, what is the product- specific metric that's going to drive the area of the business that we're hoping that product is going to drive. I mean, at the end of the day, everything's going to relate back to, is this helping you sell more? Is this keeping people from churning? Are we reducing some organizational costs. But you can basically work to trickle down to understand where these relate to the product, and you have product- specific North Star. I know for this particular product line, it was really simple. We knew that this was... We definitely cared about the revenue opportunity, and we did know that there was money to be made here. But we also really just cared about utilization. Because what we were seeing was as people close their practice, they weren't using our software. They were calling up insurance. We don't have long- term contracts with majority of our users. So this was really important to keep them in business and keep them running. So for us, we wanted to see what was... The the North Star of this product was, first and foremost, sales. So we wanted to see kind of the unit volume that we were selling, and then the second was we wanted to see visible visits and visit minutes. So we were looking at how many virtual visits were being performed on a daily basis, and then how long were those visits taking place for to make sure that they were sort of falling under the normal 30- minute or 60-minute kind of visit time. So that's what we kept a close eye on and watched them grow really subtly.

Eric Boduch: So obviously, this rapid development is a great example of innovation, right? Talk to me about your approach in general to innovation and experimentation. Love to hear.

Russell Olsen: Yeah. I mean, I'll start, and Scott does a lot too. So you can definitely add in. My theory and I guess approach to it is, I guess I think about gardening and the law of the harvest. I've very rarely seen something just be slammed out and be amazing, right? So some things grow and evolve, and they sheer at different rates and timelines. What I've seen not work for sure is that you just wait, and then all of a sudden, you try to do it all at once. So from my perspective on innovation and experimentation, it's really about having lots of things growing in the garden. If you have enough things growing at different rates, and you're experimenting on them, you're learning, then you'll have stuff maturing at a regular pace that the business will consume and sustain. But there's a lot of stuff you might work on for years and might not go anywhere. Then you gotta understand when it's time to stop as well. So I guess that's my approach and always have something cooking in the kitchen and going, and then I guess on the innovation front is sometimes it takes stepping outside yourself and understanding, " Well, maybe are we the right ones to solve this? Has somebody else already solved this problem? Is it really a partnership and not a build and really just kind of evaluating all angles." Scott, you want to-

Scott: Yeah. Yeah. No, totally. I fixed one just important framework that I think is good for anybody. We've had to sort of set what PT is, the knowledge that innovation is going to take time. So quite frankly, many things that end up on a roadmap every year, our innovation projects, and we probably won't see benefits from them, right? Business line benefits could take a little bit. It could take a year plus so those seeds turn into a full harvest that we've been understanding the importance of. At the end of the day, though, I think the projects relate back to, what can we do to increase confidence in an idea or concept? So specifically, I think the important tenants to innovations, what we did with virtual visit with this particular product launch was, how can we increase confidence quickly and cheaply so that we know that we should invest more in this area of innovation. So what we're sort of always poking around on is, what can we man in the box? What can we test? What ideas and concepts can we start to test and iterate on well before they ever get expensive and require R& D effort or require big multi- year projects to complete. If we can increase confidence around a certain concept and area, it allows us to build a stronger case for where are we going next? So I think that's really the foundation of innovation there inaudible.

Eric Boduch: Now, using your garden analogy, Russell, you talked about knowing when to stop experimenting or pruning an experiment or maybe I'll pull the plant out, so to speak. I'd be remiss if I didn't ask you, how do you figure that out? When do you know?

Russell Olsen: Yeah. I wish I had a perfect answer. If you look out in my garden over here, you'll see it's made of plant that's half dead and half alive, and then I'm debating, do I pull it, or do I leave it in there? It's got three tomatoes growing. So I'm going to leave it and see if the Prius is more. But honestly, I think that's the hardest part. As I've done this for years, you get an idea. To get an idea off the ground, it takes a lot of passion and energy and investment, and I've debated myself for a long time. When does that passion, energy investment bind you. In some cases, that's good. You need to be blinded because sometimes these things don't make sense. But when does that blinding become detrimental. I think for me, it's going back to the data and looking at the metrics. Initially, you got to give enough space to grow. You got to let it curate and get its own feed, and you have to protect it in that phase. But there does come a time where it's been long enough, and it was still not producing, or you're not getting the growth that you had hoped. Then you do need to take a hard look at it and say, " Well, is it time to invest that energy somewhere else?" So for me, it really goes to setting out the key metrics of expecting growth and expecting sales. Are people paying for this? Do you ever get a situation where you might have to get away for free to the first one to go in with you? But if you're giving away for free after the third or fourth or maybe in the second, that might be an indicator that you're not quite there if you're not producing value. I guess that kind of maybe is that... The simple answer is, are you producing value, and is that value enough to be sustainable? Sometimes it takes a long, hard look in the mirror to answer that question.

Scott: Yeah. I think it's the hardest question in product management for at least one, because it's... I totally agree, Russell. Especially as a product manager who founded an idea, there's been the initial advocate of the idea. It's almost impossible to be the one to kill it off. Sometimes you need a friend to come in to do it for you. But the only other piece that I would add is I'd say the time to kill off a project is when there are bigger priorities with bigger opportunities that this project is taking away from. I think always being able to assess the potential value of a project and both the pluses and the minuses, right, doing every... Every single thing we do ultimately comes with some trade- off. Being able to articulate and understand those trade- offs is such a key to ultimately communicate and agree upon whether something's going to continue or not. So I think that'd be the only thing I had.

Eric Boduch: Well, thanks. This has been great. As we're kind of getting to the end, I thought I'd turn the questions to more of you personally. So I'm curious to know, what are your favorite products? Maybe you can go first, Scott?

Scott: Sure. I need to think about this for a while. So I'll do a work product and then non- work products, not paid to endorse either of these. My favorite work product is that email client called Polymail. I was a huge fan when Dropbox created their products on mailbox. It's one of the, like speaking of killing off product lines, I would love to see the case study grow at Dropbox, still mailbox. But it sort of revolutionized how I manage email. So Polymail is a great email client that I get a lot of similar value from. It's again changed sort of my workflow and how I engage with email and as somebody who builds in part email marketing automation software, I both realized the blessing and the curse that is email. So anything that can help you not let that on your life is getting. So I like calling it a lot. Then decide I really enjoy playing music. I, for a long time, lived in very small apartments in cities, and so couldn't bring drums and pianos and electric guitar ants around. So started to do a lot more on my computer. There's a company called Output that makes awesome music production software, and my favorite of all of their products is a product called inaudible. Basically, it's a very simple application that is... Actually, it turned into a SaaS product. Again, you basically pay them a monthly recurring license for, and they keep making it better and better. And adding more sounds and news, just like fascinating sort of creative inputs that you can mess around with. I enjoyed it so much because it makes the process of being creative and expressing creativity. It sort of gives you this really fun starting point, where you sorta dial a bunch of knobs and press a button, and then you're off in a new direction. I think anything that you can foster creativity. It's an amazing product. So I'm a huge fan of that, for sure.

Russell Olsen: Yeah. I would say, for me, it's Legos. I think fell in love with it when I was a kid. It's a product that's endured time, right, over... They found ways to stay relevant and alive and the logo of Star Wars and that whole thing. It's a fascinating story. But I think fundamentally, it's some pieces that are sold today and are in the sets, the exact same pieces that were back when I was six years old and building things. It is a fascinating concept of something that you can follow a set of instructions with no words. You can create this kind of amazing looking thing, and then there's an infinite number of combinations that you can take thereafter, and I think, especially in today's age, it's all about that commentarial evolution of things, where we're often building using building blocks from other people. So for me, it's got to be Legos.

Eric Boduch: So your favorite Lego creation, what's the favorite thing you ever built or-

Russell Olsen: Yeah. So I still have it. There's a space station that I got when I was a little kid, and it opens up and launches this little spaceship. Yeah. In fact, we rebuilt it over COVID. We got out all the Legos and divvied them up and got out all the old instructions, and I've got four boys. So we spent a lot of time in COVID building Legos and rebuilding and enjoying those memories.

Eric Boduch: Awesome. That's awesome. Well, one final question for you both, three words to describe yourself.

Scott: Oh, man. You want me to go first, Russell, or you want me to go?

Russell Olsen: Go for it, man?

Scott: All right. So I would use, and figure out this a lot, so I would use idealistic, because I think in many ways I am an idealist much to probably Russell's frustration at times. The second is pragmatic, because I think I've had to learn to be pragmatic. I think pragmatism is really important, balance the idealism, and to be a good product inaudible you're unfortunately really pragmatic most days. Then the third would hopefully be collaboratives. It's one of my favorite things that I love to do, and it's probably the most important thing for me in my role. I love collaborating with people to solve problems. So those would probably be the three ideas.

Russell Olsen: Yeah. I'd probably say empathetic is probably the first one that comes to mind. I do best when I understand what I'm solving and kind of feel the pain of the use case or the persona. So definitely strong passion around empathy and understanding. I think curious. I love to take things apart and rebuild them or just understand how they work. And I think passionate. I get really excited about... I got to be really excited about what I'm doing, or else it's just not worth it.

Eric Boduch: Well, awesome. Thank you guys both. This has been highly enjoyable.

Russell Olsen: Yeah. Thank you.

Scott: inaudible.