Season 3, Episode 5 - "What does a good hybrid cloud implementation look like?" with Ryan Patterson, Systems Engineer, Uber

Andrew: Hey, it's Andrew, and welcome back to Season 3 of Network Disrupted. Where I, along with some very smart guests, help fellow technology leaders trade notes on navigating disruption in our space. This season, I've set a goal of exploring the issue of enterprise cloud adoption from as many angles as I can. Today, I'm joined by Ryan Patterson, who is a Systems Engineer at Uber. I wanted to bring Ryan on today because, among the customers that I work closely with at BlueCat, they are ahead of the game from figuring out the cloud perspective. I'm not saying Uber's approach is perfect but, I think that if you're on iteration two or so of your cloud journey, it's useful to hear how somebody like Ryan talks about his company's infrastructure practices. Listen to the things he seems to take for granted to understand the kinds of processes and approach they seem to have. All right, let's get to it, and if you have a moment, please leave me a review on Spotify, Apple Podcasts, wherever you listen to these. The feedback is always so helpful, and you'll be helping more people like you discover the show.

Ryan Patterson: Maybe you can give me a sense of the complexity.

Speaker 3: We love the pilot proofing concept approach.

Speaker 4: Influence is everything. It influences the human experience.

Speaker 5: There was several failures along the way.

Speaker 6: We want to be early adopter customers.

Speaker 7: You are handling sensitive information.

Speaker 8: It's work, it's love, dude.

Andrew: So Ryan, thank you for joining, really looking forward to this discussion. You started out working for Geek Squad. So you've sort of transitioned, I mean... Sounds like you've touched everything, but you've also transitioned from working with consumers toward this corporate environment. But so in the projects today are pretty diverse. What sort of projects are you involved with?

Ryan Patterson: Right now we're kind of moved past that rapid growth stage that I was originally part of when I first joined Uber and now we're working on consolidating and automating a lot of our processes to be as efficient as possible. So we have a very strong focus on efficiency and finding what things we can do to make things more effective within our infrastructure.

Andrew: Right. That drive towards efficiency. How do you measure that? How do you show the progress along the way?

Ryan Patterson: Yeah, a big part for us is we are very data- driven. A lot of the things that we try to implement through our infrastructure is what kind of metrics can we use to evaluate the work that we're doing. That's one of the first exercises we take whenever we start a project. So, for example, if we're moving towards automation in a certain category, DHCP reservations, or anything else along those lines, we first have to gather the metrics on what we're doing and how much time are we investing currently into this process. So we would go through our ticketing systems to see how many tickets we're getting for something, how much time we're investing into these individual tasks, and then use those asymmetrics to see if it's something that we should be investing our time in to automate or to streamline or to self- service in those regards. So the first thing we always do is look at data.

Andrew: It's almost like test- first development. But I think that's critical. I mean, obviously, there're tons of stuff to potentially automate, but that idea of being able to measure success and measure true value back to the business based on those metrics, I think is a critical way of showing the value. It's a discussion I've had multiple times in our industry, in the world of DNS and DHCP and IPAM. Oftentimes people look at this as operational value. If it's on, good. If it's off, bad. It's like a binary state, it's really bad if it's off, and it's normal if it's on which makes it very difficult to look at things like return on investment for instance, or help guide and justify any sort of spend. So beyond that sort of drive to efficiency, do you also use this data to sort of justify investments in technology or additional staff or whatever the case might be?

Ryan Patterson: Yeah, definitely. One of the things... It's been the hardest thing for me to grapple with in IT is a lot of businesses are about profitability, how much money you can generate from investing in this product vs this project. But for IT, you don't generate money for the company. Your job is to save money and be as efficient so everybody else can do their job. So using the data and other tools like that, our job is to step in and say, how much money can we save the company by doing X, Y, and Z. We have to almost flip that theory of money making on its head and do the opposite, right. How much money through this project or this project can we save the company in the long run?

Andrew: Right. That's fantastic. And that's a broad goal across Uber IT?

Ryan Patterson: Yes. Every company I've worked for in the past, before Uber, has had that same mentality. When you're working in IT, you're a service, and so you're a non- revenue driving. So you have to go out there and find the ways to be the most efficient for the people that you support.

Andrew: Yeah, no, for sure. And I think that beyond the cost- saving side of that, the efficiency also, I would imagine, aligns with business goals at this point because things need to happen faster. At some point, you can't keep up with the help desk tickets even if you hire 20 people, stuff is changing faster, correct?

Ryan Patterson: Correct. And that's one of the things that when you're building out a service or a system, you have to look for that flexibility and growth with it as well. You have to say yes, this works for our systems now, and then you factor in, are we going to grow 10%, 20%, or Maytech companies go through that phase of rapid growth. How much can this system or service that we're developing or deploying with our infrastructure grow with the company. And do you hire, do you compensate for that growth with employees or through automation or services that you're deploying? The services need to be deployed in one way or another, whether you have four offices or 200 across the world, you have to make sure you deploy systems that can scale easily, as well as be minimum investment on the employee time. A lot of people don't factor in the amount of money it costs to hire those 20 people.

Andrew: Right.

Ryan Patterson: This service costs a$ 100,000 dollars to run in our infrastructure. But on the other side, you're hiring three people that make much more than that just to do the same tasks. So you have to do a cost- saving analysis.

Andrew: Yeah, no, for sure. Total cost of ownership and that operation expenses obviously. Oftentimes dwarves the cost of the technology or the product or whatever you've implemented. So you in IT at Uber, you're spending obviously a great deal of time ensuring that that people, devices could connect to the systems they need, making sure the systems are available. I think at Uber, you also have this product you guys build with this, probably massive set of people on the production side of the house that are just jamming out software left and right and going down that road. We often talk to companies and people where there's this conflict between the IT team and the cloud team, and both conflict at the personality level, as well as the change threshold. And it creates tension. How does that work? How do you guys interact with the production team? Cause at some point, obviously, all this stuff comes together.

Ryan Patterson: Yeah, so for us there is a pretty strong line between our production team and our corporate team. To the point where I don't know much of about our production environment. They have their own set of tools that they use to get everything done. Our job is to provide everything below that set of tools. So DNS, DHCP, Active Directory, all of the different services that we provide. Then also the communication tools, Slack, Gmail, all the stuff that we use for our communication. So as long as we're doing our job effectively, we don't hear from them. And I think that's part of the IT motto is people only know you exist when something goes wrong and that's kind of one of the things that we do here. So on the corporate side of things though, we have a very great collaborative team and I think we have a very good structure going forward. Each of the teams are responsible for a horizontal slice of our infrastructure. So we don't have these vertical styles, like somebody that owns a platform and a network piece and so on. So we try to clean things up and organize them in such a way that we have these service owners who are responsible for networking, who are responsible for cloud and platform, who are responsible for operating systems. And that allows us to have specialists who navigate the entire thing and then provide services to a production team, depending on what they need. Do they need a Linux host or a Windows host? Well, then at that point, you can go to the same team and they'll be able to take care of you in the infrastructure. So for us, and in our change management processes are built in for all of our teams. We have a change board and we do all these communications to make sure that the changes that we're making on one level of these platforms is not affecting another. So I don't think we have a lot of problems on our side. I think we have great collaboration within Uber in terms of the corporate side of things. We work really well together to provide a world- class experience for our production side of things.

Andrew: Right, sure. What I just heard is good fences make good neighbors in many cases, and obviously between those domains there's security considerations, and a bunch of other stuff where you want those things to be separate, but that's great that there's harmony in that relationship.

Ryan Patterson: Agree. At the end of the day, if I had my way there wouldn't be fences at all, right. Having that ability to be open and be able to have those communications and talk, it allows for a lot more collaboration. I'm always... Who isn't, but for collaboration over everything else. So the more communication you can have between those different layers, the better you can build services.

Andrew: Yeah, agreed. I wasn't actually saying everybody should build fences.

Ryan Patterson: No.

Andrew: But it is from a virtual fence standpoint, obviously, for a lot of very good reasons, different parts of this architecture on the production side and the office side need to be separate. At least architecturally and so, obviously there you need fences. So Ryan, in terms of the data you collect, you give an example of automating DHCP reservations or something. Is it around man- hours? What is the data? And I guess the answer probably is going to be different based on the use case. What sort of data are you going after normally?

Ryan Patterson: So for us, you're right. It depends on the use case and what we're trying to accomplish within our infrastructure. So using the case of DHCP reservations. What we'd start by doing is collecting all of our tickets that we've done before in the past and just see how much time we're investing into a product to see if it's even worth it. And how we do this is we use the Atlasian suite, the Jira tools for this. And we have all the people that are performing the day- to- day operations tasks, add their hours to this. How many hours did you invest? How many minutes did you invest in these tasks? And then we have labels.

Andrew: They do that happily?

Ryan Patterson: No, but there're certain things... For me, there's two different ways that you can collect data. One is you can collect the data, have them put the information into the system so that you can build metrics. And the other one is you start enforcing those metrics on, hey, you need to complete this many tasks and you need to... Those are two different sides of that data collection process. For me, it's about data gathering right. At the end of the day, if you're working hard and you're putting the time and effort into it, then we shouldn't be expecting you to put in 40 hours of Jira tickets in a 40- hour week et cetera. We should be looking at these to say, hey, what was accomplished and what can we get off of your plate? So for all the different services we have on our infrastructure, not just DDI, but Active Directory, we have these metrics that we gather that are saying, hey, we're spending a lot of time doing group additions to active directory or DHCP reservations for BlueCat. And if we start seeing that, hey, 30 hours a week is being invested into these projects, what can we do from an automation point of view or a self- service point of view, right. I know the end goal for a lot of companies is to automate everything and then eventually not have a workforce, but that's not going to always happen. crosstalk So also self- service... There's always something going on. So self- service, putting that on the other teams. If somebody needs to make a DHCP reservations, configure your system to allow them to do it for their VLANs that they're working on or anything else like that, and offload that responsibility from your team so that your team can invest itself into other products that are going on.

Andrew: Yeah. What I always find interesting in that process is, and I'm thrilled you mentioned this a couple times. Whether it's self- service or, it's automation, you're thinking about the customer. And your customer, who's probably internal mostly, fine. So you're thinking about your customer and oftentimes the customer doesn't have the information that your experts that might do this manually have, or don't have the same understanding. And I'm not just saying of the UI or the API, it's more of a domain expertise. And so, when oftentimes when I think of successful automation, I think about who's going to execute this. Is it going to be executed from ServiceNow and then therefore, what information is in ServiceNow and if it's going to be done through some API from somebody else who's automating, deploying a virtual machine, and part of it is they want a reservation or something, right. So who is that user? And then what understanding do I believe they have? And now I'm going to craft that self- service for them to be successful. And I guess my point is, I often see things like automation being measured just with man- hour savings. But in some cases you're just sort of pushing the complexity to somebody else vs thinking of I'm like a successful... Did I meet my user's requirement? And I'm curious if that sort of thinking goes into your process as well.

Ryan Patterson: Definitely. And I think it comes down to just having constant engagement with your customers. Everybody thinks of a customer in your general sense. You go into a store, you purchase something, but anytime somebody is consuming a service of yours, in my opinion, that's a customer.

Andrew: Yep.

Ryan Patterson: Your job is to provide them a world- class experience in whatever you do. So for me, it's having those constant engagement with those customers. If you look at anything, Safeway, Best Buy, et cetera, they have reward cards and loyalty cards, and they're trying to build your trust. And for me on the infrastructure side of things, it's having those one- on- ones with them or with your stakeholders. A stakeholder inaudible something for a project, it's an ongoing thing that happens within your infrastructure. So for me, I know all the different teams that consume my service and what they're trying to accomplish. When we're doing project planning for the year, I always try to reach out and figure out what projects they're working on so that I can adjust and have them included into whatever I'm working on as well. So for me, it's a constant engagement process to work with my stakeholders, to understand how my service is being consumed. A lot of people lose sight of it and they have all these little things that I want to accomplish, but all the things that they want to accomplish within a given year might not actually have any impact on stakeholders that need something from that service. So for me, I try to base my projects on the work that I'm accomplishing on what my stakeholders need, not what I think I need.

Andrew: Right. So right now you're at your home. We're still in the grips of this pandemic, hopefully on the tail end of it. Hopefully those who want to, will be going back to the office at some point, but I'm sort of curious with you and your team. Has this shift in the way you're working disrupted your ability to meet with those stakeholders? Do you think your team's more efficient now or less efficient? What's been the impact?

Ryan Patterson: For us I think that the needs of our stakeholders have just changed. You know, we still have a lot of requirements that are needed. For example, we have a lot more focus on enterprise offices and building the infrastructure in our offices. And now we moved to a VDI solution in which we have all of our users logging into AWS or Azure for some one way or another. So for me, the requirements has just changed. We're just doing different things. We're getting in the same services, but just in a different facet of the business. So for me it's... And then once that eventually winds down, hopefully when this pandemic comes to an end, it's just going to change back to how it was in a hybrid sense, maybe. Maybe we have a lot of people stay on AWS or BDI or whatever it is. And then maybe we have half of them in the office. So it's just more requirements farther than less.

Andrew: No, for sure. And so, your team and obviously IT is involved in cloud as part of the solutions and services that you're delivering in across multiple clouds. How is your team dealing with the different skill sets that's often necessary to drive out on- premises infrastructure vs cloud infrastructure? Do you tend to have different silos of people working those two different areas or the service, and it might be some of it on- premises, some in the cloud, and it's more of a service focus?

Ryan Patterson: It depends on which layer of the infrastructure you're talking about. So for example, our cloud and platforming team is responsible for managing our on- prem and cloud- based infrastructure. So basically having all the way up until where you start hosting servers. Their job is to make sure that we can deploy a VM in any situation, whether we need it in Azure, or we need it in AWS or on- premise in any of our data centers around the world. So we have a team that's dedicated to making sure that we have a platform where we need it, when we need it. And then as we move up to the operating system, the same thing, right. We have a team that's dedicated to making sure we can get you the right office operating system configured in the right environment where you need it, when you need it. And then that continues up, service is the same thing. Do you need DNS in the cloud? Do you need DNS on- prem? Do you need Active Directory? What do you need and we'll find a way to use all this flexibility that we've built on the layers beneath in order to deploy what you need it when, you need it.

Andrew: Got it. Yeah. So your teams, the Uber IT teams are building the appropriate abstractions so that they can provide the same service from the perspective of the end- user. They have a VM, or they have an operating system, or services available, but that you have the optionality and scale to go deploy where it's necessary.

Ryan Patterson: Yeah. And a company of our size, flexibility and scale is a huge thing. And luckily for us, we have the teams, at least in my opinion, that can support that kind of thing.

Andrew: Look, you work for a technology company and you work for a company that... When I look at across sort of the different verticals of companies I work with in many cases the cloud, it's like there needs to be another team working on that, because those that are building these services on- premises, either from a time perspective, from a skillset perspective, they can go learn, they're smart people. They can go learn that stuff, but this is what they're doing. So you end up with silos working on- premises in the cloud. And I would imagine in technology in general, in those verticals, there's less of it. And I think that's what you're saying. No, of course, you're responsible for DNS and so therefore you deploy where it needs to be deployed.

Ryan Patterson: Yeah. And it comes down to another type of methodology that we use here is the idea that we have co- service owners as well. We don't have one person that's fully responsible for everything. So we have the ability to go learn other services in our infrastructure. So if I'm the sole owner of DDI within our infrastructure, that doesn't mean I can take any vacation. So one of the things we do is we make sure we're cross training and collaborating with other people on our team so that we have other people that could understand at least on a conceptual level, what's going on with the different services so that somebody can step away or move away, et cetera. And that's the same thing that happens for example, on our cloud team is that we have specialists in each of these different cloud environments on- prem vs off- prem. And then they collaborate to make sure that they have the knowledge gaps filled.

Andrew: Right. Good. Fantastic. Ryan, thank you so much. Super interesting. And I always enjoy talking to you because you just seem like you love what you're doing so much. And so, I can sort of hear your passion through the phone, I was going to say, but through the Zoom.

Ryan Patterson: Thank you very much for having me. I really appreciate the time.