Episode 7 - Intelligent Facilities & Assets

Mike Hollinger: The future is now. Site reliability engineering to me is one of those places where it's almost too easy to fall into the trap of just taking action after action after action. We talk about proactive maintenance or proactive monitoring and all those things. I think sustainability is an extension of that premise. How can I plan for the workload in a way where I can still meet my SLA, but do it in a way that I spread the work out over cheaper, more sustainable compute. I think SRE is really positioned well because sitting at the point of observability, sitting at the point of architecture, sitting at the point of delivery, that's an essential pinnacle where you have that all seeing eye to be able to interact and push and pull and make differences I think everywhere.

Kevin Yu: Hi everyone. Welcome back to another episode of Making of the SRE Omelette Podcast. Sustainability is no longer just an ambition. Over the last few years, we're starting to see companies actively doing things and changing their behavior to be more sustainable. There is a lot of talk around improving how we manage assets, facilities, and infrastructure to drive clean energy transition and decarbonization. And I can think of no better person to speak on the subject than our guest today, Mike Hollinger. Mike is a distinguished engineer, master inventor, and CTO of IBM Sustainability Software. I'm so excited to have him here with us today. Welcome to the show, Mike.

Mike Hollinger: Kevin, it is absolutely a pleasure. I am so happy to be here. And howdy from Austin.

Kevin Yu: Oh, yes. I remember we first talked about having this conversation when I visited the beautiful Austin design studio. The circle is now complete. Mike, can you get us started by sharing with the audience how you arrived at this influential role?

Mike Hollinger: I have a background in sitting between subsystems, between different pieces of technology. So my formal education is in computer engineering. And for those of you that don't know that straddle computer science and electrical engineering. My first job in IBM was writing firmware for our hardware team. And I've moved in the layers of the stack until I was actually building an AI solution that we folded into the Maximo organization. For those that don't know, Maximo is a large set of capabilities now in a suite, Maximo application suite, that are designed to help people leverage their data and drive business outcomes. So being the AI guy amongst the asset management people, or being the software person amongst the hardware people, that's always been my career track.

Kevin Yu: We have been another thing in common. I also graduated with the electrical computer engineering degree.

Mike Hollinger: Ah. Nice.

Kevin Yu: But I haven't done much hardware engineering since I started working.

Mike Hollinger: Nice.

Kevin Yu: I got to fix one wire on one card one time, so I have that.

Kevin Yu: So Mike, I'm curious, as a distinguished engineer for sustainability portfolio, what does sustainability mean to you in the context of reliability engineering and facilities and assets?

Mike Hollinger: I love trying to define this by actually backing away from sustainability for a second. You hear the term ESG rattling around, right?

Kevin Yu: Yes.

Mike Hollinger: And we just lump it into, " What's your ESG report?" And it begins to define something in someone's head. But taking those three letters apart, like the, E, the S, and the G for a second, talking about what sustainability means in terms of environmental, in terms of social reporting and sustainable reporting, in terms of governance practices of all of the above. The premise of creating intelligent assets, the premise of creating a system that can self- heal, or that can tell you what's wrong with it, or taking a super complex machine or something as big as a building, and enabling that to live and breathe as one of our customers might say, that's what sustainability means to me. It's that ability for the system to not only just tell you, " Hey, there's an alert firing," but to offer a suggestion on what the next action is that might try to save energy or save costs down the line. Those things directly impact someone's carbon footprint on the one hand, but they also impact the bottom line. They impact the choice of where you're choosing to spend your money and they maybe help free up money for other areas.

Kevin Yu: I love that quote. To live and breathe. An intelligent system to give recommendation on where to spend the effort to reduce carbon footprint as well as impact on the bottom line. So Mike, is reducing cost, saving money, the intersect between ESG, SRE, and intelligent assets?

Mike Hollinger: Well, I think that money is a big piece of it. We are a part, as IBM, part of a for- profit business, but our customers are for- profit, nonprofit, government.

Kevin Yu: Right.

Mike Hollinger: And the incentives for all of those are different. If I talk to someone who's a roadworks manager for a given state, so Department of Transportation head, they're going to care about their constituents and the constituents' ability to get where they need to go or to transit materials and goods where it needs to go. That service that they're providing is a roadway or a bridge or a tunnel. In SRE, there's a similar premise of us providing a service to enable someone. And I think that there's an interesting parallel between the two, because if I do my job, you never notice that I was here in some situations, right?

Kevin Yu: Yes.

Mike Hollinger: That's, I think, a similar thing in both sides.

Kevin Yu: That is totally true. I always joke many people think SRE as firefighters. Call us when there's a fire. But I would love to be in the state where people think SRE as more of a fire prevention. Talk about sustainability is new. I'm curious, in your talk with customers, do you feel there was a collective aha moment that changed how companies look at sustainability?

Mike Hollinger: I think that there hasn't been an aha moment. It's not like suddenly overnight something changed. But there's this groundswell moment that we're in where you hear from constituents, you hear from customers, you hear from suppliers even, or people that are buying, people that are in procurement, this question of, " Hey, what is it that it's costing me? What are you having to do to produce this thing that's in front of me?" It's interesting right now because for IBM, IBM has been around for over a century. I was part of our hardware team, like we talked about. And we have a massive supply chain and we operate in many countries and many locations and many buildings and we've been tracking our environmental footprint for decades. We've been reporting on that for decades, and I think are one of those that has been leading the pack there. Now, outside of IBM, there are other companies that are similar taking steps forward that have been always at the forefront. In fact, I actually went up to Montreal and spoke with a customer who provided for the first time this past year, a combined ESG report and financial report. So not only were they reporting on the health of the business and projecting forward, they were also saying, " Here's how we're working on equity for our employees and our customers' different cities and places where we do business. Here's how we are pushing our suppliers to be transparent." And that I think is a little bit different compared to say, five or 10 years ago.

Kevin Yu: So it sounds like a collective curiosity from consumers, customers, supply chain, asking those questions that has led to the change.

Mike Hollinger: There's an interesting parallel to data privacy and the EU with GDPR or California in the US or almost every geography has something similar in some form where we see a similar upcoming or already the beginnings of this reporting being required. And all of this comes back to, at the end of the day, data. And it's trying to extract from that data some meaning or some insight that you can then drive an action. And being in technology, we love data. We love generating data. We have so much data that we call it exhaust data at a certain point. And it costs money to keep it around. And the interesting opportunity for a lot of our customers is how do I take that thing that I'm paying money to keep, and start extracting value from it, to drive my business or to drive transparency for my suppliers or so on and so forth?

Kevin Yu: Right, right.

Mike Hollinger: And we're starting to see that we can now start leveraging that.

Kevin Yu: Mike, I don't know if you know this. As site reliability engineers, we love data. And you're totally right. Often it is easy to just keep the datas because we never know when we may need it again. Bringing this love of data close to facilities and assets, can you share some of the challenges your customers may have faced?

Mike Hollinger: There's a problem that comes up in managing this kind of stuff because you might hear data and think it's all the same. It's not. And the broadest line I can try to start compartmentalizing things in is to say there's this IT world that a lot of us live in. There's also this OT world, or operational technology world, that a lot of the real world lives in. And in digitizing or modernizing at a certain point in time, if you take any given project, they're going to apply the best of breed technology at that moment. They might try to future- proof as much as we can, future- proof.

Kevin Yu: Right.

Mike Hollinger: And we make a choice. We know that we're going to do something that's going to be a trade off of better, cheaper, faster. You can choose two. And for a lot of companies, what I see is that they've made the effort to digitize parts of their supply chain or digitize their facilities and they now have this operational technology infrastructure, this OT infrastructure, that speaks in terms of SCADA alarms and all of the stuff that powers the smart lights that are in my building or the thermostats that are managing the heating and cooling units and stuff like that. And that's a different network and a different set of infrastructure and different requirements with different protocols from all the infrastructure that operates the IT side of the house, the back office, the day- to- day stuff that lets me walk in the door and have my security work because my badge is mine. Those are two different systems. So bridging that IT to OT divide, that's a challenge that we have right now.

Kevin Yu: Right.

Mike Hollinger: And just it's as simple as this system speaks in a protocol that's talking bits and bytes and buses and kilohertz updates. That system talks TCP/ IP and networking. How do I communicate this super high frequency thing to a network that's off in the ether somewhere in the cloud, whether it's private cloud, public cloud doesn't matter, but how do I do that? And how do I do it at a scale that matters?

Kevin Yu: Right.

Mike Hollinger: So that challenge ends up creating this, I think really tangible example of how edge compute can help address these types of situations. It creates this tangible example of how aggregating data and presenting it appropriately to drive a dashboard or to drive an insight or drive an AI model or whatever the case is. That's really frankly the challenge. If I have this massive haystack of every sensor from every system in every building worldwide for my enterprise, what do I do with that? And a similar thing on dashboarding. Whenever anyone asks me, " Hey, we need a dashboard." My immediate answer is, " Cool. What decision do you want to make from that?" And helping to work through that decision- making process, that outcome that you're trying to drive, tells you what data becomes necessary and what truly is going to be valuable for you.

Kevin Yu: My favorite question similar to yours is what question do you want answered?

Mike Hollinger: Yeah, that's a good one.

Kevin Yu: And with it we would know what data you need to give us your insights. Because like you said, there's so much data.

Mike Hollinger: Right. Right, right.

Kevin Yu: If we just present them a dashboard, people would simply go, " Now what?" Mike, while we're on this subject, can you tell a story of how a customer was able to take successful action from the data?

Mike Hollinger: Absolutely. I mean, one of the biggest examples I can give you is actually a public reference for IBM in the sustainability space, but there is a massive, massive bridge, the Sund& Baelt bridge in Denmark. Sund& Baelt operates this massive suspension bridge that is designed to last for multiple centuries. And the entire thing's instrumented, but it's in a sea, it's in the real world. It has to deal with sunlight, it has to deal with wind and erosion and rust and algae and all these different things. And the fusing of visual data with sensor data to be able to operate that mega asset. That's what allows the Sund& Baelt team to do things where they don't have to put someone in a rock climbing outfit to go out every single day to cover just a part of the bridge. They can get a much better view of the entire asset over time and choose where they're going to invest. That's a great example to offer.

Kevin Yu: Right, right, right.

Mike Hollinger: It's amazing because it's that confluence of data, of analytics, and then driving an outcome back to the business to drive, choosing where I'm going to spend time to fix issues or to proactively fix issues.

Kevin Yu: With data, we can be objective on our prioritization and make decisions.

Mike Hollinger: Absolutely. Absolutely.

Kevin Yu: Mike, there was a viral social media post a few weeks and months back, I think, that originated from someone taking a video of a rollercoaster and the video shows that the structure was being disconnected every time the rollercoaster goes to a corner.

Mike Hollinger: Yeah, I saw that.

Kevin Yu: I remember I was thinking, " Oh my gosh, this would be a perfect use case for the joint inspection to keep it safe versus being notified by customer or worse yet, from the news."

Mike Hollinger: Absolutely, and that's why I bring up the civil infrastructure example. I mean, if you take anything as big as complex as your rollercoaster or as a bridge or as a tunnel, the part you see is just a small bit of the overall thing. And if I have someone who's driving over a bridge or taking a ride, they're only going to get a glimpse of something. And that something may or may not be important. It may be okay in some situations to have some apparent issue because it's designed like that. The building that I'm in is actually a building that we affectionately call the rust bucket.

Kevin Yu: Oh.

Mike Hollinger: Because its facade is exposed metal and it's designed to have a rust powder coat on it because that's just a way to build a building 50 years ago. And it looks kind of cool, but it's brown. And it's okay because the entire facade is this rust metal thing. But you wouldn't want that same finish on a brand new glass skyscraper.

Kevin Yu: Right, right.

Mike Hollinger: That would be out of place.

Kevin Yu: Those are great civil infrastructure examples. I especially like what you said, that what we see is only a small part and requests much more instrumentation to get feedback for us to take meaningful action. So Mike, joining inspections, AI, and machine learning to give us insights of those data, I'm curious. A CTO of sustainability software, what would you consider a success and know we have arrived at this future?

Mike Hollinger: Well, the future is now. We live in 2023. I have a device that I walk around in my pocket that has a quad core processor, probably five different kinds of radios in it. Think about that for a second. I have three, four cameras actually thinking about it.

Kevin Yu: Very true.

Mike Hollinger: Laser scanners. I live in the future, man. And that success point is us acknowledging that we have this data, we have the ability to drive insights. We can take all of these really gnarly hard problems. Like in ESG reporting and environmental reporting, you might hear a term like scope 1 emissions. That's how much carbon are you putting into the world? You measure that by your power draw, by the things you make, and a few other fairly basic, this went in that went out. Scope 2 emissions is for your suppliers. For all the things you're consuming, how much carbon did they put in the world? So you ask the question to them and you do the math. You say, " Okay, I bought 17,000 of this. I bought 5, 000 of that. Multiply by this. Multiply by that. And here's the answer." My supply chain starts to affect me there. Scope 3 emissions, that's the important one.

Kevin Yu: Tough one.

Mike Hollinger: Because that's the tough one that says, " For my supplier's suppliers, going all the way back, what's the complete supply chain? What's the complete story?" And that's a data problem again. Knowing and acknowledging that we have this information available to us and then being able to orchestrate and pull it together means we need to be able to consume the data in the first place, make sense of it and normalize and drive it with a dashboard to help drive insights.

Kevin Yu: I thought you can't help me with that.

Mike Hollinger: Exactly, right? Second, the key thing is that if I have a dashboard and I take no action from it, what good is it? That's the key thing. So now driving action back out, that's what matters. It's driving those actions into the real world. And that's where in the sustainability world, that full feedback loop of, " Hey, I pulled this together. I had this insight." I can roll a truck and have someone go out and actually go touch that device and make a change or make a repair or make something to drive the feedback loop to closure. That's to me, I think when we've arrived. When we can close all of it. That's why I sit in sustainability.

Kevin Yu: Love that. The importance of driving action in the real world is what matters. Mike, in SRE you have this saying of actionable alert.

Mike Hollinger: Oh, right.

Kevin Yu: That is alert someone via Slack, PagerDuty, and let's the alert is actionable, and actionable means it can be by people or by automation. And the future is being able to inspect the outcome, the whole feedback, and understand the impact of those actions. I think you just described how we operationalize data and dashboard to arrive at this future.

Mike Hollinger: Yeah, that's always the hard thing, is choosing. One thing I joke with my team about whenever we talk about prioritization is we have way too many opportunities, challenges, in front of us. We can do a lot. It's choosing where are the two or three things that will affect the problem that's at hand that we can make the most impact? That's the hard question to answer.

Kevin Yu: Mike, I was speaking with our colleague, Chris Hammond, also in Austin, and I'll likely mess up the quote, but he said something like, " Design is done not when we have nothing else to add, but when there's nothing else to take away." And his challenge was for us to be really intentional on what is a must have.

Mike Hollinger: Oh, yeah, I love that. So one of the things that I am a huge fan of is design thinking practices. And there's a bunch of ways to do that. IBM has our practice, which is observe, make, reflect, or observe, reflect, make. And the order doesn't matter, but you must do all three. And it's that closing of the loop back that really is what enables you to take action and drive things like product market fit or drive responsiveness or drive a higher SLA or drive an outcome for a constituent or for a customer.

Kevin Yu: I couldn't agree more. Closing the loop, the feedback is so critical. So Mike, let's pivot to call to actions.

Mike Hollinger: Of course.

Kevin Yu: What would be your suggestion for a c- suite executive, who may be listening, wonder what they need to do to get started?

Mike Hollinger: The first thing is understanding what data you have. At the most basic level so many enterprises have data silos within them, either from mergers and acquisitions or different departments that came up at different times. That IT/ OT divide is the most basic one. That's a true, true issue. Understanding what's available to you and just getting that inventory is step one, because you might be surprised at what you're already paying for at the most blunt level. But then two is now that you have that, what do you need? Where are the gaps? Where do you need to supplement something? Where do you need to draw an inference to understand what the inner workings are of some processor or machine? A good example of that is there's a massive customer of ours that makes cement. They've got these massive facilities. And there's a core, core, core process where a machine the size of a Mack truck takes at one end a bunch of raw material, and at the other end, one of the core constituent pieces of cement comes out, this kind of fine powder. And you can't put sensors inside of it because it's literally these gigantic ball bearings and rocks tumbling around. But you can measure the output and you can measure the input and you can build a control loop. And if you're smart enough with the control loop, you can make a decision now that will affect your output in two hours. That inference enables you to drive a better result as opposed to, " Oh, this is wrong. Hey, we need to fix it and how do we fix it?" That basic, basic, what do you have, what do you need, what do you need to create? Those questions are questions that I think anyone would do service to ask their teams and to start providing answers for. Once you have all of that, then the next question is, " Okay, how do I take action from it?" I mentioned the idea of the feedback loop in terms of driving sensing information up to some higher level system. In our case, that's something like Envizi, which is our carbon accounting dashboard. But the actions that come out of that might drive, like I said, an actual person going into an actual physical location to go do actual work. It might drive shutting down a piece of equipment. It might drive reallocating some VMs because they're not being used. Those actions going out, similarly you need to know what you can and can't do or what you can push back to your suppliers and things like that. And those, again will become some interesting questions just to drive that choice of, " Okay, what's the top two or three things where you can take an action?"

Kevin Yu: So ask the questions of understand what data you have, what's available to you, what do you need to understand the gaps to take action, and lastly, who and how can take action from it to drive better results and outcome. Now, Mike, switching to a site reliability engineer persona. And you know we love dashboards. What words of wisdom do you have for SREs who are also inspired by you and want to take actions to contribute to the sustainable future?

Mike Hollinger: We talk about turning sustainability ambition into action. And site reliability engineering to me is one of those places where it's almost too easy to fall into the trap of just taking action after action after action. In a similar way you can end up in a bad feedback loop, where you're just responding to things, you're not being proactive in situations. In SRE, we talk about proactive maintenance or proactive monitoring and all those things. I think sustainability is an extension of that premise. Because if I project forward in what action am I taking and what's the downstream implication of that, then the most basic might be, " Hey, we have a spike coming because we're hitting the end of a quarter, so we need to ramp up or we need to adjust and what have you." But a different question might be, " Well, if this is going to happen, how can I plan for the workload in a way where I can still meet my SLA, but do it in a way that I spread the work out over cheaper, more sustainable compute?" Or how can I avoid doing undue work?" A basic thing. I talk about high velocity data pipelines with my team all the time. We have an ingest pipeline for part of the Maximo application suite and we're feeding millions of data points into it and driving a bunch of cool stuff. But the basic question there is how can I do a computation once? How can I avoid having to do work multiple times? So if the application architecture is such that I can take bytes and parts of the data flow, and save a partial answer, and then use that later, I can literally save the compute, I can save the cost, and it moves the problem around. Because sometimes it's easier to redo some math in Java because I'm sitting in a JVM and it's a simple equation, and I have the data in front of me and just real quick. But if I'm talking about billions of rows of data, it starts to add up. So I think those kinds of questions, that kind of thinking, that kind of stuff is a good opportunity to look at that broader impact. And those are, I think, the really gnarly problems that show up because again, I have yet to meet someone who goes out to design something with malice. No one ever sets out to do a bad thing. But over time, requirements change. Over time, the baseline assumptions change. What started out as a batch process that runs on seasonal basis now is being run every two minutes, 24/7, 365. Those requirements drive a need to go back and reevaluate the architecture and reevaluate how you're providing service. And that's where I think SRE is really positioned well, because sitting at the point of observability, sitting at the point of architecture, sitting at the point of delivery, that's an essential pinnacle where you have that all seeing eye to be able to interact and push and pull and make differences, I think, everywhere.

Kevin Yu: That's a thought- provoking response. Mike. I was just thinking we are all accustomed to doing laundry or running our dishwashers during off peak hours to save money. So why haven't we asked the same question on when we run those jobs?

Mike Hollinger: The big one there is frankly, not even when can I postpone the job, is when can I start the job and when does it need to be finished? Those are different questions.

Kevin Yu: Exactly. Not everything need to be executed immediately. And, Mike, bringing in the S from ESG. As we have a diverse team, when we introduce someone with a different background and when they ask, " Kevin and Mike, why are we doing it this way?" And if we can't answer it, it is a trigger for change.

Mike Hollinger: Absolutely. And that's why I love that idea of the combined financial report. I mean ESG, environmental, social, governance. We lump those together and that social aspect is super important. In fact, I'm a huge fan of blended multidisciplinary teams for that reason. And one of the most basic level, I can go back to the Renaissance period. Have you ever heard of a book called The Medici Effect? You should go look for it. The premise is that the Renaissance effectively happened because of this family, the Medicis, that were these patrons of science, patrons of art, where they brought so many people together from different backgrounds that you end up with like the Leonardo da Vincis of the world. And it wouldn't have been possible, the science advances of the time, the art advances of the time, simply would not have happened without this blending together. So a similar thing I think is true in operations and building software and building systems, where you should see the multidisciplinary background, we should see people from literally different backgrounds in the conversation. And that's, I think, essential because it offers you that chance to question, like you said, but it also offers you a chance to look at the problem. I talk about gnarly problems. There's this idea of trying to work on what's the most impactful thing. It's often the question is, " Hey, where can I have the most impact?"

Kevin Yu: Yeah.

Mike Hollinger: And those problems are not easy. Those are not going to be slam dunks. If they were, they would be done, right?

Kevin Yu: Yeah.

Mike Hollinger: And there's no clear answer. So when you start weighing these hypotheticals or you start looking at, " Well, hey, there's no past to build on here." Having that diverse perspective really means that you can have someone say, " Well, yeah, so I did this, because I worked on a project this one time, and it was just how we figured it out because this was a similar queuing problem," or something like that. And that's cool. We need that.

Kevin Yu: Well said. Diversity drives innovation and challenges the status quo. We can totally have a sequel episode on this topic.

Mike Hollinger: Oh, yeah.

Kevin Yu: So, Mike, in closing, I'd like to go back to the inspiration of the podcast. What would be your ingredient and recipe for companies to turn their sustainability ambition into action and outcome?

Mike Hollinger: I think that that premise of data, knowing the salt and pepper seasoning which you need there, the fact that there's these pockets and pools of different information is absolutely essential. And then being able to organize it and make sense of it. That's got to be the recipe to an extent. But that data capability, taking that, organizing it, building it into an AI model, building it into a heuristic that can drive your dashboard, to answer your question. You've got to have that kind of stuff. Otherwise, you're just throwing a bunch of stuff against the wall and seeing what sticks. And like I said, we live in the future. We live in the era of being able to ask for a car and in two minutes it's outside the door. Think about that for a second. I can pay for things without ever having touched anything, from this device that's this smart telecommunicator or whatever, Star Trekky whatever example you want to offer. We have to take advantage of that. And a good challenge to think about. Take these as a moment. This is not a phone. This to me is a very smart edge device. It has CPUs, it has sensors, it has telemetry, it has software running on it has a jail or hypervisor or whatever you want to call it. And it's a way to drive these things that you can use to notice, for example, is the building occupied? Can I do that in a way that's private? Can I do it in a way that's going to be sustainable for not just me as an enterprise, but for my employees or for my constituents? Those are the things that I think matter.

Kevin Yu: I love it. The future is now, everyone. Embrace the amazing technology and tools to gather, organize, and surface action from the data. Thank you so much, Mike, for spending the time with us.

Mike Hollinger: Absolutely. Thank you, Kevin.

Kevin Yu: I also like to thank you the audience for listening. I am Kevin Yu, Principal SRE of IBM Sustainability Software. See you on future episode where we continue to talk about how the practice of SRE can lead us to a more sustainable future.