Episode 8 - AI for Sustainable IT Part 2 of 2

DJ: You are listening to the Art of AI podcast with your host, Jerry Cuomo. And you are also listening to making of the SRE Omelette with host, Kevin Yu. Welcome back to the second part of our two- part series. We've been exploring the intersection of AI with sustainable IT operations, where innovation meets responsibility, with Jerry Cuomo, host of Art of AI, and Kevin Yu, host of Making of the SRE Omelette. We wrapped up part one with Kevin discussing the foundational role of data in creating sustainable and efficient SRE practices using AI to fine tune our approach to technology. Now, let's turn our attention to how we can apply these concepts in a practical setting. Jerry, in part 1, you mentioned the significant energy cost of a single ChatGPT prompt equating to a bottle of water. That's quite impactful. Could you provide insights on practical measures for more efficient energy use in AI systems?

Jerry Cuomo: Sure, DJ. The sooner, the better. And that's where I'll start. The sooner you can gain an understanding of the energy at play, the better chance you have of not being surprised come deployment and inference time. And when you look at what's happening in the AI community, Hugging Face has in their ecosystem interesting tools like carbon calculators. And this gets, when you think about food labels that describe the ingredients, it has become commonplace in these ecosystems, open source ecosystems that IBM is a big proponent of, to get this really complete documentation, including being able to put your AI on an energy scale and a carbon emission scale and weigh it to see what it might be doing. And then you use your judgment, based on that, as to, as you said, the right size, provisioning, or freedom of choice. " Actually, I think I'll take this one instead of that one. Now that I know that this is higher calories or higher carbon emission, I'll choose this." But it all starts with documentation and knowing what your options are so the more prudent we can be. And then question is this. How do you know? And I think in the precious data logs of SREs lies answers, mining that data to see what past behavior has led to. In compute and consumption, we can be more wise on how we provision things, when we provision things, et cetera.

Kevin Yu: The ability to put your fingers on cause and effect, see what's contributing so we have the freedom of choice to make that judgment call, that's amazing. That's what we want.

Jerry Cuomo: Yeah. And I do think, I want to emphasize once again, it's about AI and humans, and humans and AI, and not one without the other. I think sometimes when there's a new technology, we forget. We have these almost romantic visions of the AI taking over or overdramatize. And time and time again, we see that that rarely happens. Only in movies does it truly happen. And the technologies, when used correctly, and as humans, if we keep the same things that got us to where we are, we use our brains, we use our reason, we use our logic, together we can achieve great things. And to me, I like auto assist in my car. I'm a little leery about autopilot, but I think striving for autopilot along that journey brings some amazing things. The different warning systems that you have on your car today, when you change lanes, " Oh, that's in a blind spot," those things are technologies that need to be built if you're striving for autopilot, but by themselves as augmentation to the human are very useful. And for the right job, maybe full autopilot makes sense, but I don't think that's where you start. You start with the human in the loop informing a human then saying, " Hey, do you want me to automate that for you? I'll do it for you. Just press yes." And then if you trust the system, and then word trust is really important, if you trust the system long enough, then maybe you say, " Okay, why don't you take over today? You can do that for me."

Kevin Yu: That's a good reminder, use our reason and logic to balance AI versus letting AI run it all. Jerry, I'm curious. Sorry, DJ, for taking over this conversation. Do you feel we will ever get to a state where it is humans in an AI loop versus AI in a human loop?

Jerry Cuomo: As long as it's both and not one or the other, I do think there is an application and place for it. Again, the upside here, I would look at it less about doing things like losing jobs and see it more about getting time.

Kevin Yu: True.

Jerry Cuomo: Yes, at one point, intelligent systems were put on auto factory lines to rivet the metal in cars, and people did lose their jobs for that. But I think those are the same types of jobs now that you're seeing people are sitting behind computers now spending more time looking at quality. So because they're not looking at, " Is the rivet in the right spot?" they get to step back, use more human judgment, and focus on improving the overall quality of the car because they can now sleep knowing the rivets are securely fastened. And then also the risk factor too, that it's not always the safest job. So allowing us to do more meaningful tasks and more tasks that have higher value.

Kevin Yu: It is really about the impact to people's lives at the heart of it, isn't it? And one thing you mentioned earlier before I forget about it was repetitive tasks. It got me to think about a challenge, call to action, from Mike Hollinger, who I spoke to recently about sustainability. His challenge was how can we look at doing a compute only once? I think over the last decade, we have all been spoiled with the cloud, meaning I don't have to worry about compute. I can easily get what I need. We also worry less about getting it right in one shot. We have so much reason and logic that we try to find a way to make things work. Don't get me wrong. " Oh, that is great for user experience." But we have to start asking the question, why does this JVM restart 100 times in an hour? Why did this job fail so many times? The workload may eventually have worked because we have automated that and caught those and recovered. That is great. But if we also take the time to ask those questions, to understand what led to those failed JVMs and jobs, identify the factors that led to those failures and correct them, if we can do that, it would not only further help user experience but also reduce all those wasted compute. I think that analysis is where AI can really help us going forward.

Jerry Cuomo: Yes, I agree completely. And I think so many of the old systems and the techniques used in the old systems come back eventually to influence and improve the new systems. I remember working on application serving in the heyday of e- business and the web, looking at the world at that time of Java and building distributed systems that had hundreds of instances of applications versus the big one in the sky on the mainframe. I've never worked on a mainframe, but then when I started to befriend the folks in the mainframe and they started sharing techniques with me and I was like, " Wait, that technique will also work in the distributor world. Just a little different, but the same principle. So let me borrow that, bring it forward, and apply it." And I see that with cloud computing. I see that now with AI, starting to look at things like prompt proxies and secure gateways. All of these terms that we've had for different generations of compute are now coming in, just now with the AI moniker before it. But if you squint your eye and tilt your head, it looks similar to some of the problems we've solved so let's bring them forward.

Kevin Yu: Caching.

Jerry Cuomo: Yes, you can cache a response and keep it fresh. As long as it meets the freshness date, whatever that might mean, why issue the request a second time within a period of time? So all of these little tricks of the trade get brought forward, and it's cool when that happens and we see a stepwise improvement of efficiency, of cost- effectiveness, and of course a more sustainable output.

Kevin Yu: Blast from the past. Cashing was how I, and I would say we, the industry, got around to solving many of those certain bursts of traffic. And I remember using JCI to profile JVM, try to squeeze out the one gigabyte heap we had. That kind of practice, as you said, maybe coming back as we look at optimizing compute usage for sustainability.

DJ: Gentlemen, you know I'm still here. Okay, next question. Kevin, where is the best place to start to tackle those challenges? I'm going to take a guess and say data.

Kevin Yu: Data, DJ. Someone famous once said, AI eats data for breakfast. Data in my opinion has to be the starting point. With competing priorities we must be data- driven on what's important and what to work on. It is not just because Kevin or Jerry thought about it. Now, we're not just getting data and present it endlessly so we can build another dashboard, as Jerry also called out that made us laugh out loud for weeks. SRE don't need another dashboard. The journey starts by having a clear definition of what questions we want answered, then identify what data and metric we need to surface so we can answer that question with context, and in the dashboard if we need to. For example, a common question maybe, " I want to rightsize my compute so I don't provision more than I need to reduce cost." This is a popular one because it helps both to reduce cost as well as to be more sustainable. To answer this question, we must surface the consumption of compute use today, but then just having percentage of utilization isn't good enough. Back to dashboard. How often have you looked at the dashboard and go, " So what?" Because if I see 90% of compute and my SLs are good, I will celebrate. However, if I'm using 90% of compute and my SLs are bad, meaning my user experience are bad, it may mean we have a saturation somewhere else. So we need additional data to give context and lead to something we can act on, and we can be AI or human. By the way, this is what I saw you would do today. If you present them with just utilization metric, they will go look at if SLs are impacted and only act having that context. So why don't we make it easier by presenting the context there in the first place and then potentially automate that journey.

DJ: Thanks, Kevin. Now, Jerry, what advice do you have? Please share.

Jerry Cuomo: My advice would be, first of all, engage AI. It's there for you. When the spreadsheet came out, accountants learned how to use the spreadsheet. And it wasn't like the spreadsheet took accountants' jobs away, but I don't know of many accountants who lost their job because of the advent. But I think at the time, people thought accountants were going to lose their jobs. But no, it's a tool. It's a productivity tool. But you need to learn how to use it. So my first piece of advice out there is to become a prompt engineer. I think all of us have it within us. It doesn't require a degree in computer science to become a prompt engineer. Just little practice, little curiosity, and then a little repetition in getting it right. But at the same time, I would befriend a data scientist and share with them what you've learned. And go on a quest. Go shopping for the smallest possible model that is documented such that you can maybe train it additionally, a little additional fine- tuning to train it in your area for your tasks. And I think with that, becoming a prompt engineer and befriending a data scientist to help create that perfect model for your job, I think we will be able to both consume this wonderful thing that we know of as AI, get home early, get to the game on time, not miss it, and also be able to do amazing things that perhaps were very hard for us to do just by ourselves. Whether it be pinpointing that troublesome problem from terabytes of logs as an SRE and get you focusing on things not like the riveter, but the analyzer now, who can look end to end across the system and to see where the biggest opportunities to improve quality are, improve that user experience back and not just make your job easier, but nothing's better to work on a product that people love too. So spending more time on doing things that your users appreciate.

DJ: Thanks, Jerry. And the future?

Jerry Cuomo: I think the future simply is a place that we've been before, and we see this inflection from time to time where technology becomes commonplace. And think about the advent of the car and how it changed. I know growing up, I rarely ventured past Long Island and Pennsylvania. I grew up in New York City and that was my universe. But as time goes by, as technology got better, became the world traveler and got to see this planet thanks to technology. And I think we're going to see, we see that every inflection point of technology. It becomes more accessible. We saw the internet moment when Netscape came out. And it wasn't like there wasn't an internet before Netscape, but Netscape made the internet more accessible. And ChatGPT, while I don't agree with everything that is happening in the world of large language models and generative AI, it has created a Netscape moment and has. Didn't invent AI, but it made AI more accessible for people and also challenged people's thinking and how people invest their time and energy in corporations. And I think we're going to see that. People will get out past Long Island and Pennsylvania and see now the new world that this technology has to offer. What a crazy analogy, but that's where I went. All right.

Kevin Yu: Back to the Future, that's good moment for AI. I love that, challenge people's thinking.

DJ: Kevin, your future views?

Kevin Yu: I'll add to the part of our thinking, this means to arrive at this future, we must bring in our unique perspectives to the table. As questions of SRE, sustainability, and AI, if you find everyone in the conversation are part of similar experiences or are all from Long Island, look to introduce someone new because you need that diverse point of view to challenge the status quo and to validate what we're doing is right and meaningful. Also, remember Jerry jokingly mentioned to me that my grandchild will be puzzled if they can't cause my podcast on SRE somewhere down the line and ask the question, " What is SRE?" Meaning SRE has become something we just do without talking about it. Chris Hammett and Karim Youssef use the term sustainability consciousness to explain the same for sustainability, meaning it has become part of our behavior. It is not a point in time actively, not a conversation triggered by Kevin or by Jerry. It is simply what we consider as we build software. DJ, that is what I picture happening if we get this right. Oh, and definitely prompting engineers.

DJ: Well, we're just about out of time. I want to thank our two hosts for connecting the dots on these trending subjects. Thank you, Kevin.

Kevin Yu: Thank you, DJ, for having us.

DJ: Thank you, Jerry. Are your dogs, Lenny and Sheldon, there? I want to thank them too for being quiet during this episode.

Jerry Cuomo: Hey, thank you. It's a pleasure to be here, and let's talk soon.

DJ: Okay. Well, I hope you now have a better appreciation for how AI intersects with sustainable IT operations. It's certainly where innovation meets responsibility. This episode is a wrap. Check out the episode description for links related to the show. This is DJ, also known as Digital Jerry, signing off. See you again soon.