Enterprise AI

DJ: You are listening to the Art of A. I. Podcast with your host, Jerry Cuomo.

Jerry Cuomo: Thank you DJ. Folks, there's a new buzz around AI in the enterprise. What took months to do now takes days, if not hours. For some tasks that only a few years ago required an expert data scientist, can now be done by a mortal with prompt engineering. Today I'm excited to have my friend Rama Akkiraju, an ex IBM fellow, who is now at NVIDIA as their VP for enterprise, AI and automation. Rama is an AI celebrity, and her accolades include being featured by Forbes on their list of top 20 leading women in AI research. Rama is a natural language processing expert, teaching AI to understand human language tones and sentiment. She has also led projects putting AI and ML to work for self- healing and optimizing IT operations. It's not uncommon for me to check in on Rama from time- to- time and just shoot the breeze on all things related to AI in the enterprise. Today we'll have that chat live on this podcast. And with that, I'd like to welcome my friend Rama to the Art of A. I. Welcome, Rama.

Rama Akkiraju: Jerry, it's so good to be back on your podcast.

Jerry Cuomo: Yeah, it's so good to have you back, Rama. So let's get right into the questions. I have so much I want to talk to you about today. So Rama, can we start off by you telling the audience here why you love doing what you do with AI?

Rama Akkiraju: There is so much excitement going on everywhere, thanks to what new things that came out since ChatGPT, pretty much every industry, every business problem can be re- looked at from the point of view of what can we do here with AI to significantly optimize, automate it? So for all the AI scientists and practitioners and engineers out there, these are really exciting times.

Jerry Cuomo: Exciting indeed. There's new energy in this space that I have yet to see before. Speaking about generating energy, Rama, can you share with the audience what you're up to these days at NVIDIA? What are you cooking up there?

Rama Akkiraju: Yeah. In my role, Jerry, I'm responsible for transforming the enterprise use cases with AI, generative AI. So I'm looking at a number of use cases. Say, take customer support, how can we bring the next generation generative AI- based virtual assistants, chatbots, to solve employee productivity, to provide these assistance? In the case of IT, if you look at that, how do we automatically resolve tickets or classify tickets, route the tickets and do better incident analysis and diagnosis? In the case of HR, you can think of applying this chatbots for HR benefits and those types of things. But beyond that, employee onboarding, e- learning and training opportunities, and various kinds of surveys, pulse surveys, that HR teams send out can now be more readily automatically analyzed for sentiments and summarized the comments from users and those types of things. So there are a bunch of use cases in HR. In the engineering domain, you can think of code assistance that's coming up, right code inaudible technical documentation generation. So there are a slew of those things. And across the board there are so many use cases, general productivity use cases in large.

Jerry Cuomo: Yeah, Rama, you're so right. In fact, I'm not sure if there's a use case in the enterprise where AI can't be of assistance.

Rama Akkiraju: Yeah, many use cases across the board.

Jerry Cuomo: Okay, Rama, you're known as an industry renowned thought leader in AI software, and you're at NVIDIA who is known as at least a hardware company with GPUs, et cetera. So how do you see that hardware and the software worlds meld together around AI?

Rama Akkiraju: Yeah, Jerry, you're too kind with your words. The exciting thing is that this whole revolution that is taking place in generative AI is really on this idea of accelerated compute because these big large language models that are really able to do these multiple tasks at the same time, and also multi- model as in bringing in text, visual images and all that. All of that, you could not train these models at the scale if you didn't have the accelerated compute, the GPUs powering underneath. NVIDIA is really the supplier of all of that accelerated compute to pretty much everybody out there who is building all these large language models and generative AI models. So it's exciting times in that sense. There's new advancements happening at the chip level on a daily basis. A lot of companies are leveraging these both for training and inferencing, data processing and so on.

Jerry Cuomo: Let's talk about the advances in AI for the enterprise. Now, Rama, last time we worked together, I think it was on AIOps. It was all about training and labeling, and that all took time and effort, energy, et cetera. So how has gen AI changed that game?

Rama Akkiraju: So interestingly, the foundational models are getting better and better. That's what this generative AI models and large language models are able to do. The foundations of understanding language and the amount of data that they're seeing has grown so much, and the power of these neural network models in terms of their depth and the number of parameters that are being used have also grown. So both on the amount of data that they're learned on and the ability to reason with that data to, if you can call it that, reasoning, and the ability to solve tasks out of the box, that has changed the game. When it comes to enterprise, still there are lots of questions around how the models that are trained with public data, they won't understand the enterprise context and enterprise knowledge and enterprise data. So there is still that problem of these large language models, the foundation models, still have to work with enterprise data. They still have to be guarded very carefully for hallucinations, toxicity, fairness, bias, and all of those types of things. So in order to really use it for enterprise context, you may still have to customize with enterprise data or maybe use information retrieval types of techniques, which will get the fresh context or fresh set of data to work with, and then pass that retrieved documents to large language models to summarize.

Jerry Cuomo: Okay, Rama, let's talk a little bit about fine- tuning models. Can you comment on let's say the threshold of when it's worth fine- tuning a model with custom data versus let's say using a model right out of the box?

Rama Akkiraju: I would say the following, it's one size may not fit all. There might be use cases where you may just be able to use foundational models out of the box. You may not have to do a whole lot. Things like summarization, the content generation, those are general language tasks enough that if you give enough information in prompts, and if that prompt size, token size, suffices then you may not have to do customization. But there might be use cases where you may have to customize, specifically in very specialized domains like in healthcare or maybe telecommunications in order to really understand, teach the system the language of that domain. You may have to customize. But I would say Jerry, that retrieval augmented generation where you use information retrieval solutions in the enterprise to extract relevant documents and to generate snippets, write snippets from those and pass those into prompts, and then use the general purpose large language models, it may solve pretty good chunk of problems. So one has to really ask themselves what my use case is, what works, and invest in customizing only if you need to customize, because it does-

Jerry Cuomo: Yeah, yeah. It sounds like customization should not be the first stop in your AI journey. Yeah, I agree. So Rama, you said something to me that made me laugh for a week, so I want to talk about that now. You said that you were involved with AI before it was cool, and now that it is cool, you really no longer need it. Can you explain and talk to us about what you mean by that?

Rama Akkiraju: That is an interesting question among everybody's mind, are we all prompt engineers now? Do we still need data scientists in the world of generative AI? Well, I think there is a role to play for everybody. Somebody still needs to define the problem and really understand the data and figure out what techniques, whether this is a customized model or a general purpose, large language model that would apply and interpret the results and ensure that they're tested for all of the corner cases and toxicity bias and all of those things. So data scientists still play an important role in all of those things. But yeah, the programming language and to interact with large language models nowadays is all prompt. So there is this new role now, prompt engineers who will be responsible for fine- tuning and optimizing the behavior of these generative AI models. So data scientists and prompt engineers have to work closely together to solve problems.

Jerry Cuomo: Yeah, nice. Rama, can you take out your crystal ball and share with our audience what you see ahead?

Rama Akkiraju: So these are exciting times for sure, Jerry, as you and I share notes from time to time on what problems we're solving. There's going to be, especially the multi- model, as in that speech, vision, text and all of that, that's opening up a whole new set of fields in art, creativity domains and such. So that's going to be a lot of exciting things. And employee productivity, take Office 365 Google documents kinds of area where you give word- based, text- based prompts, and you generate your PowerPoint presentations or you generate your marketing campaigns and such. Those kinds of use cases are going to be super exciting and likely productivity boosting as we go on. There probably will be more personalized, tailored experiences that we can build now because of, again, this multi- modality that we can now support with the text and images and vision. So more immersive experiences might be possible in the future. Of course, all of the enterprise problems around virtual assistance and augmenting human capabilities in domains where there was a lot of text data, that can now be more readily summarized and brought to fingertips. All of those would continue to evolve to probably to better places. There will always be room for more classic traditional machine learning in the area of predictive analytics and such, where structured data needs to be analyzed. But then at the intersection of structured data and unstructured data, a lot of cool things are happening where you can now interact with your data. You can ask more insights of your data, that may be in a combination of structured data and relational database and in documents. So these are some areas to watch out for, I would think.

Jerry Cuomo: Wonderful, wonderful. Well, Rama, you never disappoint, and thank you so much for the wonderful conversation and for joining us on the Art of A.I.

Rama Akkiraju: Yeah, it's a pleasure, Jerry, always. I look forward to your future podcast on this topic and look forward to listening and learning.

Jerry Cuomo: Well, that's it for today. I've taken the liberty to include a link to a YouTube video of Rama being interviewed by the New York University Stern School of Business in a more in- depth discussion with Rama on subjects related to AI in the enterprise. You may have also noticed that I've started an article series, called The Art of A. I. Article Series, of course. I've included a link to that as well in the description section of this podcast. Once again, I'd like to thank Rama for joining me, and I'd also like to thank you all for your continued support and interest in these podcasts. This is Jerry Cuomo, IBM fellow and VP for Technology at IBM. See you on an upcoming episode.