Open source software, cybersecurity, and OpenSSF | Jamie Thomas, Chair, OpenSSF

Luke: In this episode of In The Open, we're pleased to bring you a conversation with Jamie Thomas. Jamie is the Open Source Security Foundation Board Chair and General Manager for Systems Strategy and Development at IBM. She's a rare talent and an accomplished leader with an expertise in hardware, software, and cybersecurity. Before we welcome Jamie, let's say hello to my co- host, Joe Sepi.

Joe Sepi: Hey Luke, how are you my friend?

Luke: Good, how are you doing, Joe?

Joe Sepi: I'm all right. I'm all right. I was excited about spring coming and then it dumped a bunch of snow, which was beautiful, and now it's melted. I've got my motorcycle ready on spring. I'm really itching for it. It's a decent day out there today, but spring can't come soon enough. How are things by you?

Luke: It is, it's 50 degrees out there. It does feel like spring. I just got new altering tires for my pickup truck, so I'm ready for the mud. No problem.

Joe Sepi: Yeah, you get a lot of work to do out in the woods and get muddy with your new truck. That sounds great.

Luke: I'm enjoying it. It's a fantasy come true. But without further ado, let's welcome Jamie, our guest.

Joe Sepi: Hi Jamie.

Jamie Thomas: Hello. Great to see you, Joe and Luke, and thanks for having me In The Open today.

Joe Sepi: Yeah.

Jamie Thomas: Yeah. I'll just chime in on the weather here in Raleigh. I was playing golf last weekend. It was quite warm. And now, of course, I'm wearing a sweater, so it's pouring in. Strange weather we're having this year.

Joe Sepi: Yeah, it really is. It seems like it gets stranger every year. That's a whole another conversation.

Luke: Yeah, there might be something to that. I feel like there's a trend there.

Joe Sepi: I think so, too. So yeah, welcome Jamie. Thank you for joining. Maybe we can start off with a little bit of a self introduction and go from there.

Jamie Thomas: Okay, sure. I'm really delighted to be here. I'd have to say I think I have the most interesting job in IBM, but I'm going to self- declare that. But as you said in your introduction, I started out as a software programmer in IBM, most of my career in software. And then I moved over to the land of hardware, which of course doesn't run actually without software. So, there's a little bit of an entanglement theory there. But today I manage the advanced to processor and systems design from the design, to the development, to the manufacturing. So, I do get involved in the whole life cycle, including the manufacturing and supply chain, which has been pretty exciting in the last few years. Along the way, I developed a really keen interest in product innovation, of course, given the kind of teams that I manage and product security. And so, about two years ago I actually assumed the ownership of IBM's enterprise security mission as well. And it was really, I think founded on my long- term involvement in software and the hardware product security. And that office really owns the CSO role and the team is responsible for cyber operations and product security.

Joe Sepi: That's really fascinating. And those two things really do intertwine, right? To the hardware, the software security, it's all interrelated.

Jamie Thomas: Yeah, absolutely. And I've had a keen interest in that over my career. I guess it was when we did WebSphere many years ago that we really conquered this notion of how do you really meld the software with the hardware and create this differentiated value. And it's interesting because that's about the same time we actually started doing a lot of our open source work as well.

Joe Sepi: Yeah, that's interesting. We definitely will spend a lot of time talking about security today, but I think it might be cool to dig into some of your open source background. I know you said you started out programming and stuff and you've mentioned Eclipse. Tell us about your open source story.

Jamie Thomas: My first involvement in open source was probably with the Apache web server work that we did. And that was the early work that we did in WebSphere when we decided we wanted to do, take an open source approach to that endeavor as opposed to a IBM proprietary approach. So, we started on that. We did a lot of, obviously we learned a lot from that effort. And then our next big effort that I was involved in was the Eclipse Foundation, which is where we took a lot of our intellectual property around the tool chain, particularly for Java. And we spun that out into the Eclipse Foundation. I ended up managing what was then the rational organization. We had acquired Rational and they were very keen in the construction of a lot of the concepts and methodology that went behind this. But the important thing was we really were intent on creating an open ecosystem for Java programmers. And we did feel that this endeavor was successful and allowing more Java programmers to participate in an ecosystem that added value, as well as work with IBM on the work that we did after that. I guess the next big thing we did is we, IBM, invested quite a bit in Linux. And I was actually still in WebSphere, a lot of the... during when that first started, but one of the fundamental things we realized is that Linux did need to be a cross- platform play. And so, we invested to make sure that Linux was a first class citizen with our hardware platforms, particularly our Z franchise. And of course that's been beneficial to this day. I guess the culmination of that whole story is the acquisition of Red Hat, which was quite a big deal and I was really pleased to see the Red Hat team join IBM. It was really a culmination in this long- term effort around Linux. But in between that, of course, I've seen many different open source projects, but historically those are the three that I think really pivoted IBM into this direction.

Joe Sepi: Yeah. Those three are huge and I think oftentimes people don't realize, and I take every opportunity to share with them IBM's long, rich history and open source. Those three things alone, Apache, Eclipse, Linux Foundation, I mean those are huge players in the open source space and we were integral in spinning them up and getting them going. And I think too, the important thing that I think you're touching on there as well is the whole concept of open governance and having everybody participate equally and having no one have outsize influence in the development of these, the software and these foundations and IBM's been key to that work in a variety of ways. So, it's really, I love sharing that story whenever I can. So, yeah, where do you want to go now, Luke? Where should we dig in?

Luke: I feel like maybe we should defer to Jamie, but my inclination would be let's talk about what the landscapes and because security is such a big deal and we're talking about open source, let's talk about the Open Source Security Foundation and your work there and give us the layout of that landscape.

Jamie Thomas: Okay. I got involved in the Open Source Security Foundation late last year because the foundation itself recreated itself. It created a new governing body, if you will, of senior executives across the many firms that are participating. And the aim of the core objective was to ensure that open source is secure for all of those downstream consumers because the world depends on it. And we set forth a charter of some basic concepts of what we wanted to achieve. And interesting enough, it wasn't long after that Log4j occurred. So, Log4j of course was a large security event affecting one of the most commonly used open source modules in the industry. And it did suffer from a critical vulnerability of course, but the amount of use of Log4j is really what struck the industry at large because everyone had to understand where it was, how to patch it from a delivered software perspective, as well as for anything that resided on infrastructure within an organization that would include cloud services or internal infrastructure, et cetera. So, it was different in terms of what had already occurred from a cyber perspective in the industry. And I think this then only reinforced the fundamental principles that OpenSSF had already embraced and it started to embark on, but it did in a big splash. You can't imagine the concern that occurred from the world at large from financial institutions, to healthcare organizations, to critical infrastructure, for industrial infrastructure because Log4j was so highly prevalent in our code. It was a real test of everyone's ability to patch expediently to also execute cyber operations. It did solicit a lot of governmental attention. So, you can see out there that the White House did convene a meeting around Log4j and what we needed to do to further protect ourselves. I attended along with many of the other members actually that are on the OpenSSF. You can see out there in the White House press articles who attended including the OpenSSF team, the Apache Foundation, et cetera. So, it's a good collaboration because it's going, it will require industry collaboration in my mind to really conquer this kind of problem and to make sure that we can, the adults in the room, if you will, to provide the insight, the skills, the best practices and the oversight recommendations that allows open source to be effective going forward. Now one of the key points that we did make is that this notion of curated open source, which we are really big sponsors of, is absolutely critical in this journey. Red Hat is a great example, if you will, of curated open source. And when we're consuming open source from an organization like that, they have developed the approach, the acumen if you will, to be responsible for a lot of these different aspects of open source delivery into enterprise organizations. We at IBM, obviously, we're taking advantage of Red Hat ourselves on a big scale and of course that's one of the interests we had in Red Hat in terms of enabling that capability for the rest of the world. But I think it's going to be a double- edged sword working with the curation aspect of what a firm like Red Hat can do and also of course making sure that the new open source endeavors start secure from day one and that they have the ability to stay secure going forward.

Joe Sepi: Yeah, there's so much to dig into what you just said there and I think it's interesting, it's easy to say with Red Hat and OpenShift enterprise ready and be all tested and all those sorts of things, but this is where it really means something in terms of being secure and curated open source. But I wanted to touch on, you were talking about Log4j and how OpenSSF restarted itself. Maybe you could touch on the thought of these sorts of things happen and everybody focuses attention on them and then they fade out and then what happens where the OpenSSF can come in and help keep our attention on these things?

Jamie Thomas: Well and I mean, I think that's a really important point because in December there was Log4j and then not too long ago unfortunately there was the Russian invasion, which is also another flavor of cyber concern if you will. So, these things are never ending. We have to make sure that we have a systemic focus on improving what needs to be improved regardless of the latest event du jour, if you will. And I think that's really the important endeavor of the OpenSSF. It's making sure that we have put in a systemic framework for education and skills, a systemic framework for identifying the most important projects and making sure that they have the tools, the best practices and approaches to ensure security despite who the contributors might be over a period of time and deploying all the techniques we can to ensure systemic runs along with what I describe is the never ending opportunistic landscape of cyber. And so, OpenSSF is critical to this. In IBM, we also have a very honed approach, if you will, for our own systemic operations. But also we have upped our contribution, if you will, for this kind of endeavor with OpenSSF to make sure that we are there and where our opinion is rendered along with the other key players in the industry that are participating and the players are out there for everyone to see. But it's Microsoft and Google and AWS and Meta, some of the financial organizations, GitHub. So, it's a very important set of players, I think, that are participating in working together.

Luke: I was going to ask that Jamie, so we hear a lot lately about software bill of materials. I'm wondering how that fits into the play here because it, I'm imagining when we're talking about opinionated stack with Red Hat in a way obviously that's what they're doing, right? They're really looking at what's in there and making choices and hardening that. But how does the software bill of materials play into the ecosystem and what everybody else is going to need to do?

Jamie Thomas: Well, I would say that there's probably many different opinions as to what the software bill of materials should actually contribute over the longer term. That's perhaps that it really has too many roles and definitions if you think about it. But to me, in the world of software delivery, the most important thing is you understand the origin of your software, that you understand for every discrete service that you're building or for a package set of software, you understand the DNA of that software. And the DNA is really represented by the software bill of materials. Now, there's been a lot of discussion about revealing the software bill of materials to organizations. There have been some discussions in the executive White House orders as to how we would use the software bill of materials. From my perspective, we need to use it to have fidelity of operations around software delivery, understand whether that's that these software projects that we're consuming are mature, that they're secure. In other words, my advice would be consuming software that's only controlled by a few contributors is risky, right? It's not that it might not be good software, but you have to look at your risk mitigation. So, to me the SBOM is going to give us that insight over time. I had rendered my opinion that I don't think SBOM should be made public. And when you think about that, if you think about it from the context of an organization like IBM, why would I necessarily share, for instance, where Log4j resides. Ahead of having a patch for Log4j, it almost says there's an opportunity here to come attack this particular asset or whatever. So, I don't know that there's value in having public SBOMs. I think there's value in using the SBOM information intelligently and to the benefits of improved software quality, security and delivery to all the consumers that are using it.

Joe Sepi: That's really interesting. And you've mentioned that the White House meetings of a couple of times and that's actually how I thought to mention a loop that we should reach out to you because I work with the LF people often OpenDesk Foundation and Whatnot and they mentioned you being at the meetings and how great you did and I was like, " Oh, we got to talk to Jamie." So, I'm curious to hear a little bit more about the meetings in general, but also in some of our previous conversations you had talked about the sort of two sides of this sort of security issue. And I'm not sure if I'm phrasing this right, but like cyber and vulnerabilities perhaps, or how would you phrase that?

Jamie Thomas: Well, I say that there's two interrelated topics if you're managing security for an organization, like an IBM in particular, right? Because not everybody's quite like IBM and that we're delivering a lot of software to clients. But in our world, cyber operations is job number one. You have to have an inherent ability to protect yourself from the hackers, the bad guys, whatever is happening. You have to have ability to protect yourself from bad decisions that unfortunately developers make every day. And we can assert best practices but not everybody's going to follow the best practices. A lot of it has to do with training and knowledge. So, in IBM, we build a very automated cyber operations capability that is very layered and we're able to handle a large number of events on a daily basis and intelligently do something with those events. And the goal is to implement zero trust cyber operations where we assume that people will make mistakes and that the bad actors will take advantage of those mistakes. And that's the principles of how we want to run cyber operations. But if you think about it, when you're in a system like this, we're a company full of innovators and so our other job is to make sure the innovators understand how they design secure software from day one. And we have a policy call secured by design that's part of our formal IT security policy that all of our teams are expected to follow. It's a series of best practices, CICD framework recommendations, and outcomes that we expect these teams to achieve regardless of they're developing a product that runs on premise or whether they're developing a product that's going to run in the cloud. So, they all are expected to adopt these. One of the observations that I would have, and we do the same thing for hardware security, I was very much involved in Spectre Meltdown, which was one of the bigger events we saw in hardware security a few years ago. And you have the same objectives there, albeit a little bit different in terms of how hardware comes to life if you will, as opposed to the software. Software's a lot more fluid, a lot more dynamic. But one of the observations I have is that we need a further state of the art development I think around CICD pipelines where they're not only automated but they're also, there's this element of similar to what we see in cyber. If someone does something wrong on their iPhone or their laptop, I get a flag in cyber operations that they downloaded something that's not allowed. So, while I've got guidelines, I've also got insight into what they're doing, right? And so, I think we need to have CICDs that have a lot more insight that are able to flag things that are perhaps of concern that we go out and we address. And it's not just that we have this with code scanning, but we need something I think more intelligent than even code scanning because that's what really is helping us in cyber operations. We can have the rules, we can have the impediments, but if someone does do something, if they go out to a website that is hostile, we're immediately notified, and we can actually shut down the device or certainly take other actions. So, I think there's some maybe a really big opportunity here in this evolution of these CICD tools and everything to have something much more sophisticated.

Joe Sepi: Yeah, that's fascinating because we are talking a lot about AIOps. And I think a lot of the times we're thinking about bugs or logs and analyzing performance and all those kinds of things, but it's interesting to think about that applying to security.

Jamie Thomas: Well, and it's not to penalize users, it's really to alert users. So, for instance, during the pandemic we had a lot of individuals, they were all working from home and we saw instances where individuals were downloading education software that they were trying to help their child with, but the software had these vulnerabilities that were not acceptable to us. So, then we were able to go out and tell them, be careful or we're going to basically we rebuild those machines frankly. Not everyone can be a cyber expert, right? And that's why you have cyber experts that are paid a lot of money. And likewise, not everyone's going to be an expert on some of the things that are really bad programming practices versus good, and certainly everyone's at a different evolution in their training. So, I think this is going to be interesting, is can we have AI for the CICD, have a lot more intelligence that we can then alert organizations so that they can help the individuals be more productive.

Luke: What comes to mind when I'm hearing this too is the enterprise perspective. And I think, it's a need actually to look at IBM because this is really where we shine, right? Because with hybrid cloud and the way modern applications for enterprise are built, they're spanning on- prem, different clouds, maybe legacy co- located stuff. So, it's like a huge attack surface and so many plates to spin. So, I guess my question, I don't know if I have a question there exactly, but I guess comment on that, what is the modern enterprise? I imagine it's a huge attack surface, right?

Jamie Thomas: Yeah.

Luke: And there's these popular things like whether it's Heartbleed or Specter or Log4j that capture people's attention, but all those other things are still there all the time. And those plates have to be kept spinning even if they're not in the zeitgeist.

Jamie Thomas: Luke, you really bring up a really important point. What I've always told our clients historically when outsourcing became popular, that just because you outsource something doesn't mean you don't own it. So, you better have an SLA, you better have a handshake and agreed upon set of metrics with your outsourcer that says here's how we're going to work together. The clouds are no different. They're different form of outsourcing, in my view, right? You have outsourced something to someone else. Now, it depends on how you're using the cloud. They may own an entire application for you. You could be in the Salesforce- type situation, which IBM's a very large user of Salesforce as an example. Or you could be in a case where you're just renting infrastructure and you're responsible for the application on top of that infrastructure. But at any rate, you have to have a clear SLA policy and agreement with your outsourcer. I mean it's a different form of outsourcing in my mind. One of the key things though is that not every organization keenly understands who's responsible for security in these kind of paradigms. And the owning organization is ultimately responsible for security. So, once again, you have to have the right policy in terms of how you're using these clouds. What we have to do is we have to, as a multi- cloud vendor ourselves, we have to either have a really clear agreement with the vendor as to what they're doing for us and or many cases where we're running assets out on other clouds like Amazon or Azure, whatever it may be, we do have endpoint monitoring on those clouds as well. So, we have observability as what's happening out there. One of the things you see when any of these vulnerabilities come to light, whether it's SolarWinds or whether it's Log4j, the first players at the table are the Bitcoin miners. I just think Bitcoin miners are in the scheme of what can be done. They're a little bit harmless and they're just trying to take over to capacity to find the Bitcoins but it becomes disruptive. But those will be the first actors that show up and you have to have the ability to detect that because the next set of actors are the ones that really have much more malicious intent typically. So, I think that if you're a organization who has a multi- cloud environment, the bug stops with you. You're responsible for your security. And you need to assume that is the case. You need to assume that you have the right vendors that can help you with that model because it does become much more complex. And every one of these situations I've seen it is natural to see that the cloud environments are the first that can get attacked first because they're more exposed. Something that's behind multiple layers of protection, firewalls, et cetera, can be attacked for sure, but it's going to be a little bit harder. You see what I mean? So, in this hybrid environment, you've got to be really astute to that. You've got to be astute to who controls your applications, who works on your applications. One of the things I think unfortunately that this whole terrible situation that we're seeing on television every night with Russian and Ukraine has brought to light is your location strategy for where are your apps being developed in a cloud environment. There's a great productivity coming out of these environments. So, I'm not saying that there's no productivity here, there's tremendous productivity, but who is doing things is sometimes a lot more obscure to you, to who actually is the administrator, who actually is developing your application, who's securing your application if it's an application that you're purchasing. These things become, they're very important but they become a little bit more difficult to trace without this, once again, this close SLA policy with your provider, this understanding that ultimately you own it. I always tell the developers, if you ship open sourced, it's your product, you can point to other projects and everything, but at the end of the day, it's your product, it's your offering, your service that's consuming that.

Joe Sepi: You mentioned about open source and shipping open source and I wonder, we had talked about the top 500 projects coming out of a Harvard study and the work that says going on OpenSSF to identify top projects to be focused on. Maybe you could talk a little bit about that.

Jamie Thomas: Well, we did get a readout most recently from the last Harvard study, the most utilized and important projects. And that we have to take that data and understand how we help the top projects become more mature should they need the help. So, there'll have to be an assessment done of the top projects to understand the contributor map, the kind of tools they're using, the kind of challenges that they have and how we can accelerate their maturity if needed. Some projects will likely be fine for different set of reasons. And one of the things that I'm doing of course is matching our most used assets inside of IBM to those projects to understand that map because there can definitely be a difference between the most used and where people are currently contributing. And it's just natural because open source has been around for a while now. So, there's a maturity map of open source just like you would see with any software. So, you have to look at both the dimensions of both strategic evolution of what you need to do to drive your business forward as well as what you're consuming. That was one of the things that came to light with Log4j because I think it's been around for 21 years, so it's had plenty of time to proliferate. I work on quantum computing a bit with the team in IBM. It's not like quantum computing, it's not quite that complicated in terms of the module and what it does, but very important in terms of how it's used and how widely it was used. So, that's a little bit about what the OpenSSF hopes to achieve with that analysis of the top projects.

Joe Sepi: Yeah, it's interesting too. I work in the Node. js space and with the OpenJS foundation and tangentially with the OpenSSF and such and this Alpha- Omega initiative is really interesting and that we're looking to take advantage of it. Node. js, I think we have a proposal in, I can't share any details yet until it's all public, but I'm excited at the aspect of being able to work with the OpenSSF and try to make positive changes.

Jamie Thomas: And those, I think, are important projects kind of looking at this problem from different angles. One focusing on a more broad application of how we can impact the projects and another on a more focused application for those projects that might be deemed the most important. But those are just examples I think of where the foundation is really starting to get a toehold, if you will, of what can really be done. I think it's going to be important for us to take practical steps that make an impact and that's going to be one of the challenges because there's so many things we could go do, we're going to have to make sure that we select the projects that can have the most impact in the shortest amount of time.

Luke: I have a two part question, Jamie. So, the first part is when we were talking with Brian Behlendorf recently, he did mention these initiatives and he didn't give us an exact number, but he mentioned that if there was an effort to harden these top projects, it would cost you maybe some tens of millions of dollars to harden the top projects and help them get to that point point, which seems like a lot of money, but when you compare it to the numbers you hear of the losses through hacks and different things that are going on, it's really not a lot of money. So, my two- part question is when you're talking with companies out there, clients, your peers, I have this impression that businesses are reluctant to invest in security because they don't see a direct ROI. So, first question is, do you see that changing because of the public nature of all these hacks that are happening, like Colonial Pipeline comes to mind and so is there a mindset where folks are more willing to invest in security? That's the first part of the question. And then the second part of the question is about what we were talking about this supply chain. You often hear that companies are using open source but not necessarily putting that effort to contribute to it. So they haven't maybe jumped that hurdle. Do we see them putting this effort in just to help secure open source as well? Because one of the things I've heard, I know I'm going a little bit long in my question, but I hear about once there is a disclosure of a hack, it's not like problem solved. It's like not everybody's patched it, there's high levels, like 40% of computers still aren't patched after there was an announcement and then that's when all of the Vultures come in, like you're saying, the Bitcoin miner. So, maybe nation states had those zero day vulnerabilities or some kind of criminal organization but then all of a sudden it gets announced and now the Vultures has come in and start scanning ports and looking for places to go for people who don't have a real plan in place. So, I guess I'll stop there.

Jamie Thomas: Well, there you did have about five questions there so I'm going to try to remember if I missed one, you can come back to me, Luke. Let's start with the average client, you mentioned Colonial Pipeline. Because not every organization's like IBM has thousands of developers who can go be an open source contributors and everything. They're just consumers of software that the rest of us produce. What those organizations I think really need to think about is making sure that their software is up to date. And this is a very hard task because I can tell you I see a lot of evidence of organizations that are running on end- of- support software. End- of- support software is going to have underneath it, end- of- support open source. It's going to have vulnerabilities that are not patched because those vulnerabilities are being patched in the latest software. That has been a very large misunderstood topic. And so, I think unfortunately these bad events are starting to educate a lot of organizations that this end- of- support software is not a good thing. You're going to have to be keep your software up to date, you're going to have to move to later releases. And we, as software providers, have got to understand that we got to help organizations to be able to do that, right? They've got to be able to do that in a way that allows their business to run. In other words, if we are shipping software that is on premise, obviously, you're making a big change every week, then it's not going to be easily consumed. And so, we've all got to make it easier for the clients to adopt the latest software patches because that's imperative. It's imperative that they do that. When something happens, a lot of the organizations, they don't easily have the ability to understand what's affected and what's affected even in their infrastructure. So, that's another skill that has to be built. If you're managing a big set of infrastructure, you need to be able to assess where Log4j is and where it's patched. I mean we in IBM can do that. We have huge amount of infrastructure. It's not just infrastructure for a CIO, it's infrastructure for all the development laboratories and everything else we're doing that have to be patched. My belief is that even with patching, the best practice for an organization having to patch a lot of servers in that case is a centralized patching team. Because I've never believed that assuming people are going to do this at midnight and their spare time is a good strategy. And in fact I've actually never seen it work too well. And I've seen every time you centralize things in a company, people say, " Oh it's too expensive. Get rid of the... Decentralize it. Tell people to go do it in their spare time." Patching and spare time and assuming people are going to get all that right, I think, can be a very dangerous strategy. So, my take is you get a small team that's effective, you get that in order. That's what a lot of these organizations need as well. Now, for those of us that have the benefit of having thousands of really skilled developers, I think it is our obligation to be participating in the right way. And that's why organizations like Apache and OpenSSF and the Linux Foundation are so important. That's why leveraging what we could do through Red Hat is so important. It's our obligation to make sure we're taking those actions to help the other organizations who are not development organizations, they are consumers of software. Now, some of those organizations are so large in the case of some of the, if you look at financial services, they do have thousands of programmers. So, they are able to also bring to bear contributors. There are companies that are of that size that can bring to bear those best practices and participate in these organizations as well. But if you look at a lot of the organizations that get taken for ransomware, hospitals, educational concerns, city governments, they're not out there able to pay for hundreds of thousands of dollars for cyber experts. So, we have to help them with some of these basic best practices, which is run up to date software that doesn't have vulnerabilities, have some investment in that, have some basic cyber protection, make sure you have a way to get your servers patched in a meaningful time. Those are the things we have to help them with along with the things we're doing to make the software inherently more secure, if that makes sense.

Luke: It does make sense. And when you're saying that, it reminds me, I was recently listening to a podcast about a school district hack and the situation was they had automated some sort of disaster recovery backup system but they hadn't checked if it wasn't working and they were really stuck then. And the other thing is, and this is to comment on your mentioning, maybe it's not a good idea to have public software bill of materials. Something that they pointed out is the reason that a lot of these hospitals and school districts and governments and stuff are getting hacked is they publicly have to publish their finances. So, these hacking groups know exactly what they're, they know what they have and say, " Hey, we know you can pay this because you've published it." And I'm imagining a scenario like that too where if you're too public about what your stock is and what's in your thing, you're also giving a roadmap if there are those organizations that have zero days and things that are not known, they know where to get you now.

Jamie Thomas: Well, I think that is really an important question. So, deciding what you're going to publish and where you publish it is important. Whether it's software bill of materials, I certainly wouldn't go out and publish the bill of materials for the quantum computer. It Would be quite an interesting thing for a lot of people. Publishing, you have to be astute about the information that you publish. But what I would say about a lot of those organizations you hit on a key point is the data protection is not where it needs to be. So, aside from all these other things, your most critical asset is likely your data. And if you lose access to your data through ransomware, obviously you cannot do anything. And from my experience, most ransomware attacks take at least seven days to recover from. So, I personally went out and got gasoline in my car when I saw the Colonial Pipeline attack. I think I was out that day because I actually almost ran out of gas the last time the pipeline went down to due to a hurricane here in Raleigh. So, I was like, " I'm not going to go and be at a gas line with a hundred other people again." I'm going to have my gasoline because my best guess is it'll take over seven days for this thing to come back, which is, I think it took about seven days. What organizations need to do is what you said they don't do, which is they need to make sure they have a copy of their data, they need to make sure it's up to date, and they need to make sure that it can actually come back to them in their lifetime. The amount of data that gets stored is so vast that it can often take days for the data to come back. So, do you have a way that you can get the data back more quickly? And that's why an IBM, we have invested in our storage products capability, it's called safeguarded copy, which takes snapshots of this data, which is immutable, it's tamper proof, and they can get it back quickly. Now, you can then store a lifetime of data. The petabyte is out on wherever you want to store data storage, tape, et cetera. Because a lot of organizations have to keep it around for a long time. But you have to realize the amount of time it can take for huge amounts of data to come back, that's your only option is often days. And that's not well understood by a lot of the teams, right? So, you got to protect yourself from ransomware, but you also have to have that data recovery strategy. And the two of them not being done is often the case, right? Protection's not there. The basic cyber protection's not there. And then the data protection strategy is not there. And I'll, just another tip, do not put the copy of your data on one device. So, don't have your production data on the same storage device as your backup of your data. That's a common thing that happens too. And so, if that one device goes down, guess what? It's all on the one device. So, you got to think disaster recovery in these scenarios and think about those types of things. And I find that the smaller organizations are struggling to think about those kinds of things. Particularly in this job market, hiring the right kind of skills is very difficult. We, as vendors and we, as experts have got to provide the techniques, the best advice, the products that allow people to have more flexibility in an affordable manner to do some of these things. But I've been, for a long time now, talking about up to date software. I've been talking about supply chain disruption. I've been concerned about supply chain, the reality of our supply chains. All of this came to reality here in the last few years with COVID- 19. We saw an increase in ransomware and hacking. We've seen an extraordinary disruption of our supply chains. That's another case in point of disaster recovery. Do you know where things are coming from? Do you know what the cause and effect is? Should one country get shut down or should the shipping lanes get shut down or should air flights be dramatically reduced? These are all kinds of things that many organizations have taken for granted, right?

Joe Sepi: One thing that's come to mind, I think you've touched on some aspects of it, but I know with the Log4j's situation, we were able to, at IBM, figure out where we're using it and get patches out for our own stuff. Do you have any more suggestions? I think you've touched on some of them, but in terms of companies dealing with something like that. But we talked about a central organization for patching, making sure you're up to date, are there any other things that you think people would benefit from our experience?

Jamie Thomas: Well, obviously we had to do a number of things. We had to track the efficiency and the completion of all the product updates. We had to do that. We had a central team that has tracked the implementation of the servers, the remediation of all the survey infrastructure, which quite vast across company. One of the decisions we made early on is if we didn't think it was mission critical and we couldn't patch it, we actually shut them down. Now, we had that ability to do that in some of the development lab capacity, R& D capacity. So, it happened over the Christmas holidays if you'll recall. So, we didn't make some calls that, listen, everybody's out for the holidays, we're just shutting down a bunch of this stuff. We also shut down some cloud operations of different software packages that we saw vulnerabilities released or we were aware. So, we might have shut that down for a day until we could get it patched. So it was both, it was kind of a triage operation, but you have to have the backbone of systematic execution. So, do you have a team that's going to monitor and make sure everything is patched? Do you have a team that's going to make sure you get all the patches from the vendors? And once you get the patches, are you getting those patches deployed? Because I know that a lot of the patches we produced are not deployed across all the clients yet. I mean, there's some of them I'm tracking very closely, and I can see that they're not yet deployed. So, that's another challenge in this whole thing. It's that when something's that prevalent, it's going to take you a while to get those patches rolled out and you have to have someone that's on point to make sure that happens. And it has to happen in a time that is effective for your risk management assessment. So, parts of your infrastructure may be deemed more risky than others. And so, that's part of your judgment in terms of how you roll out that patching. I did have someone tell me not long ago that they had a vendor that was providing the patch in the fourth quarter of this year. Now, that is a little bit late for the patch. So, I told them that I think they need to go back to that vendor and have a different conversation. Realize though that this was very complex for a lot of organizations to manage and I think we've all learned from how would we do, what would you do different from a communications perspective? What would you fine tune? How do you increase the velocity? All of those kind of things.

Joe Sepi: Yeah, and I think with a lot of these things, people tend to be reactive with Log4j and these other big ones happening recently. The shift to being proactive is really key, and I think that's underpins a lot of what you're saying. I wonder, and this is maybe getting back to one of Luke's earlier comments, do you see, in terms of companies that are able to invest in open source, do you see that happening more? And also, I mean that in terms of money as well as resources and involvement at OpenSSF and such. And I wonder not just companies, maybe you could talk a little bit about the White House meetings and was it a two- way conversation? Should we expect more from the government? What's going on there?

Jamie Thomas: I think that I see more large organizations thinking intently about their open source strategy, about participating in these kind of forums if they can, about contributing. In the case of OpenSSF, the core members have contributed some investment there for the ongoing effort. I see organizations much more thoroughly using scanning tools to scan their code for vulnerabilities. So, I think we're going to see a lot of the regulated industry be a lot more assertive about what they expect from their providers. We're seeing that because the awareness has been dramatically increased. And part of that is due of course to all the executive orders that have come out from the White House around what organizations should be doing and what they are expected to do by the government, which has been changing over the course of the last year. I've found that this particular meeting we had was a very collaborative meeting, very good job by Anne Neuberger, the associate assistant director of National Security who you may have read about in the press recently, also helping the Ukraine according to the press. It was a great forum for us to be able to share across the larger organizations about certain topics and share what could work and what we thought we think could be most beneficial. And I think that kind of collaboration across the major players will be important going forward. And so, we at IBM certainly look forward to working in that vein in a collaborative way to understand what will make the biggest difference. Shortly after that meeting, unfortunately, in the last few weeks, these whole events did play out of the invasion of Ukraine. And you can see there where cyber attacks have been a part of the warfare, right? It kind of going both ways between the two countries, but certainly there was a big attack on the infrastructure of Ukraine from what we can read about in the press. So, it is just an indication of what we all suspected that a battle doesn't take place just with the traditional warfare, right? It can be exacted in this vein, and that's why it's so important for us all to be diligent about it. And that was obviously part of the messages that have been coming out from the administration and the White House over the last year. The heightened awareness started with SolarWinds, if you all recall. And that happened at the end of 2020. It happened at almost the same time in December as Log4j happened oddly enough in 2021. So, we had about a year separation between SolarWinds, which was a software supply chain attack into the SolarWinds asset. And then you had the vulnerability exposure in Log4j, which was a different kind of a situation, right? And so those two big events really were a catalyst to why governments have gotten a lot more involved in this topic. I would say SolarWinds was disruptive, but not in my mind, as disruptive as Log4j was given the prevalence of Log4j everywhere. Now, that's just my perspective because they're very different attack vectors in terms of how they evolve, but they were both in involving software.

Luke: I have so many things and I realize we're running out of time, so I'm sort of triaging in my mind what I want to ask. So, I guess the first thing I wanted to say is it can seem really daunting. I imagining if you're a company, especially with the challenges from COVID and supply chain and now the disruptions that we're seeing from the invasion of Ukraine, which it really goes to show how connected the world is too. Because I'm surprised like just yesterday, I was talking with a developer who I had met at some of our IBM meetups years ago, and his startup, his lead developer was in Belarus and his team was in Ukraine and the chaos that they survived all these supply chain issues and COVID and figured out, and then now to be faced with this, and it really goes to show you how connected the world is. But what I wanted to parlay this to is that it seems daunting, it's very complex, so we mentioned all these attack vectors and across the hybrid landscape, but something I want to bring it back to is we do I think great cybersecurity training for all of our employees at IBM. And I'm imagining a lot of our listeners do it at their company, but I feel like it seems like there's all this advanced stuff, but really if you bring it back down to the basics, solidify yourself on the basics, right? Because so many attacks come from clicking that wrong link in the email or that becomes the chink in the armor that brings them in. And the other thing I want to mention, and I guess my question would be, is this a good thing to say or am I right here about it's also really careful what you post. People post little details about, " Oh, it's so annoying, we're getting an audit again like this." or" Oh, I don't like this thing." And you say it on Stack Overflow and I was listening to my favorite cybersecurity podcast, Darknet Diaries, and that's where penetration testers go, and that's where these nation inaudible and organized crime. They're reading your Twitter, they're reading Stack Overflow, they're looking for those little pieces of information that give them the insight to start their social hack or to know what tool set to use to attack you. So, I guess my question is like, " Hey, basics." Right?

Jamie Thomas: Yeah, I think so. And as these kind of social tools have become more prevalent, even inside organizations where we're all using things like Slack and GitHub and everything. I think you need to still discreetly define who your universe is. You don't need to share everything with everybody. Particularly if you're in a large organization, you certainly don't need to share all of this information outside, right? It's not appropriate and it's going to create issues. And so, basic education around cyber techniques, development techniques and everything need to be a critical part of everybody's journey when we were talking about that for organizations. Some of these things are very simple. I am a huge fan of LinkedIn, but I get a lot of LinkedIn messages coming to me and I don't click on the links and I have to write people back. And I'll say, if I'm interested, " I'm sorry, but you're going to have to get to me otherwise I'm not clicking on these links." You know what I mean? So, we really do have to continue to educate everyone because it is very complex, but there's some basics. You can overshare, you can create too big of a team that you're working with. You don't need to share everything with everybody. I've talked about SBOMs, why should we share some of this stuff with everybody? But it does go back to education and skills. So, I think that's something we can all do for those of us who have very skilled teams. How can we help education? How can we create the next generation of cyber experts for organizations at large? It's actually exciting because there's a lot of opportunity for people here as well.

Joe Sepi: Yeah, for sure. This is an amazing place to grow your skills if you're working in the tech space. I want to make sure that we didn't miss anything. Jamie, is there anything that comes to mind that we should have touched on before we kind of have to wrap up here?

Jamie Thomas: Well, I'm going to say one last thing because International Women's Day was earlier this week. And so, in that vein, I'd just like to say that a lot of the topics we've been talking about are very interesting, whether it's software delivery, software development, cybersecurity, security for organizations. So, just put a plug out there for all the ladies that might be listening or people that have ladies in their family or friends that this is a great opportunity for people from a career perspective. And we still don't see enough women in technology and in some of these fields. So, getting back to our point there about skills and everything, I'm just a real passionate proponent of getting more women into these organizations to increase the folks that can help us, right? And to relay to everybody, it's pretty exciting stuff. It can be fun and exciting and you get to see a lot of interesting things that are occurring in the world around you and have a big impact on that as well.

Joe Sepi: And I guess I'll say on that, first of all, my team is hiring, my Twitter DMs are open. If anybody has more questions or wants to talk about any of these things or wants to get into the space, feel free to reach out. I'm happy to help and point people in the right direction. And I don't want to make you blush or anything, Jamie, but I'm so happy to have you here. When I would talk to people in the community after the White House meetings, they were effusive about how happy they were to have you in those meetings. Your experience and intelligence is on display here. So, I really appreciate you coming and talking with us.

Jamie Thomas: Thank you. I really enjoyed it. You're a great host and it's a very important topic, something I'm very passionate about. So, it was great to be here today and let's hope for warmer weather so we can get back to golf and things like that.

Joe Sepi: Yeah, sounds good. And if you ever want to come back on, something's happening you want to talk about, we'd love to have you back. inaudible-

Jamie Thomas: I'd love that something's always happening. So, it's like a matter of when you guys want to have me back, but I appreciate it. Thanks Luke, and thanks Joe.

Joe Sepi: Great. Thank you.

Luke: Thank you, Jamie.

Joe Sepi: Cheers.

Jamie Thomas: Cheers.