Automation and Transformation in IT Infrastructure with Jordan Lowe

Avi Freedman: Hi, and welcome everybody this week on Network AF. I'm talking to my friend, former collaborator and former boss, Jordan Lowe. Jordan runs a company called Deft and we're talking about what's fun in and frustrating in networking nowadays. Jordan, could you give a little bit of a brief intro?

Jordan Lowe: Yeah. Thank you, Avi, for having me. Excited to join you on here. So, as you mentioned, I'm the founder and CEO of Deft, which was previously known as ServerCentral. Before that, I was working with a company called nLayer, which got acquired by what is now GTT, which is a lot of the networking stuff we'll be talking about today, I'm sure. But that's it. I was with ServerCentral. As we rebranded to Deft now, it's over 22 years now. So, I've been mostly doing the same thing for quite a while. It's been a lot of fun.

Avi Freedman: Well, it's a little different every year over a year, both on the tech and business side. Where did ServerCentral start?

Jordan Lowe: That's a great question. It started in the dorm room. Truthfully, it's one of those stories that somehow ended in success. Yeah, we started with an old courier modem at my college apartment. I still remember updating the firmware on that thing, totally. V dot x dot fast, or whatever the latest thing is to get that extra 2 or 3K off the dial up speed, but 22 laters, here we are.

Avi Freedman: And you were also doing DNS and things like that, overlays?

Jordan Lowe: Yeah. When the company started, we were doing hosting. We were one of the first people to combine domain names and web hosting into one plan. So, we've basically given away domain names, which was unheard of at that time. This is back when Network Solutions were charging$ 35 a year for a domain name. And we started bundling it in with a$ 5 or$ 10 a month hosting account. People didn't believe it was true, but that that ended pretty well. A lot of people got thousands and thousands of customers there. And that's what kicked us off into what we are today. We wrapped that business up and sold it off in, I think, 2008 or 2009, but that's basic customers and that experience is what got us where we are today.

Avi Freedman: Cool. So, you've benefited from another of my guests and friends, Elliot Noss and opened SRS and Tucows and what they were up to.

Jordan Lowe: Absolutely.

Avi Freedman: Yeah, no, that's cool. So, you've been running networking for quite some time. I want to ask you, what year you first got enabled? I think, yeah, for me, it was way too early. It was actually in college, late'80s and I tried not to do too much damage. But things are different, but things are the same. What's exciting about running a network? What's cool about running a network nowadays?

Jordan Lowe: Awesome. I was just thinking back to when I first started playing, actually.

Avi Freedman: When was that?

Jordan Lowe: It was a little bit earlier. I think my first computer was a LISA, which inaudible.

Avi Freedman: Oh, wow. Running Mac OS or running LISA?

Jordan Lowe: I think it was probably LISA 2, because I'm a little younger than you. It was probably'88,'89, so it would have been the end of the LISA world.

Avi Freedman: The first time I used a LISA, I demoed a LISA using LISA. The first time I used a LISA was when they made the Mac clones that could run like Mac OS.

Jordan Lowe: Yeah, I still remember that. There's like a phone cord connecting the mouse. It was pretty classic. Yeah, then I got a Mac LC, I think, in the early'90s, my first real computer. We all remember the first bone on the first. It wasn't really my first business at ServerCentral when we started that. Actually, I ran a BBS for a little bit way back in the day. I remember having a couple phone lines in my parents house. That was some classic stuff there.

Avi Freedman: That was the one we got at home. That was how I got into the ISP side because I had pent- up frustration from not running a multiline BBS, but we weren't into it. My parents weren't into it when I was growing up.

Jordan Lowe: That's funny. Sorry, I apologize. You're asking about what challenges networking has today or what it looks like?

Avi Freedman: Or let's look at the positive side. For me, it's like I plugged in 100 gig connections. And I was expecting that the heavens to part and/ or there to be tons of problems and instead it was just like, "Chup, chup. Link up." It was like, " Wow, that was pretty anticlimactic." In some ways, it's pretty cool what was going on with networking today, so I thought inaudible.

Jordan Lowe: It's awful. Yeah. I'll open up with us being old. When we started things, we're a lot different. If you get a T1 or a DS3, it was a huge deal. It was very complicated. It wasn't that fast, even back then. But yeah, having inaudible yourself just work today, 400 gig is there's plenty of carriers out there, you can get 400 gig ports today, which is crazy. And yeah, it does just work balance a lot stuff. What's lasting over time is the business is complicated. It does generally just work, which has taken a lot of pain and a lot of, I'm sure, bugs to multiple people, including us, who've found over the years to keep this stuff working all the better. But it is pretty nice. I will say things are much easier when it comes to adding capacity and just general things at the moment. It makes a lot of things DDoS, less painful. You don't have to worry as much about the stuff that you did back then when anybody could take you out. Now, it takes a little bit more energy than it did before, slightly. Slightly a little bit more energy to throw more than a couple of 100 gigs around.

Avi Freedman: Who was I talking with last week? We were talking about the trick that people used to do, who had IRC servers of buying a connection for someone you hated and single homing to IP space from them, your IRC server, so that they will get the attacks instead of you. It was like early DDoS defense.

Jordan Lowe: Yeah, that's actually one of the things that trained us and keeps us on our toes. We've had an IRC server the entire time, basically.

Avi Freedman: Oh, wow.

Jordan Lowe: So, we still run it. It is still up and obviously, these days, it doesn't get too many attacks anymore, as IRC isn't too exciting after things like Slack copied and they get along well.

Avi Freedman: Maybe we should show IRC to Elon Musk. Maybe he wants to bring back IRC and use that for decentralized future glories.

Jordan Lowe: It would work. It's still out there.

Avi Freedman: It is and still decentralized. So, what about automation? There's a lot of hype about it, but I was talking to some folks that are customers of yours and they said that it's a big part of doing the design and getting automation and having stuff up. How does that differ from 20 years ago when you turned someone up with a inaudible that has multiple switches and routers and servers and stuff.

Jordan Lowe: Oh, yeah, man, it's changed pretty wildly. Back in the day, the best you could do was some scripts. But then a lot of the router vendors, we've always been preferential with Juniper. Juniper allowed us to have a little more flexibility in the early days and load more of the customer side config'd into the router as a template. So, we were able to make it a lot easier to turn the customers up, not having to repeat 17 commands every time you're adding a new BGP neighbor. But these days, it's a completely different world. It's all and what would be like a Ansible setup or these Python scripts that really allow us to really do a much more modern job. And really not have to log on to switches at all. Not saying it doesn't have its own drawbacks and then problems. The generation of people that are used to being on those routers and checking things, and the process they do is a little bit different than what it is today, so it is easy to miss a couple of things. And we've definitely had our own challenges as well. With the automated side, you might not be monitoring the same type of stuff that were before. So, the same thing is we'll kick off CLI or you can write that. A CLI will happen or it might not actually work and then you don't know about it. So, obviously, every time we make a change like that and we learn and script it out, script it and check the imaginary condition, it's programming.

Avi Freedman: One of the big challenges is I remember when I was trying to use Brocade for the first time and I was cutting and pasting because it was so close to Cisco. And they had a thing where every time you made a change, it would say like, " Yes, I made a change." So, I didn't notice that it wasn't actually taking the route maps and then I turned on BGP with no route maps and it became made clueless because. So, whether it's CLI or when you try to automate and do things too fast and then, hopefully the vendors don't change, API response codes. And as SaaS company, we have the same challenge, which is people build to your APIs, that becomes a promise. In the router world, CLI is a promise, API is a promise. We could move to API if people would just make everything only available or available via API also and some vendors do better at that than others.

Jordan Lowe: Yeah, but realistically, all of our new deployments, they've all been completely automated and it's a very different way to work and it's-

Avi Freedman: Ah-

Jordan Lowe: Go ahead.

Avi Freedman: So, are you at the point where for, I know this one deployment it wasn't cookie cutter, but for a reasonable deployment that you're really just getting in via CLI either to bootstrap IP address or to just check that it feels right at the end?

Jordan Lowe: Yep, yeah, yeah, generally. And if even that, so again, that's only the newest stuff. A lot of our, I'll say, Legacy equipment, but it's still a very, a lot of our older stuff is not set up for that spec yet. Our team is working really hard to get there and they're doing an awesome job, but being around for 20 years, gives us a lot of, what we'd call, a cruft. We have lots of devices, lots of things that do not perfectly align with everything else, so you can't really automate everything yet. But we're working hard as far as we can to make it all the standard stuff moved off. And the database isn't where we can really have it all automated. It's great. That idea for us, obviously, allowed customers to do a lot more. We've always had a couple of fun things customers could do that were groundbreaking, we did it. Letting customers at their own like no routes or add their own firewall rules. That was cool 10 years ago, but now, it's just expected to do stuff like that. But realistically, a lot of people have not been exposed to this stuff. Even our customers are saying, you're probably going to inaudible, for example. It's life changing. We have customers every time we show in our portal demos. It's not something a lot of people offer. It's super appreciated. It gives them insights to something that they never think about. And that's actually, it's a huge win for something that we've obviously quoted that when we started the company years ago. People love it. And a little something probably makes a difference.

Avi Freedman: Well, it's probably because we have to talk through it with some companies, because they're like, " Wait, but if the customer can see why their links are full, then maybe they won't buy as much." It's like, " Well, but maybe they'll just be happier and buy more from you. So, it's like the old QS discussions, if we rate limit, sell them as much as we can and hopefully, they use none of it. And that's how we're going to make money. It's just very different from the modern business philosophy of, " Let me make this customer super enthused, and they will go somewhere else and bring us in. And they will tell their friends and they will adopt it widely." It's doing well by doing good, but brought to the business side, which is pretty cool to see.

Jordan Lowe: Yeah, definitely. Yeah, it's been fun, man. Networks Office is definitely close to my heart and I love playing them. I love getting the latest stuff out there. And it's a very gentle prodding we have to do to move stuff along because most of our customers don't want to take an outage to shift platforms or shift deployments, because it doesn't change anything for them then. They're not really getting a big win, we're getting a win, obviously. But we have to say, "Hey, this is long term. This is for stability. This is to get you on the latest everything."

Avi Freedman: Well, you do make sure the Greenfield stuff, the new deployments are run that way. It is-

Jordan Lowe: Yeah.

Avi Freedman: When I was learning TCP/ IP, there was still a question in late'80s about the ISO protocols. I don't know if there's anything left, but is this from ISO land in modern networking. But I really, in the late'80s, again, early'90s would have thought that by now, the way the internet would be built would be different. Instead, we have IPv6, but we also have IPv4. We have BGP version 4. We still have inaudible. And so, that's a testament. That's really awesome, especially as someone that doesn't enjoy doing the endless protocol meetings and RFC type stuff. I'm thrilled that there's people that love IETF. It's just not for me with my ADD. But, no, that's pretty cool when I look and see, well, we're communicating over Zoom and the internet is mostly working. And so, it's a mystery to me that it hasn't moved, but also, pretty cool that it's been able to work just going from a megabit to a gigabit to pretty simple half terabit interfaces.

Jordan Lowe: It was pretty amazing when COVID started and everyone started doing Zoom and all the different meeting platforms and it generally worked. Obviously, a lot of people are running around crazy upgrading software on the back of all the consumer networks. And I'm sure everyone at Comcast will tell you stories of all the networks. " Oh, we had to do all this stuff and all this blitz and all this stuff." But, in the end, they got done, it worked. Zoom was able to spin up a ton of servers. They did do some things to limit bandwidth, obviously, initially because they've been toughing out. That was a huge increase. And they want to limit bandwidth for both their equipment, Amazon's equipment. I'm not going to say not their bills as well, but they can kick some defaults down to 4ADP or whatever the lower bandwidth rate is. But they already had a pretty good platform. They already did a good job of making all of the speakers video spin up to full speed. When they were speaking, everyone else slowed down to a lower bit rate, which was already existing, which is hard.

Avi Freedman: It's also a customer base, just people had a bunch of overprovisioning in place, so that was really good. So, sounds like lifecycle automation is something that you guys have, that is really where we see most of our customers working towards. And of course, if you can't see it then that's not really helpful. We don't really have any customers doing the magic streaming telemetry, stream 400 gigabits of state from the thing and have magic ML show you everything that's wrong. But there's a lot that you can do just with device metrics. And of course, traffic and performance and all that. So, what are the challenges of running a network? It's much more efficient to run network than it used to be, certainly per bit. But what are the challenges?

Jordan Lowe: Yeah, we're just off the bat. Lately, it's been hardware availability. That's the big thing, both on the DWDM and optical side, and top rack switches have been a bit of a challenge. We were lucky enough, our team did suggest we place a lot of large orders for both network equipment and compute and storage, right when this all happened. So, that's all been flowing in over the last year or two, which has been working pretty well. So, we haven't had any major issues, but we've definitely delayed a couple of upgrades here and there. We definitely have a couple of locations in our network that are a little behind. We have some sites running all the routers, and then we're planned. Nothing catastrophic, but hardware wise, that's been the biggest challenge, really. People, it's been a challenge. I'm lucky to have an awesome team I'm working with. We have a bunch of really great guys, between operations, automation, platform teams, that all, network side. It's really great. So, we've been lucky we've got a great team. What else? The stability is obviously always a big challenge today. We've been using all these newer protocols we've talked about. The newer deployments are all EVPN, and VX LAN, which are while they're not that new, they're still new versus things that are 20 years old, so they're not as stable as the old stuff. Now, that's a pretty small percentage of issues, but there's still more issues and bugs on the new software than there ever was before. So, we've got to be very careful about testing, very careful about writing the right versions of code, all the right stuff, because there are issues. There's still some interoperability issues. There's still just a host of things that we've got to watch out for all the time. So, it takes a lot documentation on review, a lot of testing, especially on the stacks that are on the latest and greatest stuff.

Avi Freedman: Which ties back to the people, right?

Jordan Lowe: Mm- hmm.

Avi Freedman: Which is, I hate it and I love it that networking is like physics, where you have to know the first principles and be able to build in your head a model of what the thing is doing in an Einsteinian thought experiment to say, " Well, this is what it should be doing or is this a bug?" Sometimes, just things break and sessions die but sometimes routes just don't get advertising, like maybe I used it wrong or maybe my scripts to automate it is wrong. It's not as bad as the'90s is it when you'd stick the fitty card in and sparks would spit out the hissy part and you're like, " How could they make the software do that?" And it was all like a one- operating system program loader in the old big battled Cisco days.

Jordan Lowe: I was actually taken back to my dorm room. Remember, we had a local talk inaudible. So, I don't want to get caught up there, but yeah, we had those local talk things and inaudible.

Avi Freedman: XNS, SNA, local talk, yes.

Jordan Lowe: Yep. Yeah, that was before then I went to... before, I just remember doing all this stuff over... what was the original coax? 10BASE2 or something or what was the original coax networks?

Avi Freedman: Oh, after thick, so you weren't doing vampire taps and all that. It was just the net. It was just Ethernet, the net stuff.

Jordan Lowe: I was inaudible.

Avi Freedman: Turning it into a Zoom, yeah.

Jordan Lowe: But back to the networking challenges, because you're saying that we've had a different experience than people coming up today. The kids as we'd call them today coming up with the networking, they never got the chance to do any of that stuff. A lot of these guys have they've grown up writing networker's code. They've grown up starting out playing with Amazon's, their original platform. They didn't build the server. They didn't build the switch. They didn't program it, set the IPs. And get the serial console working with, back in the day, trying 14 different USB adapters like we've had to do over the last 10 years. That's been a little bit of a challenge. Just people learning the basics that when problems do happen, you need to think back to how it does work, so that can always be fun. Obviously, as I said, most of our guys are awesome and then have some of that experience. But for the new guys, it's always fun to get them in the lab and break something and they learn how to fix it the first time. When it went, it's not something that's running. A lot of people come into a running network.

Avi Freedman: So, do you have a lab both for training and for testing new configs and software and all that? And is it virtual, physical, both?

Jordan Lowe: Both. So, we have both the physical lab for even from the DWM side. We have DWM labs. It's for testing and training for both our datacenter operations guys. The guys that are going to patch it off, so they can learn how to do loops and light level testing and cleaning fibers, things like that. And then that's going through the network labs where we have basically one of each type of router and switch that we deploy, if not multiple. Then we can go ahead and stage code on there, run upgrades and get into the network, do traffic tests, stuff like that to make sure everything works properly. And that's stuff is, as I mentioned, is more and more important because before you can get away with this, doing a shotgun upgrade. These days, you got to be very sure everything is working correctly and test everything very carefully. It's just not worth-

Avi Freedman: Hopefully, there's some gals among the guys at Deft on this level. But yeah, in the early'90s, I think I still have some people that work for me who have trauma. We used to have a bat light in the office that would go off when I enabled. And when I was coming to a POP, I would just pull the UPS and make sure that it worked because if it was going to die, it would be faster if I was there because we were very lean and we had a very small org at the time. And so yeah, things have definitely changed hugely in terms of how we think about that and yeah, customer is always on. So, networking doesn't work. And even the Cloud outages, most storage outages in cloud are really network code or configs or everything is layered down. It's not like it isn't the same technology. It's just weird and different names.

Jordan Lowe: Yeah, scaling issues, deployment issues, or code upgrade issues that break, 17 dependencies to make it all work. People learned hard lessons. Yeah.

Avi Freedman: Yeah. Which do you see is more challenging, taking someone that is Cloud native in their thinking, so they're using peering and tunnels and things like that, but maybe not thinking about them as that? And then getting into networking and primitives, and how to live in that hybrid world, or the reverse and taking people that have come up on networking and computers and storage and know that stuff. But don't know the names and the bugs and the APIs and don't think program first or get ops first. What's your approach, and which is more challenging?

Jordan Lowe: I would say, most of our customers are not super technical. We've got a lot of awesome tech for companies with great staffs. But a lot of our, I will say, more enterprise customers or just normal customers are not, don't get into the weeds. They just expect it to work. And the downside of that is when it works perfectly for years and there is an issue, they get upset, because they got away with not having redundancy and disaster recovery, things like that. Where if you have a little blip or a little problem, the world descending, so luckily, that doesn't happen very often. But versus the technical customers, I think, understand if there's issues or problems a lot more. And we really enjoy talking to them. One of the best things we can do is when our staff enjoys talking to our customers. When our guys can talk about and geek about stuff all the time and get into and talk about challenges and issues and build new cool stuff. And that's really fun for the staff, too. So, truthfully, a good mix of that stuff is the best. We have a bunch of people that are just happy being stable and running. And then we have the fun customers, we get to talk tech with and build all kinds of new solutions for.

Avi Freedman: And so, do you have different teams for Cloud networking migration versus network ops versus some of the automation and tools side or...

Jordan Lowe: We do.

Avi Freedman: ...where can you converge and not?

Jordan Lowe: So, there's a couple of different groups. We always separate it out, I would say, maybe a year or two. It was kind of the projects and tooling team from the Operations and Support guys. So, that way, before, it will be the same guy, so we try to do everything. Take the support calls, work around the automation, test an X upgrade. The problem is customer stuff always comes first and they'd get backlogged. And we wouldn't get to all the fun projects, we wouldn't get everything tested and upgraded in time. We separated all that out, which was a good strategy. Now, we have three different groups that handle all those different items. And that way, the automation guys, the tech guys are focused on customer turn- ups, automation, going back and fixing stuff. Making sure, for example, some of the automation requires all report descriptions to be the same or exact. Stuff like that, going through devices that might have been sitting there untouched for a long time, other than software upgrades to making sure everything's perfect. Making sure it's all going to work. Think about little stuff and then tuning all monitoring. We've got all kinds of different groups for the stuff now, which helps us get through everything a lot better versus just hoping, pushing someone to make time for it. Now, it's just their job is to do that part.

Avi Freedman: Yeah, just to come back to it about automation, that does seem to be the real core that everyone needs to have, which is the lifecycle when stuff is turned up, turned down. When we onboard, it's almost like we have customers that ask us if we can do PS and we're looking for good partners there. When someone comes on board and they want to automate monitoring to follow applications. And we can do that on the Kubernetes side or on the Cloud side. But when you have a network full of interface descriptions that are wrong, that's definitely a heavy lift to get into, but it isn't always possible to just say, " Well just redo it all, Greenfield." And then figure it out.

Jordan Lowe: There's that. Yeah, man. There's thousands and thousands of, tens of thousands of interfaces, in our case. And we've made mistakes. For example, one of the easiest mistakes to make is one of those typos and that means the customers might not get billed, for example, or the customer might get logged in my network, monitoring might not be great. So, it takes a lot of energy and time to make sure that we put checks in for each of these changes, to make sure that if there's a port up without matching description, it does something like that. So, all those sad cases and corner cases that we catch, it's a lot of work and it's a big job.

Avi Freedman: So, about how many people is Deft right now?

Jordan Lowe: 130, something, I believe.

Avi Freedman: So, you mentioned you used to own nLayer. I was there for that. And so, you sold nLayer, but you still do peering and run a... is it global or is it national your peering and transit network?

Jordan Lowe: Global, yeah.

Avi Freedman: So, why do you do that? Why not just by transit? What's your view on that? Is it automated, it's not a big deal? Is it for cost performance, security, availability?

Jordan Lowe: All of it. First of all, a lot of our customers, so the backbone portion allows us to do a lot of things. Everything from, like on the peering side, obviously peering with our big traffic, it'd be people who's seen a lot of traffic. In the US, for example, like Comcast. So, we have multiple 100 gigs with Comcast in each market. If there is an issue with Comcast and one marketer, they have a router maintenance or some issue, having the backbone allows us to send that traffic to another city, route around the issue. Pulling traffic from Chicago at Ashburn, send it out in Ashburn, so everything stays smooth for our customers. The other big part of this is disaster recovery and DR between the locations. Having a backbone allows us to move IP addresses around the network. So, if we have a customer that's got an active setup, we can actually automatically move IPs between markets. So, as part of world, for example, we normally do quarterly DR tests with our customers. We can flip from active to passive and from Chicago to California, for example. And move the IPS across, which for some legacies people and for some systems is a little bit, I don't want to say it's the most modern way of doing failover and application stuff. But it's a really good way for a lot of applications that are a little bit older to have, just. Everything keeps running on the VM, so you just point, the public IPs change. The public IP stay the same, it just gets moved across the market. So, that's a fun thing. And also, allows us to do MPLS networks, things like that, for customers. We have a lot of customers that we interconnect offices around the world. So, they might have an office in London, an office in Ashburn, an office in Dallas, and we'll be able to link all those back to the same backbone. And us, running our backbone, allows us to fully control that. It's like control your own destiny stuff. Where we see the backbone, we can tell if it's an issue or an outage. We can do the routing ourselves. We can do the quality control, to make sure stuff has scaled properly. Just allows us to see everything and makes any issues a lot faster to troubleshoot versus trying to call into one of the carriers. And wait on hold and hopefully, stuff gets fixed.

Avi Freedman: So, yeah, I used the word control there, to be able to control the packet Destiny. That we see that also is a big thing. Is peering setup mostly automated now in your lifecycle stuff or?

Jordan Lowe: I'm not sure, actually, that's a good question. Now, we don't do a ton of public peering at the moment. The peering world goes through, there's two parts of it, as you surely know, is both the price and the size of the ports and the peering changes. So, our CacheFly brand, the CDN side does peering on all the markets, obviously extensively. And we interconnect with them in some markets as well. But on the Deft side, we do peer in certain markets, but most of our peering is private peering with like the Amazons or people like that. Obviously, we're not supposed to talk about who we interconnect with, but most of our peering at the moment is private peering. We are in some of the exchanges. But it's one of those things where there's a couple of business cases behind it, both the cost of the ports and the cost of transit. Depending on those two things, it makes sense to either do peering or not to peering. And right now, in the cycle of things, we're currently in the it doesn't make a ton of sense to do a huge amount of peering right now. Traditionally, I would be on the other side of that. Traditionally that transit was more expensive and it was faster and better to do all the peering, appearing. Right now, we do peering where we need to, but most of our stuff goes out over we have multiple drivers, the different transit inaudible on each market and it's a little bit easier.

Avi Freedman: Right, or if the CDN was a core in- house product then it'd be different. So, people don't yell at you to stop playing with routers anymore?

Jordan Lowe: Okay. I'm not in the routers too much anymore. It's pretty unusual I'm in there, which everyone will probably tell you is a good thing. But I definitely can get in there and I know enough to be dangerous still. But all the latest MPLS automation stuff is people better than me are handling that now.

Avi Freedman: I've got to go down. So, I'm going to ITW in a few weeks. And then I probably won't get to peering forum after that. But I throw in a couple Aristas and it's time to get the 20- year- old cat to 723 PXL out of there with filtered/ 24s, so I'm not about break anything. That is not our production network. That is a lab network, by the way. It's not they don't let me use the production network, but they're really again, there's no need for me to be on our production network. It's automated push, everything. CLI is only for super debug.

Jordan Lowe: I don't know if you can see this shock in my face for you to even say you still have a 720 running, but that is impressive, sir.

Avi Freedman: About five years. No, it works okay. Then again, the Aristas I'm doing, they have something called selective advertisement. So, basically a route map between the fib and the ribs, so we have a bunch of customers that use that. So, I'm looking forward to playing with some performance based and selective, selective advertisements for people that want to use. Especially given the equipment shortages, older stuff as edge devices. That's a specialty that they've had for some time.

Jordan Lowe: Yeah, man. That's why in my days, I've definitely done that on multiple platforms. Having to see, " Okay, we're running out of fib space, we got to start limiting a few things." Especially these days with, again, we don't have anything in our core there. Obviously, we need that, but there's definitely devices out there that the amount of routes out there now is insane. Everyone's had to find every 24 they can find under every carpet. And the number of routes is just-

Avi Freedman: Well, we'll have a...

Jordan Lowe: Every time I see it, it's unbelievable.

Avi Freedman: ... I don't wantto call it a debate, but maybe a real talk on V4 versus V6, at some point. I'll invite you on and we can talk about that. I have been surprised that the routing table hasn't blown up more. Until there's someone major that says, " I'm only going to be reachable by V6 or eyeballs or other or content," we're just going to continue to live in this dual home world. Which seems like it's going to continue for some time.

Jordan Lowe: I think that would be a great show and the other one inaudible.

Avi Freedman: Yeah, we'll do one on our day show and we'll one on that, yeah.

Jordan Lowe: Yeah. That would be entertaining. Yeah, I actually agree. I don't know anyone that could give anybody a business case to say turn off before, unfortunately. It would obviously be great, but realistically, we're still, as we've all been saying, we're years and years away from.

Avi Freedman: It's just topical, because the dialogue thread about using 240 and reminded me of when I was on the Parent Advisory Council, IPVA, Jim Fleming. Jumping through a wormhole and peering with UNS and it's like I don't know. But yeah, it's problems. So, what is new and emerging and hot and interesting, as you think about the next few years? What are you looking forward to see coming down the road on the networking and the tech side, and the business side, also?

Jordan Lowe: Yeah, networking and tech, there's a lot of cool stuff coming out networking wise, and especially on the optical side, there's a couple of new things, it will make a little bit easier to do what we do on kind of what they've branded DCI, Data Center Interconnect. As you can imagine, we have a lot of circuits between data centers, and between-

Avi Freedman: Is this for robots or Glimmerglass and electronical to electronic tunable prisms, or what is it?

Jordan Lowe: No, just the integration between platforms, I think will be a lot easier. And also the platforms themselves. There's always, I don't want to get too far into the optical side. There's all different systems that should work together, that work together, but don't really work together in between all the major vendors. It'd be nice if that worked together and worked better. And there's a couple of proposals out there and things that seem to be happening that will make that a lot easier, which will be nice and you can just. Right now, most systems, if you want a 10 gig and a 1 gig and a 40 gig and 100 gig, it's not that easy just to plug all those things in the same box. And there's some new stuff coming out that's going to make that a lot easier, hopefully. So, we'll see when that comes out. Again, still a couple years out, but that will be nice if we can. Yeah, that way we can just plug in 10 gig and 100 gig and 40 gig for a customer. And we don't have to have a different type of card, different everything on everything.

Avi Freedman: So, you're using more things that are not just passive prisms boxes?

Jordan Lowe: Correct. Yeah. Yeah, so almost all the stuff we do now is active. Active carrier great optical here. And a lot of that stuff has cost put up 100 gig.

Avi Freedman: I see, so that's more fixed. Because on the routers and switches, they don't really care. And the passive stuff doesn't care that much.

Jordan Lowe: Yeah, there are some changes on the passive side, just the sizes of channels for stuff. The traditional stuff is all the 100 gigahertz band. Obviously, there's people that are much more expert than me at this stuff that would be happy to chat about it. But there's some changes coming there where there'll be a little bit more. I always liked the idea of we call alien wavelengths or buying a spectrum instead of waves. Some of that stuff, I think, we'll be enabling that lot more as long as the carriers go along with it. That way you can have less things to break in between everybody. I think it makes things more reliable. Less cards, especially right now, would be easier. But yeah, on the network side, I'm excited. As time goes on, more people going to get into more of the protocol SDN, more of that becomes reality, which is great when it's working. When it's stable, it's awesome. As most things when it breaks, it can be a little bit harder to troubleshoot, so I'm hoping that gets a little better as well.

Avi Freedman: I don't know, who originally said it, but it was Russ White, who quoted it to me. He said, " Automation does not mean simplicity." It can be simple to consume, but when it breaks, yeah, it's tough. What about hype? I have my list, but what do you see that you just groan and you go, " Architecture?"

Jordan Lowe: I groaned a lot about more, what are they calling SDN, SD-Wan type stuff. I complained about that for a while. But I think it's gotten to the point where it's pretty cool and working, now that the underlying network can support it more. So, I would normally groan about some of the SDN stuff going on there. I'm sorry, not SDN, but SD- Wan. I know it has five different names, depending on who you're talking to about it.

Avi Freedman: SASE now.

Jordan Lowe: Yep, yep. It is useful in the right spot now. I don't know. There's a lot. There's a lot going on inaudible.

Avi Freedman: It's just always tough for me to figure out like what's a box? What's a service? It's all Cloud provision, okay. But who runs a backbone? Who doesn't? What requires? What is basically a mushroom, multi- homing box versus whatever? And so, obviously, we contact. We get to see some or all our customer. Across the customer base, we have some of everything. And we bias towards the things that have great telemetry, especially if they can peek into the performance side, but we'll work with anything that isn't closed. But we still see a lot of confusion on the customer side about what is the actual difference. And you have to go talk to actual people, because the marketing talks about value as one, as it does in an enterprise. It talks about value and problem solved, and not really technology, so it can be hard.

Jordan Lowe: Yeah. What happens, they go out there to these offices that have, let's just say on the SD- Wan side, they got offices, they have gig circuit from AT& T, gig circuit from Verizon, both private MPLS that cost $5, 000 each. They say, " Hey, you get a 1 gig from AT& T and 1 gig internet from both these guys." And my mind, your mind goes firstly, " Okay, how is that going to work? Is M2 going to be too small? How's the performance? How's it going to fill over?

Avi Freedman: Yeah, they needed magic. Cloud and-

Jordan Lowe: And then it just works. And what I didn't think about some of the stuff is now, you can do it now. It's not the way it was, I think with the encoding and the way they can, there's enough CPU now on these boxes to shove it all through and for it to work properly. But there were definitely some limitations when it first came out. We would run into them pretty hard. But these days, it's cool. And I think it's neat, especially, there's been this round of pretty major price drops and the corporate and business internet circuit major players these days. So, you can get internet and a lot of buildings now for not what you can get it for at home, but close. And that makes a lot of sense, especially with all these offices that are mostly empty these days. So, it's a nice cost savings option, and it makes it easier to manage as well. So, it's something I've been excited.

Avi Freedman: And I think it's definitely a practical application of automation, just as number one, SD- Wan thing is probably VMware. So, it's deployed works pretty well, but not trying to do anything super complicated.

Jordan Lowe: Yeah. It took them a lot of tries. It took them a lot of tries to get it working right with VX LAN and all the networking and I'm sure they've had many versions of their SD- Wan on the horizon. But yeah, these days, it does work quite well. We do a fair amount of it as well.

Avi Freedman: Yeah. It's just Google started the idea of program everything and just like the streaming telemetry and just like with automation, it can be easy to read the marketing and things that everybody is fully automated, no network people anywhere. The network itself heals. Everything is self- driving or closed loop or whatever. And in reality, when you're talking about lifecycle, some of the SD-Wan stuff, some of the data center stuff, that's where we see the real automation. And yeah, you still need people who are involved in design and setting it up and designing it with observability in mind. And yeah, life is better, but still work to do. Which, to me, that's exciting.

Jordan Lowe: Yeah. Because yeah, man, if you pop that box in all these places and it shoots back to places like you guys to make sure the performance is there, everybody's happy. There's a lot of cool thing. The other cool thing I've been monitoring is on the hardware side and the server side for us and server deployments. So, we do a fair amount of medium size server deployments for our customers, that liked to build these what we call large private clouds, to either. A lot of our customers, they basically run their studies take load.

Avi Freedman: Thousands to hundreds of machines.

Jordan Lowe: Yeah, yeah. So, they run their steady state load in the data center, they put the burst stuff on the Clouds, which is absolutely great way to do it. The challenge on some of that stuff is always the deployment management, monitoring of all those hardware boxes. There's lots of solutions out there that are good, but there's nothing that's great. And what's happened recently is some of these hardware- based solutions, there's an awesome PCI cards you can pop into those machines that handle a lot of the security and deployment and monitoring and backend stuff. It's the same type of stuff that's in AWS and Azure. The way they do it, they've done obviously, you guys, I'm sure you've seen the presentations on them. But they have the cards, they handle the encryption security storage. It's just a PCI card that's popped in the server. Having that available to the rest of us, I think is going to be pretty cool in the coming years.

Avi Freedman: Does that add to the complexity of the system if there's abstraction from the computer perspective? Does it make it harder to debug or does it make it more efficient or does it actually make it easier to design and operate?

Jordan Lowe: I would hope in the end, it makes it easier. Yeah, right now, it's a new thing, a new layer, a new system and other platform you've got to deal with. But it does replace enough other platforms that I think can make it easier, long term.

Avi Freedman: But I've been following it, too and we have a few customers. Even among our big CDN customers, most of them don't use Smart Next, but there was definitely a lot of chatter about disaggregating, aggregating, hyper converging, hyper converging versus disaggregating as an architecture. And for us, we get pitched and it's like computers are pretty damn fast nowadays. And again, we're a SaaS company. We have big data on the back end, but even that's fine for that. And we try to do the compute near the storage, at least, so it's not on.

Jordan Lowe: You know what? You're making me think about something. So, we've got two customers now doing some pretty innovative GPU stuff, which is always entertaining for us.

Avi Freedman: inaudible phase stuff?

Jordan Lowe: No, just high end, just racks full of GPUs that are power drives in the ridiculous numbers. We got servers doing 8 kW each, which is entertaining.

Avi Freedman: A server doing 8 kWh. So, you're doing immersive cooling? What's your maximum density for those cabinets?

Jordan Lowe: That's a good question. I think we definitely have a couple in the 40s that are still doing air, but we're probably limited, I think, around 40s and we can put the box on the cabinets for the water cooling. It's easiest to handle it that way, just cool the rack as the air comes out. They're loud. I'm not going to lie. There's some turbo fans going on there. Man, these are all mostly available, mostly commercially available servers you can get out there with a bunch of those new 8100, whatever their GPUs are in there. You get these boxes that have I think eight of those. And yeah, they're up there. They're there cranking the power. But what I was getting at though, the cool thing that a lot of people are doing now, we have another, one of those guys is testing something right now that disaggregates all that stuff. So, disaggregating the GPUs from the storage from the RAM even. So, you can basically go and allocate this GPU and this much RAM and this much storage all over some. It's probably InfiniBand or something, some funky network that's super fast. That's actually really interesting to quite a few of our customers, so I'm excited to see that stuff working.

Avi Freedman: Yeah. People been talking about it for a long time, but our peers are more SaaS companies. We don't see them using it, the InfiniBand. There's been this promise of InfiniBand switching, not sand, per se, but oriented at this disaggregation and virtual computers and stuff. And you could put GPUs on it. But one of the issue with GPUs is always, "How do you get enough data in fast enough?"

Jordan Lowe: Yeah, just, yeah, yeah.

Avi Freedman: Well, that's another area I look forward to learning more about. It's cool to see the innovation. What about Edge? Is that hot, hype, or both? inaudible taking advantage?

Jordan Lowe: So, done right in the right place with the right people at the right location, it depends how far to the Edge you're talking about. So, I will tell you that I think there's plenty applications that can take advantage of Edges and Metro areas, similar following a pretty standard CDN deployment. I don't think going past that makes any sense anytime soon. Let's talk about 5G. For a minute, 5G is supposed to take the internet from being at one data center in those, in each city or one data center in another city. That internet Edge moving from that data center to that core, almost out to the tower. I don't know if there's a huge amount of apps and stuff and use for that at this point. This obviously will be another call. And it's one of those things where I'd look back at this video in five years and be, " Man, I was an idiot." But right now, I can't think of a ton of apps that would make a huge difference being computed that would take care of that, that lasts a minute, a second or two?

Avi Freedman: inaudible place, but even there...

Jordan Lowe: Metaverse was one.

Avi Freedman: ...having 10,000 cities and 200 places per city is something that would be hard to coordinate, even for our most sophisticated customers. But yeah, we'll see, we'll see. We'll definitely see where that goes. The good part about that is Cloud native architecture, being able to deploy it, one. It's a coordination and telemetry and management monitoring and state. That's hard.

Jordan Lowe: That's there, yeah.

Avi Freedman: At least, we've got the other stuff, which is running it in a standard way. It's just OpenStack and OVS and all the ways that people were OVAs, OVFs, whatever. Yes, you could package up a VM, but how do you turn it on, turn it off, hook it to stuff. Kubernetes is a massively complex beast, but at least with Kubernetes, whether it's one location or 400 locations in a place, we're starting to have a lingua franca there, which I think will be beneficial. Whether it's just someone running off stuff in their network at the Edge or eventually letting people deploy third party stuff, out near the tower, so that's pretty cool.

Jordan Lowe: Yeah, you can see. The one thing that's been doing better is the Cloud gaming. Cloud gaming, I'm not saying there's a huge amount of adoption, but that's something that's been working better. It be advent of a lot of people in these big metro areas, having fiber at their home, and they are a couple of milliseconds away from these data centers now. That's starting to work. But I don't know if that scale is ever going to make sense to install. Servers compete for that in the middle of inaudible.

Avi Freedman: Well, it's like IPv6, I think. And again, I don't want to imply that I'm down on an Edge and were things are going. Again, it's driving architecture in a great way, but when IPv6 was initially sold, it was, " Oh, well, it gives you security. You don't get it with IPv4. It gives you routing scalability that you could do get with IPv4. It gives you QS." None of those were true. And so, people were like, " Well, what's going on here?" And I think some of the first applications were like, " Oh, well, self- driving cars need it. It's like, " Well, not really." But maybe something does.

Jordan Lowe: Yes, they do.

Avi Freedman: And again, I look at you and it's like-

Jordan Lowe: If they do, I'm not going in it.

Avi Freedman: I was like, " Ready Player 1," or/and now, we've got Meta. If we really wanted to do that, again, hard computer science problems to be solved, but we will see.

Jordan Lowe: Yeah. Backing up into what we'll take advantage of and I think some people right now are deploying code. Right now, what percent of sites are really optimized for Edge at all right now. Most sites, there's still some database somewhere in some city that you are waiting for that database to confirm something before it comes back to you. No matter what it is. And so, a lot of the cool company-

Avi Freedman: And Edge ML is a case because people are trying to build very lightweight composable meaning that you can do the work distributed and add it up. And then, you got some stuff where power is cheap at the Edge and we're clean, so there are some applications where maybe you don't need it, but maybe it's better for the planet to run it distributed. inaudible-

Jordan Lowe: Yeah, yeah, there are some.

Avi Freedman: Again, to me, it's awesome that things are changing, because who wants to get bored. It creates a lot of opportunities to start in companies and stuff.

Jordan Lowe: Yeah, yeah, just application wise. Right now, people need to take advantage of what's even available at the moment. We've got great some people, fly. io, for example. They're one of the people that are doing or enabling other people to do stuff on the Edge without doing too much work on the application. But allowing that stuff to work in these new data, like CockroachDB, for example, like these. There's a lot of technology out there that people are just starting to adopt now that allows you to do creative stuff. Localized database that will work on the Edge. And that stuff is pretty cool, I think, just using that stuff. That's all available today. Today, you can go from being in one data center, two data centers like a primary DR to actually having much better application performance at the Edge. Which, with the advent today of everyone being on mobile all day, it does make a big difference. And just getting in the right city is half the battle, I think. The last, where I will argue is, going from Chicago, for example. Being five miles away, it doesn't really matter if it's inaudible or if it's 10 miles closer to me. That's going to be a hard argument to win at the moment.

Avi Freedman: Or it's just the management challenge of 50 per city versus just having 50 cities or two times 50 cities inaudible.

Jordan Lowe: Yeah. inaudible.

Avi Freedman: So, speaking about what else is cool. You're in Chicago. And I assume you're a promoter for your fair city. Any restaurant recommendations for people that come to town?

Jordan Lowe: Ooh. That's a great question.

Avi Freedman: What's your favorite place in Greektown?

Jordan Lowe: I'm a traditionalist in Greektown. I really like the Greek Isles, just because it's big. It's been there forever. They're very reliable, and I really appreciate a place you can go to and it's going to be good every time. A lot of stuff has changed with COVID. We lost a lot of great restaurants. A lot of things have not been great.

Avi Freedman: I wonderful if my niece inaudible is still there at Fogo, the inaudible.

Jordan Lowe: Yeah, truthfully, Chicago being the variable city we are, it's like we're definitely, we're just getting going now. Like I was just telling you, I've only been out to a couple of restaurants here in the last few months. Me, having a couple of small kids doesn't help the situation. But I'm excited to get back out on the town, to tell you the truth. So, I think I have to do a re- tasting, basically. I think I need to go back out to all my favorites and trust new ones.

Avi Freedman: Horrible.

Jordan Lowe: Because I don't have a great answer for that question yet.

Avi Freedman: I don't even know if Q is still there. We need to check. Well, for any listeners that are into science fiction, the World Science Fiction Convention is in Chicago this year, September, I think, first through fifth. And we will be sampling food for any science fiction or network nerds that come to town then.

Jordan Lowe: Yeah. Come join us.

Avi Freedman: Yeah. So, how can people find you, Jordan and Deft?

Jordan Lowe: Thanks, jordan @ deft. com. That's the easiest.

Avi Freedman: Okay. Well, I am Avi Freedman. I am Avi Friedman on Twitter and LinkedIn and avi @ kentik.com and Kentik is K- E- N- T- I- K. And Jordan is jordan @ deft. com. And thanks, everybody for joining in this episode of Network AF. And maybe, we'll have a network community dinner when the world turns on this summer.

Jordan Lowe: Yeah. Thanks for the rock.