Internet traffic and current events with Doug Madory

Avi: Hi, and welcome to Network AF today. I'm talking to Doug Madory, the man who can see the internet. We're talking about internet and the news, what's going on in the world, and how it's affecting and being affected by the internet. We're talking about Ukraine, Egypt, and outages there, and other effects, and internet censorship seems to keep happening for various reasons. If you're curious at all, please give it a listen. If you're interested in Network AF, you can find us on podcasts and favor us there. We will link the show notes today to this episode, and also some of Doug's blogs you can find him at Doug Madory, a blog on the Kentik blog, and also Doug Madory on Twitter, D- O- U- G M- A- D- O- R- Y. Hi, and welcome to Network AF, today we're doing a current events update and I've got with me, Doug Madory who's been on Network AF before, Doug, you want to introduce yourself quickly?

Doug Madory: Hey, Avi. Yeah, I'm Doug Madory. I'm the director of internet analysis for Kentik.

Avi: And where are you physically right now?

Doug Madory: Presently in Bonn, Germany, I was invited to be a panelist at the Deutsche Villa, which is the public broadcaster of Germany. They have an annual conference called the global media forum, and I'm in here talking about internet disruptions.

Avi: Cool. And I guess we'll talk about some of that today, for those that don't know, you are also known as the man that sees the internet, and have done that at multiple companies, and most recently at Kentik where we enticed you with visibility, not just of BGP, but of traffic performance and other data. I think the thing really topical over the last months and top of mind for everybody is Ukraine, and what's going on there, obviously, there's a lot of different layers of that, but underneath keeping people connected is the internet. And I guess I'll just start off what's going on within internet infrastructure in Ukraine, what's been happening recently?

Doug Madory: Yeah, we haven't spoken on this topic on this podcast about Ukraine, but the invasion took place on February 24th, this year, 2022 and for the next month or so following that, that was a pretty busy time for me, just trying to keep up with what was going on, I think a lot of people were keeping up with what was going on in the news and there was a lot of things happening internet wise, and now the conflicts, which sees no sign of resolving anytime soon has evolved into a few different, I don't know what stage we're in now, it's definitely not the... We've left the first stage, maybe we're on the third stage or something where the-

Avi: That's different, horrible stage, so.

Doug Madory: Yeah, so in the outset, Ukraine was getting attack from every direction, we were seeing all kinds of outages all over the country, now the impact seems to be focused mostly on the east and the south of the country, and that's in keeping with if you follow the news, the Russian strategy now has been to try to focus its efforts on those areas that they can apply away from Ukraine. And I guess the most recent development is just seeing some of the networks, the ISPs that operate in Ukraine, getting plucked off and rerouted through Russian transit, so in the city of Kherson in the south, at the beginning of May, there was an outage, the city was down for a couple of days, and then one of their main providers came back up using Russian transit, basically going down south through Crimea through Miranda media is the Rostelecom. Rostelecom is a state telecom of Russia. They have a local agent in Crimea that they set up after they annex Crimea in 2014. And so this Miranda media then became a transit provider briefly for Kherson telecom, which goes by another retail name, I don't recall sky something, that only lasted a couple days, it would've reverted back to Ukrainian transit and the CEO published something on social media saying," I had no choice. I had to connect us to keep us online." And then that story seemed like it was over, but then towards the end of May, then there was another outage, they switched over to Miranda media again, and they weren't the only ones, there was, it's been about six or seven inaudible that we've seen switch over to Miranda media in that region. And it's not one single event, some of these are on different days that are getting switched over. So the implications are for anybody who's living in those areas that uses service there that your internet and telephone service could be surveilled, intercepted, manipulated by the Russian government is the risk these folks faced now. Yeah. And that's the latest.

Avi: And so for the most part, it seems to be people still in control of the networks in Ukraine, but being forced by economic or other projectile weapon- based incentives to form the right transit relationships so that the packets go through unknown infrastructure that could do whatever to them.

Doug Madory: Yeah. I mean, I heard from one of our Ukraine contacts, one of these was a shift at gunpoint, and so these are the same engineers that keep the telecom up, but they are now forced to route through Crimea.

Avi: Is that a first, so, a hijack as in, a route Jack, as far as you know, not via BGP exploitation, but by say working-

Doug Madory: Yeah. I bet that's happened before. I mean, I can't think of an example that comes to mind, but I doubt that's the first time that that's happened. But I would say this, that the second time around when providers started switching over and cursed on to Miranda media, it wasn't like the first time where there was completely outage and they had to bring things up. One of the providers switched immediately, the circuit was ready to go. And so maybe this was a bit more of a planned switch over than what had happened at the EMA.

Avi: Now, I guess, RPKI wouldn't be effective because it's only origin validation.

Doug Madory: Yeah. Right. It's not useful this.

Avi: See if any of their transit providers use route registries and required updates that could cause a partial outage the first time, but, and it's pretty, it seems like it's definitely a little bit of a shift forward in a... We'll just say to be non- judgmental, we'll say expands the universe of traffic engineering methods, but definitely not good. Now it's interesting because early on, it's a different kind of networking, maybe Russia hadn't thought about the implications of using roaming cell phones for soldiers and had some implications there, and early on it looked like, the internet was really being as reliable as you would expect it to be as long as one last router of whatever type it was, was like up and connected. Like it was really only when the physical damage got almost complete that those networks were going down.

Doug Madory: And even in those cases, like as been, I think publicized and properly rightfully celebrated, there's been a lot of technicians in Ukraine that add great risks to themselves have gone out in fresh rubble to go reconnect fiber optic cables destroyed in the fighting. So that's, I take pride, this is our industry, you and I we're all in this, those are our people, and it's a... Yeah.

Avi: After 911, there was a, I should really remember the gentleman's name, but he basically, he was posting from one of the mailing inaudible I think he was in telehealth, which was in the no go zone, 25 broader, Broadway, I should know that, down in broad, I think, no Broadway, down in Lower Manhattan, and he was doing hands and eyes for basically everybody because he was in the building, that was maybe uncomfortable but less dangerous.

Doug Madory: Yeah. Now people step up to the challenge and that's what's happening in Ukraine.

Avi: So if you look at the total amount of traffic to the country relatively, as the man who sees the internet for the most part, there's still a tremendous amount of internet activity and is it down?

Doug Madory: There's been a decline since February 24th by volume, we're just looking bits per second of, no, not even that, I mean maybe 20% or 30%, it's a fluctuating signal, so, but I it's not, I wouldn't say it's half. And so there's factors there, there's been millions of people who have fled the country, so those are internet users that are no longer using the internet that are gone. There's been things that have been broken and that obviously is going to hinder internet use. And then also that for the public people who have stayed, life is not quite as it was, so they may not be Netflixing and chilling as much as they did prior to the 24th of February, so that can affect traffic volumes. So there's been a noticeable decline, and we've seen this, I think I posted a list of what we saw a bunch of other outfits, public cloud-

Avi: Like CloudFlare.

Doug Madory: ...Public. Yeah. CloudFlare, Google, IOTA, which is a completely different set of data, and that actually even tracks the IOTA data, this is the internet outage detection, what's the a stand for? Project that's out of Georgia tech now, and so this is the B2B routes, the active measurements, so pinging things, they have this sort thing where they measure background traffic coming out of Ukranian address space and all those are different metrics than traffic volumes. And those two have dropped with some, not to half, but maybe 25%, 30%. But otherwise the country stayed online. It continues to stay online and I don't foresee it, yeah. The country being disconnected.

Avi: Well hopefully things go as well as they can possibly go for the people there as well as for the infrastructure and hopefully how to VPN does not get blocked on search for people from those affected networks. But my guess is there's already some VPNing going on there, I know there were some special offers that were going out basically just to help connect people, people reducing, or not charging for access to different services for people from Ukraine, so. In other excitement, Egypt, there's all sorts of jokes about sharks, with lasers cutting, international-

Doug Madory: That's right. Yeah.

Avi: But this was not sub sea, these were cables that were affected on land?

Doug Madory: Yeah. So it's an interesting situation. So his occurred while we were at nano up in Montreal, and we saw the traffic fluctuation in a bunch of countries confirmed that this was sub sea cable related, and I used the term, if you'll look at anything, I put out that there was an outage meaning that service was on... But I didn't use the word cut because I don't know that the cable was cut because that-

Avi: Incident, an event.

Doug Madory: ...that means something else, all right? You can say outage because the service was not working on the cable. So it was out but not necessarily cut, you have to be careful because the cut and in reality, the submarine cable world is so opaque, it's very hard to get a full accounting of what takes place here, but in this case we know that there was a cut on the Overland circuit. So a lot of the big submarine cable routes of the global internet connect Europe to Asia, going through the Mediterranean, and then somehow making their way through Egypt, to the Red Sea, out to Indian ocean and on to India, Singapore and so on.

Avi: If people want a picture of this, they can look at your blog post, just Google, Doug Madory and...

Doug Madory: Yeah. Well, I had an old one from a telecom Egypt presentation. I used to go to a lot of submarine cable conferences, and talk about partly what we could see from internet measurement on submarine, cable incidents. And so that was one of the first tasks that was handed to me when I joined Renaissance, we were 12 plus years ago was like," We'll see what you can do." People always ask us about submarine cable, see if you can find something, so I started to try to figure this out, and then, I mean, I it's still a very imprecise process of inference, but you can get pretty good at it. But anyway, so then these cables, these large routes, so SEA- ME- WE 4, SEA- ME- WE 5, SEA- ME- WE stands for South East Asia, Middle East, Western Europe is the acronym, the flak acronym. Those are consortium cables where a bunch of different telecoms banned together, all those are led by Singtel is the head hon show, but.

Avi: Any content providers in there or are these all type-

Doug Madory: Let's see, I guess if you went and checked the submarinecablemap. com, they usually list the owners, and so probably starting with SEA- ME- WE 5, probably about there's content providers, it was like a day.

Avi: Yeah. The Google and the Meadows.

Doug Madory: Yeah. Everybody showed up and they all started driving the submarine cable industry, which considering its importance to the global internet and global economy, these are projects that are sometimes a billion dollars to put in a cable, from one far away place to another, a lot of work goes into this and the payoff, the ROI is actually considering the risk that's involved. It's not super attractive so it ends up being a tough, it's a tough go-

Avi: People that need it, make it happen.

Doug Madory: Yeah. Right. So then, you need a consortium of people who are going to meet the captive audience, the captive audience is the ones putting up the money to build this thing because they absolutely need it, but it's always like this and submarine cable conferences, there's always a discussion about building business case, models are trying to justify the expense, and then all of a sudden content providers show up and just with endless resources to build anything, and so then, there's a lot of handing around in the industry around, is this a good thing or a bad thing? Are they just crowding everybody out? Or are they-

Avi: I mean, when people are talking about balloons and blooms for internet, there's the one hand in which it's," Hey, it's great to provide people access." There's the other, in which the cynics would say," Well that's because there are some people that think it's unfair if somebody in the world can't click on an ad." So somewhere in between there's truth.

Doug Madory: Or let me put another idea is that, let's say because it's cool, they're going to fly balloons into country Xs, well, there was a startup that was trying to make a profit at creating a local business to provide service, and now they're up against these balloons that have no, they don't have to return to profit, but they don't have to do anything and they'll just operate until they find it's not cool anymore, and just leave, in the meantime, this company goes out of business and these guys go bankrupt. So there can be some of these, sometimes these unintended consequences.

Avi: Let's try to get them on a future podcast. It's always interesting. We still need to get Jared with his local ISP and we got Elliot Nosse.

Doug Madory: Sure.

Avi: I need to follow up with him about his... I talked with him about why would you start an ISP nowadays? But there's some cool stuff.

Doug Madory: That would be it. That would be a great conversation. I would love to make one in live.

Avi: Anyway, sorry. So back to Egypt. The impact in what countries performance.

Doug Madory: Right. So let's see, we knew there was a major submarine cable outage affecting basically countries in East Africa, the Middle East and South Asia is where it was showing up in our traffic stats in the end, it was an Overland circuit over Egypt. So the way all these cables go is, ships will go through the Suez Canal, but the submarine cables don't go through the canal. They have to come up on land and they go either, there's a couple of routes that come up in Abutulara or Alexandria or the Port cities and in the Mediterranean, they cross the desert and then drop back in the Red Sea. Or there's also an option where you can run it in the cement casing on the side of the canal itself, you're not in the water, but you're up out of the ground or out of the water, but there's inaudible right, if there's a break and get into it. But so everything comes out of the water and then it goes back into the water. And so because it's out of the water, that means if something breaks, there's a fix, you can send a technician over there and it's a matter of hours versus, it could be days or weeks if it was in a deep sea location, the submarine cable. But so in this case, again, it was something that was on land, they could get a fixed connection, a technician there to fix it, and that was just a matter of, I think something the order of four hours, but we could see, yeah. A lot of different countries impacted, others could too, so I mentioned in the blog or some of the other outfits that, or other internet watchers like me, we try to call what would be our unique view into this. And so I did pull up one, so inaudible which is cutter telecoms, new branding, as of, it's probably like a decade old now, but that one I thought was the nice cleanest view, it's in the blog post of a visualization of the loss of this circuit knocked out a couple their transit providers, there was a shift, they basically stayed online, but it got re- jumbled over four hours to rejigger their transit to stay online. And then the other insight that was flipping around, well, we do a lot of cloud measurement, at Kentik we do a lot of synthetic performance monitoring between the various cloud regions of all the major cloud providers. And so I know just doing this in the past that if you see a route like this, that's going through Egypt, through the Mediterranean, this cloud providers have to use the same stuff, normal people like you and I we all have to use the same-

Avi: Did he do sham of IBM says," Without the network, there is no cloud." Or the alternatively says," Network is the water of the clouds." So.

Doug Madory: Okay. There's many beautiful metaphors. But I was thinking, I was like, well, I know because we looked at this, I think a year ago there was something like this where we also noticed this and I was like," Let me go check and look at all of our routes going across between any cloud region in Europe and cloud region, South Asia." And sure enough, there was, you could see the impact fairly well, and what was interesting was that there's different impact by cloud provider, and so based on our stats, based on our data, you could see the initial blip for AWS connectivity across this route, but it resolved itself fairly quickly. Azure seemed like it had a period of a couple hours of higher latencies, but no packet loss, but by Google cloud seemed like they didn't weather this one that well, no matter which direction we went, we saw higher packet loss, higher latency, so that's interesting, so that has to do with how the cloud contracts it's routes and how well it can fail over to backup connectivity. And for these links, we're responding a lot of stuff that was falling over.

Avi: Well, it's interesting because the first question I was going to ask about this is, it can be difficult for people that are not in the routes and the packets and the bits and the bites to understand that in some ways, connectivity can be much more network topological than geographically topological. It's like how the networks connect can affect performance and routing outages and things much more, so just because two things they're in a country doesn't mean they're going to have good performance or just because something is three countries away doesn't mean it'll have worse. So when you look at what happened with the circuit outage, were there if you look for the non- cloud, were there some, were any countries entirely cut off or was it really just provider by provider for given, so if there's fewer providers and one big one is affected, that affect more of it, but doesn't knock out all the traffic to a country, were there any countries completely cut off or was it-

Doug Madory: Somalia, but that's, they're a unique situation, there's really only one submarine cable serving them. So the easy cable and they have just limited connectivity-

Avi: inaudible satellite that we may not geo.

Doug Madory: I mean, they don't, well, what I've seen over the last 10 plus years, when a submarine cable arrives the market for bulk satellite, just dries up. I mean, we saw this in Toga, just in January, I mean the volcano had a dish been up and ready to go, it might have gotten knocked over anyway, but it wasn't, it wasn't ready to go, they had long since gotten rid of their backup satellite, which was how they had stayed online prior to that submarine cable coming online. And so we'd see this all over Africa where countries that, well, as soon as that submarine cable came out aboard or came ashore, everybody switched over, satellite's gone.

Avi: You're not going to do gig bit to the home on 24 megabit uplink by satellites, so.

Doug Madory: I mean, yeah, there's a capacity thing, but also they don't even seem to keep it around, it's still even too costly as a dormant backup. It just seems to be the case.

Avi: It is very expensive, but Leo stuff is making it less expensive. So then if I could make a further leap, if you go to the snow crash view, the cyber, the dystopian cyber punk view, in essence, the large tech companies, the cloud providers will be the countries of the future. So countries in the future-

Doug Madory: Maybe we're already there. We just don't know it.

Avi: inaudible that we are, so maybe there's two kinds of Countrywide impact, one is the internet countries and one is the physical countries, and I mean, these are large communities that need to collaborate. And so they also, and I know they all do work very hard at finding redundant multi path connectivity and looking at performance and all these things.

Doug Madory: That's true. I mean, I don't want to disparage anybody. These guys are the best that they have a lot of smart people, a lot of resources. So it was actually surprising to see the severe issues that our data was showing for Google.

Avi: So I guess the last question, topically, so I saw your tweet about, it's exam time in Syria, again, let's shut the internet down. And I was just thinking, is this really the new norm? Are we just going to be seeing this is this full employment for you and others? Do you think anything's going to change?

Doug Madory: This is one, that's hard to get your mind around. I do. I even just, I just explained this, I think at this conference in Bonn Germany, again, people were like," What is this?" Or they just can't understand this, and again, for those who are unfamiliar with this, the situation is so take a rack, which was the first one that I started reporting on in, I think 2015. So at that time you imagine that ISIS has taken Mosul and a threatening Baghdad, there's a heat wave, riots for corruption, all these things are happening and the internet goes down and I had a well placed contact in the Iraqi government. And so I'm reaching out to him, I said," What's happening here?" And he's like," You're not going to believe this, it's student exams." I was like," You're going to have to explain that to me." Like, I assumed it was something. There's all these huge things happening in this country. And that's what they turned the internet off for. But here's the explanation is that, in a country like Iraq, these are this national tests at sixth grade. If you don't score high enough, that's the end of your public education, and to be-

Avi: There's a surprising number of countries where there's something like that that goes on.

Doug Madory: Yeah. I mean, it's, I don't know for anybody who read the book for economics for years back when discussion of when you make the consequences so severe, you justify, you end up justifying cheating, you're justifying extreme measures to avoid these extreme consequences. And so in this case, if you are a parent in one of these countries, your kids have got a lot of cards stacked against you, against them, and so they want to get any kind of help they can, and so then it ends up happening is over time, the test get compromised, cheating becomes rampant. And so then this is like a last ditch effort to try to regain control over a system that's gotten out of hand, but it's not, I guess it didn't happen in Iraq this year, but yeah, Syria, Sudan, someone reached out to me to asked me to look into Algeria, which I guess there doing something similar, sometimes it's the whole country's down or so I just mobile service. The way it works in Syria is that they take down the back backbone of the country while they physically distribute the tests. And then when they bring the backbone back up and the mobile service comes down while the kids are taking a test, anyway, this is what happens.

Avi: Better. I mean, you would think a Faraday cage would be better, but no.

Doug Madory: Yeah, but that's what...

Avi: And there's no... I mean, you follow some of the policy stuff, there's no UN group talking about internet as a human right and trying to form a treaty basically saying that," Internet won't be used for..."

Doug Madory: So yeah, there are digital rights organizations that fervently push back on this access now is the one that probably is, I'd argue that maybe the most prominent they're a organization that we support with some technical consultation. I try to let those folks know about things, for example, when this submarine cable cut occurred a couple weeks ago, that's now on my list is to let some of the folks in the digital right space know that something happened with that's infrastructure based and not government directed shutdown, just so they... Because they're going to start receiving complaints, or receiving people claiming that there's a shutdown or something. And so that's a high up on my list is to let those folks know something just happened and it's not, something broke, it wasn't a government directed thing, but I know that they try to push back, but in the end we still live in a Westphalian world of countries are absolute sovereigns and if they want to make a bad decision, they can do so.

Avi: We're fortunate that internet governance itself has survived internationalization, I'll just say, I'm not saying it's necessarily good for you.

Doug Madory: So far.

Avi: So far? I'm not saying it's necessarily good for any one company, a country to be the only, but yeah, when things turn to politicking and interests and especially with the value of IP space and domains being what it is, it's good that things are working as well as they are. So any other hot topics, things going on connecting the infrastructure and news or just infrastructure wise?

Doug Madory: I think that's all I've got for today.

Avi: Well, I actually do hope it'll be a calm next few months and history, especially with we going on might not, if history projects forward, you might not think that, but I certainly hope that and we'll do an update to this as there is more news.

Doug Madory: Sounds good.

Avi: Thanks Doug, for joining and so late from Germany.

Doug Madory: No problem.

Avi: Thanks to the audience for listening to Network AF and this episode, you can find us on Apple Podcasts, at our webpage and your favorite podcast listings around you, and Doug, people are curious to contact you. How can they do that?

Doug Madory: Yeah, usually you can follow me on Twitter and send me a message, to get it go, I love talking about this stuff. So Avi can vouch for that.

Avi: Absolutely. We've talked for hours and hours, and I'm avi @ kentik. com. I'm Avi Freedman on Twitter and LinkedIn. Thanks again.