RIPE 89
1 November 2024
11 am
Closing Plenary
MASSIMILLIANO STUCCHI: Hello every one. We are ready to start the last session of this meeting. And as first talk, we have something about zombies which is in line with what you had last night.
ANTONIO CHARITON: We are going to talk about BGP zombies, as Max said. This is like, to be clear, this is the internship that we did over the summer while we were lucky to host her as Cisco ThousandEyes. Before we begin, let's see what are BGP zombies, or stuck routes, as some people call them? Or ghost routes, as they were being called many years ago.
So in BGP, let's say you have a few autonomous systems, and they are connected somehow to each other, and if you have a prefix and you need every one else to learn about that so they can send traffic to you, then you generate what's called and update message, that is an announcement, specifically, and it contains the prefix information. And this message, you then send it to all of the connections that you have with the appropriate relationship, and then they send it to their peers and eventually the entire Internet knows about your prefix, and they can send traffic to you.
If the, if a link is down or if a prefix is unavailable, then you generate another type of update measure which is a called withdrawal. In this case, its the similar thing, so every one that you send the announcement to, you have to send the withdrawal to them as well. The problem is that in some cases, these messages may be lost, it may be network issues, it maybe software issues on either side but for whatever reason, these messages could be lost, and right now, we end up with a routing inconsistency. So the first two ASes correct see say that this prefix does not exist but the third AS thinks that this prefix is still there, and obviously this routing state is inconsistency. And if the third AS has downstream customers, then they are going to not inform them about the withdrawal either. Because they never learned about it. So, every customer downstream of that AS is going to be infected.
So why is this a problem?
.
Well, there are several reasons. Of course the most important is that a router's job is to route packet correctly and it can't if there are routes stuck there. The other thing is that there are paths that may be active that you don't know about, and for example, if you have a DDoS attack and you do some DDoS mitigation, you send your traffic through a provider, through a different link, and then the attack is over, you switch it back to the original path. However, traffic gets stuck and is still sent through this route. And if you have, for example, if you pay by the hour on your mitigation solution, you are still getting charged for all of this mitigation that's not actually happening.
Then of course you may have incorrect alerts that you may be getting. So, you may have a very noisy alert system, and eventually, the number of routes in the Internet will keep going up, if this happens.
And finally, you know for those people still using IPv4, if you sell it, you know, a small percentage of the traffic may not arrive at the new owner, it may still come to you.
BGP zombies are a problem if your routers have someone else's prefixes stuck there, or if your prefixes are stuck in someone else's routers. Both of these are bad for your network. And by the way we saw that this can happen with withdrawals where you think a path exists that it doesn't. However, it can just as likely happen with announcements, which means that some path will never be used.
These are much more difficult to study. However, its something that could happen as well.
But you have some problems as operators. So we wanted to see if its a big problem, like if its something that you should care about, or if its just something that you can ignore. And in order to do that, I will Anna is going to tell us that.
ILIANA ZYGKOU: In order to answer this question, we first looked at the existing literature and the first paper to look into BGP zombies was published in 2019, and it started stuck routing involving beacons. So, RIPE RIS beacons are IPv4 and IPv6 prefixes that are announced and withdrawn periodically by the RIPE RIS collectors to their peers. Starting at midnight, the collectors will announce the beacons every four hours, and they will withdraw the beacons two hours after its announcement. And thus, the reachability of the beacons can be visualised as a path strain as we see here.
So, when it comes to detecting BGP zombies the most important tram terse we can find is after how long since the prefix is withdrawn, should we consider a remaining route as stuck or zombie. And the authors picked a conservative thing of 90 minutes in order to ensure that they wouldn't falsely report as zombies any delayed withdrawals.
.
So, the authors actually did the experiments of three different time periods of of around one‑and‑a‑half to two months, and actually here in the results table of the paper we fix a typo on the second line to prevent any confusions around the experiments. So the authors found that in just five and a half months, there were more than 5100 zombie out breaks, which is very impressive, and that's why we wanted to repeat this work in order to validate the results and we want to repeat using the raw BGP data provided by RIPE RIS.
And surprisingly, when we did that we actually found 640 more zombie out breaks than the papers. And an increase of about 13 percent. So, where could this discrepancies be coming from? Actually, a methodology was using different tools than the papers one. We were using historical BGP data provided by RIPE RIS, in order to be able to reconstruct a peer stream at any time point. On the other hand, the authors were using the RIPE RIS looking glass in order to detect BGP zombies in realtime and then they later filtered out any false positives by using the RIPE RIS raw data.
So, having seen this discrepancies, we wanted to ‑‑ we were motivated to further improve the detection methodology. In order to be able to tell apart a zombie out break, as we see here, from a complete and successful withdrawal.
And actually, the RIPE RIS collector marked the BGP announcements of the beacons with some very valuable information. Most specifically they populate the BGP attribute aggregator IP address with this value where the last 24 bits represent the number of second that have passed since the beginning of the month until the time of the announcement. And this allows us to uniquely I've beacon announcements within the same month, and how can this be useful?
It actually let us avoid double counting on zombies. For example, if a prefix announced that 12 am, gets stuck in a router and it remains there even after the new one, the new announcement at 4 am and the new withdrawal at 6 am, we still count this remaining route as a zombie twice, but only once for the time in which it was originally announced.
And independent, when we apply this additional filter, we found a significant reduction in the toll number of zombie out breaks reduction of around 33 percent.
And also, we furthermore rerevisited some of the results that the paper was also doing and we found that one fifth of the RIS RIPE prefix were not affected by BGP zombies at all. And such would of around 1.6 percent as IPv6 and half percent for IPv4, to present a zombie.
Finally, as expected, over 90% of the time, the AS path of the stuck route was not the optimal one so the BGP selection algorithm, but instead, it was the one that it emerged during the path handling process after the prefixes withdrawal.
So, having revisited this previous study, and having improved the methodology, we came up with some questions that would help us understand the zombie topic more extensively. Once that question is how long can a zombie survive? Is it just a couple of hours or can it go up to days or even weeks? Another question is: How does a zombie's visibility change over time?
.
But, answering these questions requires that the prefix of the stuck route is not announced again very soon, because then the new announcement is going to wipe out the stuck route. And weir going to lose this information.
And unfortunately, the RIPE RIS beacons provide us with a short interval of two hours between two announcements of the same prefix.
So in order to address this issue, we came up with ‑‑ we implemented our own BGP beacons methodology, which we called BGP clock, and that is because in the prefix itself, we included the hour and the minute of the time of the announcement. So, we would advertise a prefix every 15 minutes, and we would withdraw it 15 minutes later. Resulting in 96 different prefixes per day. Of course, these prefixes, as you can see, they would be recited every 24 hours. So, after running experiments for several days we modified the prefix to also include the day of the announcement. And in this way, the prefix would now be recycled typically every two weeks, and these would allow us to study the lifetime of a zombie route more extensively. Of course, following security best practices, we published we created our own ROA on the hype portal for the covering prefix /32 in order for our announcements to be marked as RPKI valid and in order for them to be able to propagate fully in the Internet.
So, in RPKI clock, the announcements were made by Antoni's personal AS. Highly inter connected AS with more than 1700 neighbours. And these ‑‑ this is suggested that the announcements would fully propagate to the Internet very rapidly. In general, the notion behind implementing our own BGP methodology was that with more prefixes that could potentially survive as zombie routes for an extended time period, more than just two hours, we would achieve more diversity and less noise about which prefixes get stuck and also we would be in the position of studying how the visibility of zombie route changes over time.
And indeed we found some very interesting cases of some quite stubborn and impactful zombie outbreaks.
So this one, was found stuck in the routers of six RIPE RIS peers, three hours, even three hours after the prefix's withdrawal, and using the AS paths of the stuck route, we found that the probable cause for this zombie outbreak, it was actually a free AS, with more than 200 ASes in it's customer code. We hypothesise sized that probable all of these AS had been affected but we can be certainly only about six of them which are connected to the RIPE database RIS collectors.
And in this light here, we viz lies the zombie outbreak by using the @ paths of the stuck route. So in order to create the zombie AS graph, and in the red square here, we highlight the AS that we assume was the cause of this problem. And how we determined this is that the AS graph starts as a single chain with the original AS as the route, and then at the red square it branchs out to multiple sub traces who is are the RIPE RIS peers. So, if, for example, an AS before the annotated one, let's say this one, was causing the stuck route, then, its own downstreams would have also been infected, and we would see the stuck routes in them as well. So this branch here would have cured earlier at this AS. Similarly in an AS after the annotated one, let's say this one, was causing the problem, then this would mean that all the ASes at this level somehow missed the withdrawal of the same prefix and at the same time. But we need to clarify, we are not here to point figures to ASes. This zombie outbreak is not necessarily a problem caused by the annotated AS. It could be that the previous one failed to propagate the withdrawal message to them.
Our goal here is to be able to to understand where the problem is coming from north to facilitate its resolution.
Then we have two additional larger zombie outbreak cases. The first one seems to have come from a large German AS belonging to core backbone with more than 2,000 customer ASes, and naturally we saw more ISP peers having this stuck route as well, namely 21. And actually, even Ben Cox, in their own announcement of BGP zombies, they have observed this having stuck route.
Finally, we observed that another German AS, my lock, with a few hundred ASes in it's customer cone, might have caused very quite persistant and very long lived zombie outbreak. And now, we will let Antonio continue.
ANTONIOS CHARITON: Thank you. So, we were done with this research, it lasted about a month, and we didn't want to have it running, this generates a lot of BGP updates, we wanted to be careful with your equipment, we wanted to do responsible research so we stopped all the advertisements, and we deleted the ROA from the LIR portal, and then something interesting happened. Because that night I got an email from RIPE and for those that can not read it, this is an email that you get if you have RPKI invalids. It states if this is intentional, go issue new ROAs, but if its not, then perhaps you need to look into that, there is maybe a hijacking or something. I keep getting this email every night, three and a half months later, which you know its very interesting. So the routes are still stuck, I am happy to report that zombies are still there. It's been over four months, and well the RIPE RIS cleared recently, so there goes that, the email will stop soon. BGP tools still has three of them in there, they are still stuck, and Hurricane Electric's BGP tool kit had four. We are now down to two, I am happy to report that we have a half life of a few months for BGP zombies.
And these prefixes have been invalid for months here. So, if you want to test, you know, RPKI adoption and to those pesky Internet exchanges and this transit providers, they do filtering and they don't let you test the other ASes, what you do is you can just get a route stuck somewhere and then turn it into invalid and now you can see who keeps it and who doesn't. An interesting point about RPKI research.
So, what did we learn from all of that? The first thing is that zombies drop over time, and here you can see on the X axis, you can see the time, this is like almost at the three hour mark, you can see you know they start going down, and however they start to plateau after a little bit. This is a special zombie case that we had. Now, you want to look at the orange line, and the thing is we found two ASes that are RIS peers were almost 10% of the time the routes were getting stuck there, so we had so many routes that created problems for measurements where the orange line is if we exinclude them.
Something else to point out, is that at some point, the zombies start going up. So what's up? Like, we wanted to explore that a little bit further. And it turns out that around 3 hours, we had a new announcement. We looked into the data and someone created an announcement for a route that was withdrawn. And we identified that this happened in Telstar Global, so something you know a link went up again, configure changes, we don't not what caused it but something caused a full table to be sent and this full table included the stuck route. So the interesting thing here is that these zombies can reinfect hosts, like a lot of their downstreams were ‑‑ had successfully removed the route, but you know something changed and they got the route again. So it's really easy for these zombies to stay forever, especially if there is some kind of reinfection being triggered by normal operator things.
So with can he do something about all of that. Because that's eventually the question. And thankfully the answer is yes. There is a draft in the IETF, it's practically an RFC I would say, which creates a second timer for BGP sessions, it's called the sent hold timer. And what this does is that basically if you can not exchange data with high certainty, with a peer for 20 minutes, you take the session down, and you do this to protect your network and all of your downstreams. And we think, like we don't know yet, but estimate this is going to solve 30 to maybe in some cases even 40% of the stuck routes. It's not going to solve everything. But it's going to do something.
However, for that to happen, we know that vendors have to add support, there is a BIRD open BGP and FRR that have added support. No hardware vendor has done it yet so you can be the first if you are in this room. You know, that support is still available for anyone who wishes to claim it. And if you are not from a vendor but you buy equipment from a vendor, you can ask them to implement it, because then it has to make it to the stable releases and all of you here in this room have to update your routers and it's going to take some time. It's not something that will be done by the end of the year probably. Maybe the decade if we are lucky.
But maybe that's optimistic.
So, we think that this is a very good path forward please ask that from your vendors, it's been great to have that and ideally ask them to have it on by default. That's going to be awesome.
With that, we have three blog posts that you can download the PDF and click on these individual icons to, read more about, and I think that's it. One last note I may add. The research here was done with a personal AS, as you have seen. So they are actually useful. Thank you.
BRIAN NISBET: Okay. I think end of the decade is optimistic. Hardware vendors always put these things in so quickly. Okay, that's a full cue there I think at this point in time. We are going to start on ‑‑ with Jen.
JEN LINKOVA: First of all, you are very cool. Thank you very much. One clarifying question and one real question, IBC and Cisco confidential on the slide, should I be concerned?
ANTONIOS CHARITON: I hope you weren't participating attention to all of the slides at this time. It's too late I guess. The material leaked.
JEN LINKOVA: The Google tool is signing an NDA on the bottom, yes? Sorry, I missed the very beginning, so, I am curious, you are saying it's normally caused by the fact that the peer cannot reliably send announcements, right to the other side? I wonder first of all, what's happened to the traffic in this case really? And secondly, have you actually talked to those people to find out if it's a bargain of them the particular implementations, practically what does it mean? Do we have to use a particular vendor or software version, could it be fixed by a bug fix?
ANTONIOS CHARITON: So, we have identified some vendors that, where this happens way more than others. And we are in the process of notifying them internally and externally, so... eventually, we want to get in a position where this doesn't happen. It's almost always bugs like the TCP window is zero only in the BGP session, so data still works. Or you know there is some state corruption in one of the peers, like in so many cases.
BRIAN NISBET: I think it's interesting that people really notice the Cisco confidential bit and have not made any comments about all the other internal confidential only things that have been in the slides this week. Something about very popular names.
ANTONIOS CHARITON: We are clearly the most popular.
AUDIENCE SPEAKER: Alexander: Thank you very much for your report. I have one question, one comment, or okay. I would love to know if you tried to talk to RIPE RIS guys to ask them to change to the way the beacons is generated on their site so it will look like what you did? Because I find it quite useful.
ILIANA ZYGKOU: No. Actually we did not reach out for this modification. We wanted something to happen very quickly. So, we also were in the position where I don't have his own personal AS. We had a large IPv6 prefix, so it was easier for us in terms of time to do that.
AUDIENCE SPEAKER: I would love to see collaboration between you and RIPE RIS if future so that this research will not be just one time research, but time measurement, but it will be continuous measurement for all of us.
ANTONIOS CHARITON: I may add something before you move on. We want to do this again with v4 as well, because it's still being used in a few places, and I tried for expense the last eight but my corporate card wouldn't work. We are trying to work out a solution for that.
AUDIENCE SPEAKER: I know some company not commercial that still have some address space. Okay. The second question is, have you tried ‑‑ you made a measurement and you see that there are stuck routes of in some of the peers that are, from RIPE views or from RIPE RIS, is this possible to allocate them, just to make a list of the peers that gives you stuck routes? Because of course they affect measurements if we are trying to use this data?
ILIANA ZYGKOU: We can provide publically a list of this peers that pear to have ‑‑
AUDIENCE SPEAKER: My question is it possible?
ILIANA ZYGKOU: Yes, it should be.
ANTONIOS CHARITON: It's techically possible, let me add here, as a still employed person.
BRIAN NISBET: To be clear the queues are cut at this point in time.
PETER HESSLER: As a network operator, how can we detect that there are stuck zombie routes within or network without doing such a large scale experiment? How can we I've, and especially I've zombie routes that are stuck that are not part of such an experiment?
ANTONIOS CHARITON: Buy a Cisco product.
So the difficult thing here is that you don't know the ground truth. If you monitor your routing tables you can be certain whether this is intentional or if this is something that happens at scale. So I think the best approach is to have this kind of monitors that collect data, maybe you can have private BGP feeds that augment that and you can figure out where the stuck routes happen. But we will need to productionise that's correct I don't think it's difficult to do a community project, that would be awesome.
AUDIENCE SPEAKER: I have checked my ticket system. You have never requested a session with us but you actually have an IXP port, that was interesting. I am working here, are you reaching out to those operators actively? That's one thing. And the second thing is, I sometimes do get people who end up with me, blaming me for stuck routes and every single time it's not my network but somewhere else. It seems to be quite hard to troubleshoot such routes. So it would also be appreciated to have more like troubleshooting material. You can point people to.
ANTONIOS CHARITON: Our goal is to do that. We have not reached out to all of the networks. We want to do v4 first so our list grows a lot and then do it.
AUDIENCE SPEAKER: I would have to two notes of suggestions. First, would you consider randomise the announcements instead of doing it strictly periodic because strict periodic could interfere with something like a configurations no vantage points so if you do randomise it you can have have better stuff.
Second point, I think that there is perhaps case for BGP speakers to do periodic enhanced ultra flash.
AUDIENCE SPEAKER: You mentioned that you shut down the IPv6 experiment and after you collected the data and that you might start up a v4 experiment. Do you perhaps plan on continuing the IPv6 experiment not as a measurement but sort of as a services for troubleshooting these stuck routes probably you know like it could be caused by vendor box and such. I feel like that the route churn it causes does less harm than good.
ANTONIOS CHARITON: We want to do them both and we'll try to keep both. I cannot promise you anything about v4 because all of these v4 addresses are personal, so I may have other commitments for research or otherwise. But you know, maybe we can get some sponsorship from our lovely brokers outside where we can leave the v4 on as well. But v6 will probably stay.
AUDIENCE SPEAKER: Maria. I'd like to first say thanks to Ben Cox and Job Snijders who were actively pushing this RFC even to the extent that they approached me at the NANOG meeting saying hello, have you already heard about our road and saviour BGP and hold timer? I basically had to go and implement it. The other thing is we had to test it against something, so, there is inside BIRD there is also an option which was originally called break the receiver side as the Cisco does.
ANTONIOS CHARITON: I wouldn't know anything about that.
AUDIENCE SPEAKER: For now; you can use it if anybody wants to implement it, you can use it against BIRD with this option to actually test that it actually gets killed.
ANTONIOS CHARITON: Thank you. First of all, yes, I use BIRD as well, so I will try to turn it on in every session soon. And the other thing is yes, like, Ben did some great work how to promote this and we are here to further that work by telling you all about it. And remember to request it from hardware vendors, present company included.
BRIAN NISBET: Thank you very much.
(Applause)
So, that is our last presentation, but now we move on to the what's been happening this week section of the Closing Plenary session. So, I think Rob, you are hopefully somewhere in the room, to give the Code of Conduct update, or somebody else from the team gets to give it. Franziska stood up first. No, it's you Franziska, you win a page prize.
FRANZISKA LICHTBLAU: Welcome to our transparency report update from RIPE 88. What have we done so far? We have reported between then we got two open cases. We had team members be who needed because there was a conflict of interest. We had contact with the reporting party and that all went well. We concluded one incident. Indeed it was a violation of the Code of Conduct. We did apply consequences and in between we did another on on going report.
We got additional reports via email, so this means we usually sit down, form some assessment groups, those were loosely related to what happened they last meeting, so this is techically not RIPE 88 but it was related so we count it as this. We recused some team members because there was also a prior conflict of interest situation. We went and contacted with both parties. We concluded it. We had one incident that was a potential violation of the Code of Conduct. The subject of the report agreed to our actions. And we could conclude this to our satisfaction, and again we have one on going report, so we can't comment on that.
And now to the absolutely delightful news. No report for this meeting. So...
.
(Applause)
.
So every one has been behaving at their best and this is how we like it.
Again, every one who was at the Community Plenary knows that we are still discussing to change some details of our processes how we report, handle and deal with reports. Here's a quick reminder, how do you report? You can report to any of us. You can report via email to the Code of Conduct form that will go to all of us. We are approachable individually. We are around here, we try to be clearly marked and visible, and if you wonder whether something you experience is worth reporting, come and report it. If we figure out it wasn't anything. We will figure it out and it's all going to be fine. But we have a nice short link ripe.net/coc so you can look everything up there.
We still have happy to take new people on the team, because as I already pointed out in the Community Plenary, we try to load share, we work in different time zones, different perspective, this is all very good if we have more people, and on that more rather sad note is, Vesna is leaving us. She is stepping down, she has done a lot of long work in the entire team, those who remember still the trusted contacts, she was already on that, and that brought an insane amount of experience to our team, but it's time for her to step down because also we have term limits, and she decided this is time for her right now.
And that doesn't mean that she steps down out of our community or the general topic of interest, but this is what it is, and I would like to thank her very, very much for all she did for us.
(Applause)
.
Any questions?
AUDIENCE SPEAKER: Just a comment. We have a code for staff black, maybe you need a lanyard code for the Code of Conduct members, although the faces are displayed, so maybe ‑‑
FRANZISKA LICHTBLAU: You mean a QR code? The lanyard?
AUDIENCE SPEAKER: Like the one you wear.
FRANZISKA LICHTBLAU: We had buttons but they turned out ‑‑ they turned out to be a little bit small, and we have it on the badge actually this time. We got upgraded, but yes, I agree we should be super visible.
AUDIENCE SPEAKER: Maybe things on the bottom. Something, you know.
AUDIENCE SPEAKER: Tina. Dealing this with in other communities, I am starting to see while we want to keep everything as anonymous as possible, to protect both parties, for the community it probably would be nice to have very generalised categories of what the issues are and may be aggregate it by year or something to distract from maybe so we still keep the anonymity appropriately. So we can see where we need to improve, or if we are improving, we used to have this kind of complaint and now we don't see it any more, that kind of thing.
FRANZISKA LICHTBLAU: Thank you. Actually, yes, this is also a call to us to being consistent in what we do because in the past in the reports we actually did that. We broadly categorised into cultural misunderstanding, not taking care of others feeling. What I can actually really proudly say about this community to the best of my knowledge until now, we haven't received any harassment, sexual harassment related report which is something that makes me personally really happy to be in this community, but yes, we should definitely do that again, thank you.
MASSIMILLIANO STUCCHI: I don't see any other questions, do we have anything online? No, nothing online. Thank you Franziska.
(Applause)
.
So, now, we have Rob, which I see running towards the stage. He will give us the technical report. The presentation we have all been waiting for.
ROB DE MEESTER: Good morning every one. My name is Rob and I would like to present you with the RIPE 89 technical report.
The last time we were here was RIPE 60, and things were quite different back then. It was long before I worked for the RIPE NCC, but I had a look at the technical report in the archives, and only 6 percent of the traffic on the meeting network was IPv6, and I guess it was much colder back then because the tech team equipped some of the Chairs with complementary seat warmers. But I was happy to also see some familiar faces, for example on this slide, where they introduced at that that the meeting would be streamed in flesh for the first time.
So, this is us. The technical team. We are mostly responsible for, as the name says, the technical aspects of the meeting. Unfortunately Noella and Alexander were not with us but they were supporting us remotely from Amsterdam, and this meeting could not have happened without them.
Before the meeting we take about three days to set everything up and then after the session we take everything ‑‑ we pack everything up in three hours, and this is a small selection of all the stuff that we bring.
So, our equipment arrived in Prague on Wednesday afternoon, and we received this email from the hotel that one of the flight cases fell out of the truck during the of loading, and I quickly checked the contents and it turns out there was some fairly crucial equipment for this meeting in there. So for two days we were in this like fight case situation where it was either completely fine or everything was all broken. But luckily, it turned out everything was undamaged and to be fair, this probably isn't the first time it happened. It was the first time we were told about it though, so fair play to the hotel for being honest.
So, this is our meeting network. This the the ogical topology. All routers that we have run are virtualised on to small super micro servers running VM ware, and for this meeting CESNET kindly provided us with connectivity. One the cool things about our job is that every location we visit comes with its own special challenges. So for example, in this case, all these sockets in the hotel had already been patched. All the fibers had been utilised. The hotel IT company inherited their entire infrastructure from a vendor, a previous vendor which had left no documentation for them. The patch numbers did not match the patch numbers at the walls did not match the patch numbers in the cabinets, but luckily they had CD visible or LLDP, that made it easier to I've the correct ports.
There is a 1‑gig bit connection for everything the hotel does, which also includes hosting the entire, the Internet for the entire shopping mall that is adjacent to us, because that was likely going to be problem for us, CESNET then arranged a dark fiber for us to use, so if you experienced any issues ordering your double whopper, it wasn't us.
We requested to use the hotel switches for the network. But it was rejected, so instead they freed up some fibre pairs for us and marked cables, which worked.
They also allowed us to place some equipment in their server rooms, for example, here you see our two core switches, and the two hypervisors, they host about 25 VMs, mostly deployed with Ansible.
And thanks to the hotel, we were also able to address one the most heard complaints from RIPE 88, I am not sure if anyone remembers, but I am happy to report we finally achieved proper connectivity in the rest rooms. Actually, I heard connectivity over there is pretty decent at this meeting, but I really want to stress it's a happy coincidence, it's not a new standard.
For the meeting network, we like to use as much OpenSource technology as possible. This is some examples of the stuff that we use.
During RIPE 88, we encountereded some problems with OR routers that run Oracle Linux 9. They did not like receiving a full BGP feed, which resulted in very high CPU usage, and after the meeting, Robert Scheck got in touch with us and he told us he saw a similar issue in their environment, and he had reported it to RedHat as a paying customer and therefore it got fixed and now the fix is available to every one including us, so thank you Robert.
And then when we were setting up, we had an interesting incident with our connectivity. It worked. But every now and then the BGP session went down. And it turned out CESNET did not propagate our prefixes to the rest of the Internet, which caused our DNS resolvers just to not work. BIRD is configured to connect to the RPKI validator using a DNS name. It got stuck for 24 seconds, trying to resolve the domain name, and this was enough for the remote router to consider that. We reported it to the BIRD developers and their answer was just don't use DNS. Or upgrade to BIRD 3, which is multi‑threaded but I think also in alpha.
Just like last meetings, we had three separate networks this time. The main network is IPv6 mostly, and if you want to learn more about that, Jen did a very good talk in the IPv6 Working Group session at the last meeting at RIPE 88, so I really recommend checking that out in the archives. We still see a decline in the usage of the legacy network, which is a very good trend. And we are still really interested to hear your reasons for why you would still be using the legacy network.
One of the ‑‑ like the main reason that we know about that we have seen happen this week is because IPv6 mostly and VPN clients do not play nicely with each other. It's often related to a feature that vendors call IPv6 bypass, so basically they disable IPv6 as soon as the VPN is stet up to prevent any data leaking through it but unfortunately on an IPv6‑only network, as soon as you turn of IPv6, there is no connectivity any more. So we behaved this, for example, with Proton, Cisco. There is probably a lot of clients out there with similar issues.
We presented on this before, but our presentation is mostly the same. In each room we have two Mac minis, connected to a matrix switcher and we prepare everything at the tech desk for the next presentation in the back and at the right moment we switch the pre‑set. That setup just works really well for us.
But then everything went black. Once in a while the projector just completely went black, for four seconds, which doesn't sound like a lot but when you are standing up here or sitting at the tech desk it feels like an eternity. We tried replacing every component that this could possibly be related to, but eventually we traced it down to something in the beamers, so the AV company tried a lot of things to fix it and eventually we did.
So, in a drawing our setup looks like this. The only difference from last meeting are two HDMI audio splitters and there is a bit of a story behind this, because some of you might remember this scene from the last General Meeting, where we tried playing a rerecorded video and it froze multiple times at really awkward moments. It was because a driver issue with an old sound card and we were unable to fix it live. In the end my colleague walked up stage, played the video from his laptop and bent the microphone to a speaker so the sound would be audible in the room. To be honest this moment has been haunting me in my dreams ever since. So to fix this we added HDMI audio embedders to the chain and we got rid of the sound card completely and we got it test it already on day one. Geoff Huston did a presentation with a pre‑recorded video and it worked really well so since Monday I slept like a baby.
Statistics:
So, percentage of devices praying on dual stack has remained stable. As you can see in this graph. It likely due to the adoption of 108, known as the IPv6 preferred only option, which is supported right now by Apple and android devices. Microsoft as announced plans to support it somewhere in the future and there is a potential for network manager to hopefully implement it down the line.
So, once these systems also fully integrate option 108 we expect that dual stack usage will reduce to only a small fraction of devices.
This is our map of access points on this floor. The green ones do 5, the red ones do 2.4 and the pink ones are actually in the patch room above this floor but they are set to low strength so they don't interfere with the meeting network. Theres a special AP that I want to point out which is AP78, which is near the foyer area here, because this is a snapshot from the device associations per access point during the social on Monday, and which happened in the foyer area here, and the top three access points here are all in the foyer but unsurprisingly AP 78, the yellow line here is the one closest to the open bar, I'll let you draw your own conclusions here.
As usual, when also could have e, we try and ask them to keep track of some statistics on how many expressos they make. This was ‑‑ the first main meeting where we redundant barista setup, it turns out this resulted in a record number of coffee consumed with an average number of 1362 shots of espresso pulled per day. Which is impressive.
Concluding, I would like to give a big shout out to Meetecho, two of them are here. They are with us in Prague, they make all the remote participation run very smoothly. There is the brilliant stenographers again, which have been amazing again this meeting.
.
Then and lastly thank you CESNET for briefing us with the great connectivity here, and our host, cz.nic, especially Daniel who walked us around Prague for days to look at social venues. Thank you.
So, lastly, I would like to encourage every one, please get in touch with us if you have any ideas, feedback, we are always looking for new cool things to implement, experiments to run, please email us or let us know, we can discuss it. I am happy to take any questions, if there are time.
BRIAN NISBET: I'll be putting in a GDPR erasure notice for my MAC address and the AP near the bar for Monday night. I am increasingly worried about the NCC surveillance stage that is building up here.
AUDIENCE SPEAKER: I have one question. What is this cable going to the toilet in your picture? Is this the uplink?
ROB DE MEESTER: Can I not comment on this?
AUDIENCE SPEAKER: Can you go back to the VPN slide on the IPv6 mostly? So, this is a problem that's been on going for a while. I think it's reported, it's nothing new. It's reported for five RIPE meetings now. So, I think it was like there is presence of some of the vendors here has been reported to so I think it's time for ‑‑ like we did a cooking last time for the stick. Can people start reporting this to vendors so everybody that has problems with ‑‑ and also some people that are here from the vendors, maybe some external pressure is needed so a can everybody that has problems open cases on the VPN problems on IPv6 mostly networks.
BRIAN NISBET: Is this higher or lower priority than the zombies?
AUDIENCE SPEAKER: The reason we use the legacy network, the reason I used it, because it's reachable in the lunch area. I didn't see any photo of the lower, and I like it that you gave the word sing calling a new meaning.
AUDIENCE SPEAKER: Thank you for mentioning this. We need to talk because I think we might want to mention this in deployment consideration document for this.
I'm sorry if I missed it, but you put such a nice historical graph, but what is the statistic for this un, for uplink data?
ROB DE MEESTER: We have it, we can share it, but the presentation had too many slides already, so I'll share it afterwards. Thank you.
AUDIENCE SPEAKER:. It would be super cool if like a sanitised version of the IPv6 mostly edge config could be published on the RIPE GitHub page so that people can take a look at it and see how that works, that would be cool.
ROB DE MEESTER: Thank you, we'll consider that.
BRIAN NISBET: I think we continue to all be in awe of what you and the team all put together.
AUDIENCE SPEAKER: Question on the chats "would you consider replacing VM ware with something else given the current affairs with its vendor?"
ROB DE MEESTER: I think we will consider that, yes. Maybe not for next meeting, but we are keeping an eye on the situation. And we are considering that, yes.
BRIAN NISBET: So we do continue to be in awe of what you and the team put together for one week for all of us, so, thank you all to you and all of the team very much.
(Applause)
.
So, thus concludes the PC part of all of this, and we will hand over to the RIPE Chair. Mirjam.
MIRJAM KUHNE: Thanks again to the tech team. It's amazing, we couldn't have the team without good networking of course.
This concludes the RIPE 89. Very well attended meeting. We had over 600 people here online and on site, well 750 in total both online and on site. From 60 countries, which I always find amazing how many people from all over the place are meeting together here for the RIPE meeting week, it's great. Just a few words about the local hubs, that were following us. I saw some of the sessions participate and the one in Romania sent us this picture and they had fun there together. And so if you are interested in setting up something like this and the RIPE NCC happily facilitates this and helps you. There will be a link on the website for RIPE 90 as well.
Now, coming to some changes in community roles. So the first one, I'd like to announce the result of the NRO NC elections, Andrei Robachevsky will the next person on the NRO NC. Big applause.
SPEAKER: Thank you very much for your trust. It really touched me. I think I can express my gratitude enough to people that publically expressed support, Constanze, Constantina. ... thank you and for all of you who voted for me. I appreciate that. Through the week many people talked to me and encouraged me and supported my nomination, so, I think even if I wasn't elected, it was worth the torture that I had creating this selfie video of myself. Now I have this kind of pressure on my shoulders, and I'll do my best not to let you down. Thank you.
(Applause)
MIRJAM KUHNE: Thanks. There will be a lot of work on your plate, on the plate of the other team members on the NRO NC, and also thanks to the other candidates who stood for this important role.
Moving on. We are already heard from the Code of Conduct team again. Here are the faces of that team. It's a very important role and I would like to give again a special shout out to Vesna. Come up here, I have a gift for you.
(Applause)
.
Vesna and I started as trusted contacts at RIPE 77 before we had a Code of Conduct team in place, and then moved over, transitioning to the Code of Conduct team helped the new members there, and I am really a pleased to see some community members on the team, and I'd like to thank Vesna again for all her work. She will obviously continue and specifically help us with the DEI efforts that she is putting into the diversity session and so forth.
Out going, incoming Working Group Chairs. There were a few changes this time. I'd like to thank Bijal, she together with Kurtis has chaired the RIPE NCC Services Working Group from the beginning pretty much, so I'll thank her again next time when I see her, so, I can't give a gift right now. The other Chair who has been running the Working Group pretty much from the beginning was Martin Winter, who kicked of together with Ondrej Filip the Open Source Working Group, was it RIPE 66 I think you said during the Working Group, and you are now stepping down and handing it over to the new team. I have also a gift for you.
(Applause)
Thanks very much. And then we have ‑‑ I have some more names on there and pictures, maybe going it's not techically stepping down, he has been selected to start at the Connect Working Group as a co‑Chair last time and agreed to help the new team in the Routing Working Group, so he will step down from the routing Working Group but will continue in the Connect Working Group from now on. Then there is also some changes in the Cooperation Working Group, ended but they were actually selected back in, so here you will see some of the incoming Chairs, or back, selected back in as Chairs, so Julf and Achilleas will continue in the cooperation working group. Then we have Stefan who will taking over from Bijal in the RIPE NCC Services Working Group. We have Marco who will take over from Martin in the Open Source Working Group and Sebastian Becker who will start in, as the co‑chair in the Routing Working Group. So you'll see some new faces there also in a number of Working Groups moving forward.
(Applause)
.
It's actually great to see more volunteers stepping up for various community functions, I am really pleased to see that. So, here's the whole picture gallery of all the Working Group Chairs that we currently have in place that you will see again next time.
Then, to the next important community committee. It's the Programme Committee. Here are the people who put together the programme for this, the Plenary Sessions for this meeting. Thank you all very much much. I had really good feedback about the Plenary Sessions, so thank you all.
(Applause)
And unfortunately Moin is leaving the PC, his term was up and he was not reelected back in, so Moin where are you? I also have a gift for you.
(Applause)
.
And there is always the next term of course, the next selection, if you want to come back.
That means we have two terms ended, it was Antonio who was reselected back into the PC, and the new person on the PC will be Babak, who will take over from Moin. So, also congratulations to you and thanks for all the other candidates who were standing, wet quite a large group of candidates which is great.
(Applause)
.
Just a quick reminder, please rate the talks, that will help the PC and the Working Group Chairs to plan for the meetings ahead and will also give valuable feedback for the speakers because some of the comments can be shared with the speakers about their presentations as well.
And another QR code here for a survey. It seems to be a theme for this week. Fill in the survey. So we'd like to receive your fact in general, and I already had ‑‑ of course I am happy if you come to me or any of the organisation team to give us feedback in person. I took loads of notes during the week.
Last but not least, a big thanks to the host and the sponsors, and I have something special for the host, please come up to the stage.
(Applause)
We have a plaque here for Ondrej Filip who is the local host who gave us this wonderful meeting here. And also contributed to the dinner last night.
(Applause)
ONDREJ FILIP: Just thank you all for coming and I hope you enjoyed your meeting here in Prague and I hope it's not the last time we will meet in this lovely city. Thank you very much.
.
And we have one more certificate, I believe, for the diamond sponsor. I don't know if anybody is here from AWS. Yes, Clara is running up on stage. We really couldn't do it without the sponsors, I am very pleased to see also returning sponsors for the meetings.
(Applause)
.
And of course a big thanks to all the other sponsors, and last but not least the RIPE NCC team and specifically Kajal, I don't have a gift, I'll give it to her in Amsterdam. This was the first meeting that Kajal was running independently and responsible for everything, and I heard only positive comments at the dinner last night and the venue, she did a fanistic job, so thank you very much.
That brings us to the end of this meeting. And looking forward to the next meeting which we will have in Lisbon in me, and we'll have a short video from the local host who will have us there and who will help us out in Lisbon, I am pleased to welcome Eugenio.
EUGENIO PINTO: Hello every one. Good morning. So, we are concluding the RIPE 89 and I'd like to express my appreciation for the remarkable discussions and collaborations that have taken place over the past few days regarding Address Policy, DNS, Internet of Things, IPv6, cyber security and many others. The RIPE community is renowned for its inclusivity, bringing together diverse voices and perspectives from across the Internet community, and it's my pleasure to announce that the RIPE 90 next me will be hosted in sunny Lisbon. Me is a great month to be here. Lisbon is a vibrant city and the Portuguese people are known to be very welcoming. I'm certain that you'll have a wonderful time. And you also enjoy visiting some of its attractions like the historic monuments of Polang and probably eat some pastel de nata. And experience the vibrant night life. But also, in the surroundings of Lisbon, you can find the beautiful scenery, and the refreshing beaches for the ones that like to go surfing, there are nice spots there.
It's going to be great to have you here, and we look forward to continuing the conversations, sharing your ideas, and building on the progress that the community has made in Prague.
On behalf of .pt, the ccTLD of Portugal, I extend a warm invitation to you all of you to join us in our home. Thank you all. And see you in Lisbon.
(Applause)
.
One thing I was reminded actually, you can go to the RIPE 90 website right now and register if you are keen coming back. Registration is open already as of now.
.
See you all in Lisbon:
