Plenary session
RIPE 89
28 October 2024
4 p.m.
.
ANTONIO PRADO: Good afternoon everyone. Take your seats. We are going to have the second session of the afternoon.
My name is Antonio, I am chairing this session with Doris, who is sitting here. She is going to look at the Meetecho chats for Q&A.
We start with the first presenter, who is Richard Patterson from Sky UK, IPv6‑only with IPV4aas using MAP‑T. The floor is yours.
RICHARD PATTERSON: My name is Richard Patterson and I am a network architect for Sky over in the UK.
About five years ago I spoke at US NOF about how we had just recently finished building and had launched a brand new greenfield fixed line broadband network over in Italy for sky Italia. I covered about how using unbundled G PON in the access network due to EDPN across or metronetworks, and finally the BNG or subscriber termination layer which I touched on the use of MAP‑T for IPv4 address sharing.
That network in Italy has been growing rapidly over the last few years. We have almost, not quite, a hundred percent of all customers over there using MAP‑T now. So MAP‑T has been working well. We have decided to bring that over to the existing UK network.
So, there are things that are different in a brown field deployment compared to a greenfield deployment, which we'll get to in a moment, but a quick recap on what MAP‑T is.
So, mapping of address and port, using translation. It's one of the few IPv4 as a service technologies which is basically a colloquialism for delivering IPv4 connectivity over IPv6 or using IPv6. They all generally also have the ability to do IPv4 address sharing as well.
So, you can group these IPv4 as a service technologies into the ones that are using translation versus encapsulation. Those that are using translation can be further distinguished by ones that are using stateful versus stateless, NAT 64. And then they are using traditional stateful carrier grade NAT for IPv4 address sharing or the newer algorithmically done address plus port technology ‑‑ stateless. They all have their pros and cons, so to help you evaluate which is best for your use case, he can check to the RFC‑9313.
So map itself comes in two flavours, the encapsulation version, which has more per packet overhead because you are whacking and entire v640‑byte header on top of the v4 payload, but the IPv4 header remains intact when it pops out the other side. The translation version, there is fewer bytes of overhead, not zero, because you are swapping the 20 byte v4 header for the 40‑byte IPv6 header, or 48 if you include the v6 fragmentation header. And that translation isn't lossless, so there are attributes and options within a v4 header that just don't exist in a v6 header.
It does, however, leave the Layer4 header exposed for any hashing or inspection you may need to do. And one of the features I think is quite cool is that the translation through the v6 destination is predictable so you could ‑‑ you can address your CDNs and other servers out of that predictable range and you can get the border relay pass where you can serve IPv6 eyeballs from IPv6 content servers.
As mentioned, the UK deployment is a slightly different topology to our Italian deployment. In Italy we went to RFP for everything so when we did the BNG selection we chose something that could do map T border relate function natively on it. In the UK we already have our BNGs, they have this function. We went to a different solution.
Again, it's more of a traditional network compared to Italy. In Italy we got to use EVPNs and routing. In the UK it's more of a traditional network.
The border relays themselves for a two box solution. The absolute majority of traffic is translated by a light speed plus based Cisco ASR 9903, but there is still a small percentage of traffic that needs to be punted off box to an exception handler for special treatment. And this is the catalyst 8500 that we are using for that function.
Now, there is quite a few packet types which could be considered exception traffic depending on your vendor selection. With the light speed plus it can handle most things in hardware natively. The things that get punted off to the catalyst 8500 are packets a that are too big and need fragmentation or packets that are already fragmented and need reassembly for translation.
In the you UK or whole network is built around four super core sites where we have a BGP free peer‑to‑peer layer. All of our border relays are hanging off those P nodes. Depending on the traffic, how the traffic behaves going forward, we haven't decided whether we're going to scale that solution horizontally at those four sites, or whether we distribute them and put more border relays closer to the subscriber, to 32 satellite pops where we also have BNGs. It doesn't make sense to go beyond that because packets need to be routed via the BNG anyway.
So, because MAP‑T is stateless we can do things like Anycast for resilience, so Anycast, all of the IPv6 prefixs from all of the border relays south bound to the BNG and we Anycast all of the rows northbound. So, flows, ingress and egress flows can asymmetric and go by any of these border relays, likewise, because it's stateless and we're using Anycast, we can use ACMP for using load balancing across devices as well. Because there is no need for expensive state signalling, multiple flows to one user, end user can go via multiple BRs at different locations.
This did however raise an issue that we had in our network which was our peering and transit vendor hashes fragments differently. The initial fragment with a Layer4 header can get hashed one way and this example, and then all the subsequent fragments that don't have a layer4 header will get hashed differently and end up on different border relays. This is actually probably the main reason why we haven't got to a hundred percent of MAP‑T footprint in Italy. Not because because we don't Anycast in Italy, but the same hashing algorithm is used across a chassis for all types of hashing, including across LAG bundle members. So, even though we're not using Anycast from different border relays in Italy the fragments may land on different line cards or indeed different as six on the same line card.
With mitigated this problem by going around our UK network and enabling Layer3 only hashing. Hopefully at some point we'll go to Italy and retro fit that and be able to turn off dual stack and DHCP V4.
Now, the Anycast doesn't just follow the IGP shortest path. We steer traffic to the closest or the most optimal border relay. Subscribers that terminate in the super core BNG, will use those border relays as primary. If that fails, then flows will fail back across to the remaining three sites evenly.
Around for subscribers that terminate in those satellite POPs, further down at the network, they tend to use the two, the prefer the two border relays at the parent super core sites that that satellite POP parents to. If we do decide to go distributed, these one will pick up as primary and only if that one fails will the two parent super core POPs pick that up.
Just to help visualise that. A subscriber terminated in London would prefer Telehouse London unless in would fail, flows go to the other three sites. Subscriber would go to satellite POP, would prefer both this if they were both active. One has failed so the other one picks up the Slack.
And if we go distributed, the distributed one is primary unless is would fail.
So, the stateless or the MAP‑T being stateless is great, because it allows us to use cheaper more scalable hardware, it allows us to use. But to get that statelessness, the complexity has been removed from the forwarding plane and it's had to go not in the control plane but into the administrative plane. So, unlike traditional carrier grade NAT methods which will dynamically and efficiently allocate ports as required, in map, map gets a statelessness by using a predetermined algorithm plus a configurable map rules to BIND the users v6 prefix to a v4 address and a port set ID.
Now that binding is fixed. So it's a fixed sharing ratio.
In Italy we use a 16:1 sharing ratio. In the UK we have gone for an 8: 1. Not for any technical reasons. Could you push this much higher than even 16:1, if your local regulations allow it.
So, the 8:1 sharing ratio profile is a default which we expect the majority to be in. But we also have a 1: 1 profile which we still using MAP‑T to deliver IPv4 as a service instead of falling back to dual stack, but that customer still gets the entire v4 address to use for themselves.
We have built two processors so that the customer can opt out of address sharing to the one to one profile. We put a lot of effort into developing an automated or proactive opt out approach. This is where we listen to notifications that are being sent by the CPE, or the my sky app. Whenever a user to go else or does port forwarding, we will apply the one to one profile, we'll have to disconnect the customer because they need to reconnect to obtain a new DHCP lease with a different set of MAP‑T rules.
And then also a reactive journey as well for humans to manually be able to do it.
We use port base authentication for all of our subscribers. We don't pay attention to any user names or passwords or anything. We look for information that is inserted by the access node into the DHCP messaging, specifically the remote ID which is DHCP v6 option v7 and option 82.2.
In dual stack land, either v4 or v6 could trigger authentication but in MAP‑T we rely on DHCP 6 so we should only be seeing v6, DHCP 6 for this.
In Italy, greenfield network deployment, we had no legacy technical debt. We knew that all CPEs, they were E tending to customers, support MAP‑T so we could forcibly use MAP‑T, or force MAP‑T on them. In the UK, we have already got an existing over 6 million customers. We have to be a bit more graceful and opportunistic with our enablement process. So, we rely on a CPE that supports MAP‑T, asking for MAP‑T, and then we'll give it a DHCP 6 lease with MAP‑T options. It asks for MAP‑T by including the standard MAP‑T option, option 95, in the DHCP 6 option request option, in the DHCP 6.
However, only about 70% of our wholesalers access network actually supports lightweight DHCP relay agent or LDRA which is required to insert that information into the DHCP messaging. So, we have about 30% of these customers we have to rely on DHCP 4 to boot‑strap authentication, and then we can go through v6 and give them MAP‑T.
We obviously don't want to warn any more public IPv4 addresses than we have to, so for these boot straps v4 sessions we give them a private address, which is not usable.
So, we were relying on DHCP option 95 before to tell us that CPE supports MAP‑T. Option 95 doesn't exist in DHCP v4, so we had to get creative. We already use option 60, or a class ID for a bunch of other things so we append add MAP‑T string on the end of that so that we could signal inside the DHCP 4, hey this CPE itself supports MAP‑T.
We also saw during our trials that there are CPE out that there that don't support MAP‑T but still ask for MAP‑T by including option 95. Open WRT is the main culprit for this. So whilst it's greedy by default. It includes a whole bunch of options. So they will get a v4 lease that's broken, and they will also get v6 lease and they'll have working v6 but will expect them to use MAP‑T, which they won't. The good thing about open WRT is that it does support MAP‑T, but it's not installed by default. So you have to install the map package to get MAP‑T working. That's an easy fix. The long term fix would be to change the default behaviour of open WRTs DHCP 6 clients.
Ubiquitous unified routers also use these. But they don't have a map package, they don't support MAP‑T. So, the only real user fix that they can do, or users can do there, is to disable v6 to force DHCP 4 authentication. The alternative is they can escalate sufficiently through it and be able to disable MAP‑T on the network side.
So, again to help visualise this logic, this authentication language, either DHCP or v4 can trigger a request. The first thing is it checks to see whether we have finished enabling configure the BNGs to know about MAP‑T. We have done that now. It supports MAP‑T. The next thing it does is check for the presence of that option 95. If that DHCP 6 option 95 doesn' exist, it then looks and does a match for that string in DHCP v4 option 60. If either of those exist, then it checks Tor the disabled profile followed finally by should they be opted out to one to one or use the default 8:1.
CDN steering is challenging in this IPv4 as a service world, or indeed even a traditional carrier grade NAT world where your IPv4 topology is different from your IPv6 topology.
So, we have got CDN caches deep into our network. Happy path for IPv6 eyeballs is that we steer them directly to their closest CDN cache. The unhappy part then for IPv4 eyeballs is that we still use that same cache but that gets tromboned back to the centralised sites before coming back down again to the subscriber.
Some CDN providers use the source address of a DNS query for the steering decisions. EDNS it's usually the recursive name servers which isn't helpful. These come along and that embeds the original source IP of the query into the payload so they can make more informed decisions. However, this doesn't really help us in the situation. So, what the authoritative name servers need to do now is make decisions not just on the ECS value but also the query type. Is the query type for an A record for v4 or is a query type a AAAA query for v6, as well as the ECS value? Until such time as the CDN and authoritative name servers do that and catch up we are having to do the logic for them. So we have six Anycast sites for the platform. We are only giving these MAP‑T customers the v6 for recursive DNS. The DNS queries are always going to come over IPv6 to our recursive name platform. It does that logic I just mentioned. It checks to see if the query type if it's for an AAAA it will insert the v6 ECS value as normal. But if it's an A record query, then it will do a look‑up internally, it will try and find out the MAP‑T IPv4 address that have customer, and it will insert, synthesise their IPv4 ECS value before falling it up.
So, again having path, IPv6 eyeballs can go to the direct cache, IPv6 eyeballs can be steered towards the centralised CDN cache.
We now have a vested interest in making as much of our content as possible as IPv6. So both of our flagship video products Sky Glass and Sky Stream are supporting IPv6 since QS, 031, which is already live.
I mentioned that the this is moving from the forwarding plane to the administrative plane and the complicationings of that. Now your IPv4 consumption is directly tied to the IPv6 consumption are or DHCP 6 pulls. So previously we had automation for our DHCP 4 pull management. We had really granular DHCP 4 pulls to get efficiency out of them. That required automation. But then in v6 we through a massive big IPv6 prefix in the DHCP 6 pool, enough to satisfy the wholesale of the BNG, we didn't have to worry about automation. If we were to do that with MAP‑T, then that large IPv6 prefix would consume a whole bunch of IPv4 that we don't necessarily want to waste.
So, automation is important for MAP‑T these days.
One of the complaints I hear from other CPEs who are interested in MAP‑T is that there is a lack of MAP‑T support on CPEs. So, we're lucky enough to build an in‑house device using OpenSource IDKB, but open WRT, despite the issue I mentioned before, is also a very great solution. As I mentioned, the map package is installed by default, and it's also a recent bug that's been introduced by the migration from net filter, which I'm sure they'll fix at some point.
Keenethic, they reached out following our launch in Italy and they have their own support on their devices and the Fritz box, these have been hard at work to build their own support and Fritz, which is awesome. It's in the lab version now, if you go into the settings and tick the box it will work for you apparently TP Link and Zikso also have support.
That's all the main content. I wanted to end on this last little bonus slide, which is the unexpected DDoS protection. So, packets that come in that are malformed will just hit our border relays, not be translated and drop on the floor. So bonus DDoS protection.
That's it. A couple of minutes for Q&A.
ANTONIO PRADO: Any questions for Richard?
AUDIENCE SPEAKER: Lee Howard. It won't surprise you, it will surprise that you I have a question. You said you were detecting UPNP packets and kicking them out at profile. If that's detected because you assume that port forwarding. Are you detect that go on the CPE or the BNG or the BR?
RICHARD PATTERSON: It's out of band signalling. We are not using the UPNP natively, we are doing out of bound signalling. I was ‑‑ at one point we looked at whether we would just sort of allow it for ports, because 8: 1, there is a good number of ports that a customer does have, should we just give them a port that they with need ‑‑ if they ask for a port that we have, should we just give it to them? But, previous test results from years ago, it looks like that applications, Xbox for example are stubborn, if you give they 3074 and just don't try any other ports. So we kind of ‑‑ we threw that idea away and went if they turn on UDMP full stop we'll just opt them out.
SPEAKER: You also said ‑‑ no, I forgot the other question.
AUDIENCE SPEAKER: David Lamparter, there is a whole bunch of ways you can configure the address sharing, like, size of the pool but also address continues or not. Did any specific variations cautious with specific CPEs or did that turn out to be reliable?
RICHARD PATTERSON: So I think the ones, or the recommendation, I am the not sure fits a recommendation of the RFC or just people saying it is, is to use a 6 bit offset to you avoid all of the privileged ports for ‑‑ you don't want to source port from TCP 80. So we did that. If we didn't, would it have been a problem? I don't know. Assuming you do say that now, there is an about you go that happened last week where we're not calculating the piece of value correctly we're missing a few bits. Because we have a fixed range, we're not hit that go problem. So we'll get the bug fixed but we're not hitting.
AUDIENCE SPEAKER: But no CPEs erupted on flames in specific combinations
RICHARD PATTERSON: To be honest, like, it wasn't sort of let's test all possible combinations and work backwards. We went how we want to design our map rules. We knew the sharing ratio we wanted.
COSTAS: Hello. You have a tonne of stuff in your presentation. You have done amazing work. In a previous lifetime we had implemented something similar with a competing standard called lightweight 4 over 6. My question is you mentioned about DHCP. So, you are using the DHCP functionality on the BNGs or you have an external box that handles DHCP?
RICHARD PATTERSON: We use a local BGP v66 server on the BNG. I mentioned before that it doesn't know about MAP‑T. It doesn't. But our BNG vendor in the UK it's very powerful. There is a very powerful vendor out there that does it and we can though in HEX codes into t our automation platform will just basically shape this option, option 95 here and the hit encoded map rule as a string.
AUDIENCE SPEAKER: The DHCP process is in the BNG?
RICHARD PATTERSON: Correct.
AUDIENCE SPEAKER: Would you care to share your BGP vendor?
RICHARD PATTERSON: I guess. It's Nokia CSN 50. It's by far one of the powerfullest BNG. It's expensive, but great. I shouldn't say that...
DORIS HAUSER: So, by now we have two questions on the Meetecho but I think we only have time for one.
One question is from Dmitry Serbalov. He asks: Do we have the same OpenSource MAP‑T software for Linux?
RICHARD PATTERSON: For the border relay? Yeah, theres a few. There is at least JOL ‑‑ I think it's still perhaps in a branch, it's not in the mainline release. It seems to be pretty ‑‑ it works well. There are conversations on do we home grow our own solution in do we use an OpenSource product? I think Sander did you have one at one point? No.
ANTONIO PRADO: Okay. Thank you Richard.
(Applause)
Okay, now, for the IETF slot, we have Benoit who is talking about next era of network management operations at the IETF.
BENOIT CLAISE: I am not alone, we have got Maria here as well. So, what I want to do this session is introduce a new network management workshop organised by the IETF. It's called the next era of network management operations. So, my name is Benoit Claise and I am a member of the Programme Committee.
Okay. So if you have to pay attention to one slide only, this is this one. So, next era of network management operations, NEMOPS organised by the IETF. So, who knows about the IETF? Who is ‑‑ who is working with the IETF? Who knows about the IAB? Oh, pretty good. So, I went to LACNIC a couple of weeks ago and the answers from slightly different.
So, this workshop is a virtual workshop for three days, three hours per day, so not a full day. You are invited. So, how to participate. Either you have a submission paper but you could also fill in the survey which is, for which we have got the QR code there and helping us, so Mirja, IAB member, and I, we are here for the entire week. For any questions you might have on the IETF, on the workshop, on what we're doing. So, the ideal plan is that we speak to a lot of operators this week.
So, we have been having this habit to work with the NOGs, right, in the IETF. Why? Well to listen to the concerns, specifically whenever they develop the protocols, the architecture, the data models. To explore technologies, but also to listen to you whenever you operators try to deploy it and you say it doesn't work as easily as it should be working. So basically it's to bridge the gap between the IETF and the developments and the real world deployment. And we want to understand your views.
In the IETF, I believe that maybe there is bias towards a vendor. We don't have enough operators, right, and that's a concern. And we always be listening to the operators. I mean, we say that in the IETF everybody is individual or equal, but if you refer to the book of George Orwell, Animal Farm, we are all equal but some of us are more equal than the others, right. So, when I was an area director I would listen more carefully to what an operator would be telling compared to what a vendor would be telling.
So, to address this, the IETF took two different initiatives. The first one is to create a new Working Group called operator network ‑‑ sorry, network management operations. So, I am going to mention this later on, it's based on experiments, and maybe we have forgotten in the IETF the motto is: Running code. So rough consensus and running codes. The second thing is that we created this workshop, and we are going to publish the result of the survey, we are reaching out to the operators. We heard it too many times like okay, come and work with us. Actually, it's an always work with busy operators, that's why part of this Programme Committee, we decided to reach out. So as I mention, I went to LACNIC a couple of weeks ago, and discussed with many operators, some of our colleagues went to NANOG last week and we are here this week. But not only for this just half an hour slot but to discuss with many of you.
And we are going to submit the survey results as a paper and we will have the workshop.
So, let me show you what I believe is a good use case. Maybe three, four years ago we had several operators who get together and said: You know, it's a little bit difficult to stream all these YANG data the way I want, and actually it's streaming from a router to a collector through Kafka until a time series database and I want to seep the semantic and I want to do that very easily, exactly would like when I use IPFIX, it's easy, I stream, I get the IPFix information elements or NetFlow, and they arrive directly and my Analytics could be using information. When I am trying to deploy this simple use case we realise add couple of things that we need to change this telemetry. First of all it should be able to run on UDP, it should be faster like IPFix NetFlow. It should be able to stream directly from the line current and not go from the route processor, where there is a bottleneck. It should be about to send metadata with information because in the tool chain, we need to send when it was observed, because if you extract from the load, you are losing information. It's not like a syslog message with where you say the time is the time I received the syslog message. Well, no. And we want to be able to import the database. So that's a good example of what we have been doing with operators and experiment.
And for now we have got up to six drafts a that the in the NETCONF Working Group just for that, but more importantly, it's done with hackathon for maybe five or six hackathons.
So, a lot of you know about the IETF, great. So, these are the results in the world.
Some issues are easy to resolve. You can read the first three ones. The one that are harder is that don't have enough time. Which is a pretty fair comment if you are an operator or if you are dealing with a real network, right. The one that concerns us a lot is the last one: Don't feel my operator input is welcome. If it's your case, I would like to know more about this. We would like to know more about this.
Now, we have got a chance, we have got Mirja here, she was the IAB Chair for four years, still there, and when I put to LACNIC the distinction between IETF, IAB, IRTF was not clear. So here if you could explain to us a little bit what is the role of the IAB and how it fits in there.
SPEAKER: Yes, IAB is the Internet Architecture Board. There is no Internet architecture so the question what we're doing. Besides some administrative stuff we are doing what we call architecture oversight. So what that means is we are looking in the IETF but also outside of the IETF about things that are missing. What is missing in the discussion, what's missing in the technical work we're doing? Where can we help to facilitate people to come together and talk about something? That's often the technical work we're doing. It's looking at the kind of high level picture, looking at the gaps, trying to reach out, trying to organise people talking to each other and starting discussion. One the tools we're using for this are workshops. This is exactly that. We want to reach out. We want to have input from other people that we usually see in the IETF and we want to start a discussion and workshop is never the end of a discussion it's always the start. It's trying to figure out what are the merchant technologies, what are the problems we need to solve, what are the things we need to do next and bring them into the IETF into the discussion and then bring them into the technical work standardisation.
Maybe one more note. Workshops are one of the tools and we're doing this right now to start the discussion, but there are also other tools like programmes, if you need a more lasting discussion for example, so we are looking at this and that's the role why the IAB is involved because we are trying to facilitate this discussion.
BENOIT CLAISE: You want to say anything else about the difference between ‑‑
SPEAKER: I can also explain this picture a little bit. There is actually two leadership groups. One is the IAB doing this architecture oversight. The other is the IESG that's the area directors that care about the management of the IETF. So you have all this different areas doing the technical work. And maybe the IRTF is interesting, which is operating research groups and brings in the research community. And that has been a very valuable exchange, having people from the research community bringing input. Some as valuable as the operators input we need.
BENOIT CLAISE: Very good. Thank you.
So this workshop is about the ops spot, which is at the same time operations and management, or the management is about protocols of the model and the operation is about how to deploy.
So, it's not the first workshop done on network management. Around 2000 ‑‑ I know it shows my age ‑‑ but there was like a workshop about how does network management work? And this workshop in 2002, and by the way we went to ‑‑ not we, I was not there, RIPE to NANOG etc., and what happened is that people were using SNMP but it was never used for configuration and people were asking why. So, out of this workshop came up a very interesting RFC, if you deal with network automation, read that RFC and specifically the section about operator requirements. I will read a couple of them and this section is 14 entries, very easy.
It says "If test necessary to make a clear distinction between configuration of data, data that describes operational state and statistics.
4. It is necessary to enable operators to concentrate on configuration of the network as a whole rather than individual devices."
So, requirements on the left. And recommendations on the right. If I read this. Just one of them, right, but if you have got ten minutes, from time to time, it's a good refresh.
"The work shop recommends, with strong consensus from both protocol developers and operators, that the IETF focus resources on the standardisation of configuration management mechanisms."
.
So, that leads to what? To the creation of the NETCONF Working Group, that created the NETCONF protocol and the the others. That created the YANG as a data language. That created YANG as network element, a service YANG, etc., etc. The entire shift towards data model managements were when we have a data model, we can generate an API. So we could start programming the network which was not the case with SNMP because ICMP it could the nobody used for configure. It could use a good job for monitoring.
So, I want to show a couple of initiatives that are happening in the IETF. So, NETCONF Working Group, I mentioned that. We're working on NETCONF. Next. RESTCONF. Next. The net model is the one that works on the YANG as a data modelling language. We speak about versioning things like that. The NMOP, network management operation, is the new one I mention with three experiments. Remember, the YANG push integration into ICAV can time series. This is becoming a group item. The thing that I like about this Working Group is that it's only one of when we finished the experiments that we are going to publish this as RFCs. So it might be taking maybe a year or two, it doesn't matter, we have to go back to rough consensus on running code. The second one is about animal detection and incident management, right. So when I speak with operators and I ask how many people do you have doing AI and ML? The average is between one and two. Maybe one.
So, we know ML will have an important input into operations. But what we see is that if all the ML guy in each operator is trying to look at the same mechanisms, we're not going to get anywhere. So there is a mechanism on how we're going to ‑‑ and I start to see happening where operators start so their their incidents with others when they are NDA. The next step will be they will share the incident data because they are thinking it took me like seven minutes to find it. Maybe if you put your ML guy on it, it takes you like three minutes, with the confidence level of 50%. So, we will need to compare those.
And the third excerpt is about map modelling and how we want to have the multilayer topology view of, you know, physical and routing and service and application, but all connected with each other. And that's a link with the last entry which is called IVY, that stands for inventory, where we want to have the ability to model the hardware, the software, the licensing and all of these things.
And I want to spend one specific slide on green. Getting ready for energy efficient network. So maybe like 150 years ago there was a working group called energy management that was dealing with MEP modules. We know that ‑‑ SNPP is not there to configuring newcomers, whatever reason, it didn't take off. Now we are telling can we redo this? Explore the use cases in the requirements. Provide solutions.
Now, green, it's like very broad. So, if you are an operator and you have got requirements in there, we are actually looking at all different use cases. Some of them are defined in the Charter, and the IETF is in next week in Dublin.
So ‑‑ and it's not going to be membership obviously it's going to be about YANG. Why? Because it's a energy efficient it's about some control there, some configuration, not MIP, YANG in this case.
SPEAKER: Maybe I can add to this one. This came also out of an IAB workshop. And then we also created a programme around it to focus the discussion on the points what's the work that can be done in the IETF. What are the parts that we can help to address a much bigger problem. Part of this is that this Working Group which just got chartered and will meet the first time next week in Dublin is focusing first of all on metrics and frameworks, but then we will also have a site meeting which is called Sustain Energy, so we will also trying to chart ear research group in the IRTF to actually connect this to all the research questions that are out there for the bigger picture.
BENOIT CLAISE: That's a very good point that reinforces the fact that the programmes as you said it's the beginning of something, that led in this case to a new Working Group, that led to potentially something in the RTF, so we are hoping a lot about this new workshop about network management.
Remember, NEMOPS. 22 years later, it's about time maybe right? We might be telling that we are slow in the IETF, I will accept all criticism but regardless, we should do it.
So, what are the objectives? I will have maybe a few minutes towards the end. So feel tree to prepare whatever feedback you want. Roses, tomatoes, whatever, I am fine. Right.
So the first objective is to review the outcome of the previous workshop. It took us 22 years, right. But we still see a lot of SNMP, we still see people configuring the CLI with some tools, so, it means that somehow we want to understand what are the operational barrier that prevent you to deploy what we're doing in the IETF. And maybe you could say this is crap. Fine, I don't have the right tool. I work with OpenSource, there is no tools. Whatever. Why is this not deployed?
.
Now we want to understand the new requirements is maybe because, you know, I have like requirements that all vendors must implement that series of YANG modules and series could be IETF, open configure, you name it, right. So unless it is done, it's not going to work. So all these new requirements that you have, we want to understand. And then exactly like we shared the RFC‑4545 requirements: Recommendation. We would like to have the same thing. And yeah it's going to take some time but this is something that we have to do.
So, how to participate?
.
Well, the best is that you have a position paper, and that's the way to be invited to the workshop, right. It doesn't have to be long. If it's half a page, this is fine. If it's an intention to write a paper, send it already. The deadline is November 17th. Send it sooner if you want. So, now writing a paper takes some time. What you could be doing is, you discuss with us here, and we collect information from LACNIC, NANOG, here, RIPE and we're going to try to have an overview of all the feedback that we heard.
The next thing it wouldn't cost too much like fill in the survey over there. And don't forget that we are here, Mirja and I during the entire week for any questions. What I would dream to have is that we fill in the survey together I don't want to influence you but I want to say look we have got this in the IETF, do you know about that? Oh, yeah, and this person, what do you mean inventory of this, inventory of that, of that, or etc.? Just to get more feedback.
SPEAKER: Maybe one more word here, this is an invitation only workshop. That's not no exclude anybody. It's because the workshops we run are very interactive. We want to workshop where everybody in the room can contribute and participate in the discussion. And in order to organise the workshop to understand what we want to discuss, we need your input. So this is just a mechanism for you to tell us what we need to know to organise the workshop and if you tell us something that isn't scope and interest for the workshop we will invite you. That's how it works
BENOIT CLAISE: Very good. So, at this point in time, we're ready for any feedback that you might want to have. What I have as well, while we're giving this, these are examples of questions that you would have in the survey, think about those, and I could go through some of them if you want. But we have got some feedback already.
AUDIENCE SPEAKER: Lee Howard from... thank you for doing this. I wish I had done this when I was... Good job. Can you give examples since most of people in this room have not seen or read a position paper, what kinds of things would you look for in a position paper to help influence the direction of the workshop?
BENOIT CLAISE: I'll start and you will complement this. What I have seen, because we started to receive position papers and there are things like: Okay, the IETF is not efficient in that area. And I cannot apply what the technology that you provide the IETF because A, B, C and D. As a consequence, I believe the IETF has to work differently and this is the way that I would propose them.
So this is a typical thing that we receive in a position paper. And some of them are just two pages. Some of them are longer. So, I gave you the perfect example because exactly what we wanted to have, but actually any feedback from the operator is welcome. So there is not like a well structured format of, you know, you have to answer those questions.
SPEAKER: Also one thing to understand that like this is not a workshop where every paper is presented. Sometimes we have extra more research papers where the there is bigger analysis and sometimes we pick those papers and sometimes we ask them to present them because it provides the needed background knowledge for everybody. But everybody else in the workshop is just there to participate in the discussion. So what we really needed or what's your contribution to the discussion, and particularly if you have new requirements or barriers or some experience you want to share, that's really valuable for the discussion. So you if just write this down in a short position paper, that's what we want. That's totally enough.
BENOIT CLAISE: This is background slides for you to read.
ANTONIO PRADO: Questions?
AUDIENCE SPEAKER: Maria, CZ NIC developer of BIRD, I'd like to ask a question to all those things throwing YANG there, throwing YANG anywhere else, is the purpose of the Green Working Group actually also to evaluate whether the YANG approach is actually working. I am not saying that it's not working, but I'd like to see some assessment at least in like how much time it takes to actually implement all those things, or whether it actually managed to alleviate some of the problems the operators are having now. Thank you.
BENOIT CLAISE: Right. So it's true there is a lot about YANG in there based on the feedback from the operators in 2002. So what you will be seeing there, it's like data consistency to support richer availability, data and knowledge. What we see is if you have got something like BNP, to take your world, there is no way someone is going so an I am going to stream BNP with YANG, BGP works, so what we're trying to see is there are different different silo of information and not everything is YANG, right. So that's the first thing that we see.
Now to answer your question whether YANG is like the best solution. On the charter, it clearly says YANG, so do you have a better approach? Do you have a proposal for now? A better proposal than YANG for configuration?
SPEAKER: For the Green Working Group, the focus is on understanding energy consumption of your network, these metrics and having a framework for it and because YANG is there, I think that's kind of where the solution probably goes. But I think all these problems you have with implementing YANG, that's actually what you want to bring to the workshop.
ANTONIO PRADO: Let's discuss later.
JIM REID: I think this is a great idea and I'm very pleased to see that the IETF and the IAB is making more of an effort to engage with operator communities. That's long long overdue. And if it means more participation for those operator groups in the IETF, it that can only be a good thing. So this is a very good step in the right direction.
I have got a little bit concern about the discussion around this workshop. There seems to be a lot of discussion around tooling particularly and I wonder if there is more focus on things like the problem statements, because my sense is that perhaps in the IETF, there is a little bit of an ivory tower around this. We're looking at the solutions from the people that develop, that go to the IETF meetings and talk about things like YANG and the all of rest of it, but these people are somewhat removed from the cold hard reality of running a big global network or any sizable network, and I wonder if maybe that might desuade people from those communities from actually taking part.
BENOIT CLAISE: That's a very good point. I have been receiving feedback from operators that the IETF is not like solution driven, which is that yeah, you do YANG model there, you do a small IPFix, it you do a small part there, about you if you are not solution driven, it means that those will not evolve as a cluster. So I have been hearing that, it's not the first time I heard this. So feel free to write a second position paper on this.
JIM REID: Thank you.
ANTONIO PRADO: Last question.
AUDIENCE SPEAKER: Hello, Alex Yi. I would like ‑‑ I currently have tried to implement YANG on a network vendor on the new setup from a packet and I just was not able to just be able to have a co‑Python object to be able to to manage a configuration. The one point I have seen with YANG model, it's pretty verbose, it's complicated to isolate what is the configuration of what is the QN state of the model, and when you compare with many vendors, you can see, you can manage to generate the configuration of network box, push it and let the equipment manage the corresponding deconfiguration and if the other end you have a very verbose configuration with where you need to make a strong work to identify from each vendor what configuration is what, and when you have simple thing that identifies a complex API that's why in my opinion operators don't try to implement YANG, or only when they are a large scale operator.
BENOIT CLAISE: Very good. So do you want the long answer now or we take a beer and we go over all of this?
ANTONIO PRADO: The second one. Doris, anything from remote? Okay, thank you.
(Applause)
.
Now for the lightning talks, we have ten minutes with Andrew about route servers.
ANDREW McCONACHIE: Hi. I work for ICANN. One the great things I get to do there is work with the DNS route server operators, and this talk is about a website I maintain, some statistics on it.
What is rssac002. That stands for ICANN root server system advisory committee. They have a series of documents. 002 which is on version 5. It's basically a bunch of ‑‑ it's a specification for a bunch of counters that root server operators collect. You can see them listed there. And they roll over everyday, and ‑‑ what's good with a bunch of counters and data if you can't make pretty graphs so I thought well let's make some pretty charts. And I started in 2021, that was v1. I updated it again this year in 2024. And the major update this year was I added some data from RSS instances, so now you can see some pretty maps with root servers on them and what not.
Important note, this is really just historical macro trend analysis. This is not an operational thing. There is a three week delay on the data. So this is not an operational tool. But I do have data going back to 2017. So it does make it kind of cool to see what's been going on in the last seven years. All the visualisations are interactive on the site. I'll be showing some examples but you are free to go play around with them as you wish. It's OpenSource. If you do have feedback, I'd really appreciate it if you open an issue on GitHub, that's ‑‑ you can mail me too but better to open an issue.
And a big thanks to all those lovely people at the bottom who provided all the feedback.
So here is the first chart. I am going to go going through these pretty quickly just to give you a taste of what's up there. There is a lot more up there than what I'll be showing and what not.
Each one has a little story with it. So this is kind of interesting. You can see the traffic peaked to the RSS in late 2021. That's because Google Chrome used to send these long kind of gibberish domains into the DNS and then to check to see if it was on a public network. And they fixed that. And so since then you see this massive spike where it just goes down after they fixed that.
And now you see it's kind of gone up back a little bit. That was a big spike back in the late 2020.
And you can see that reflected as well in kind of the ratio between no error and NX domains. So, you see that on the ‑‑ that's your left. You see also like at the end of 2020, like suddenly there is more no error and fewer NX domain as a percentage of total queries. And then you have also got a heavy pie‑chart showing the same thing. They are almost 50:50 actually. The pie‑chart is pretty steady. It's been like that since 2021.
This might interest people in this room. Just showing the increase, the kind of steady increase but levelling off of IPv6 queries to the route. This is again a percentage basis. So you see it, you know, 2017 it's what it's like 15 and then it kind of, it's levelled off at kind of low 20s, it hasn't really moved.
This is a fun one. So you may be wondering what happened in 2022? So the story here is there was some ISP in North America that shipped a CPE that was doing v6 priming queries, every time it rebooted but they were only doing them over v6. So the RSS got a whole lot of queries over v6. Hence this massive mountain in 2022 that just ended in the end of 2022.
This one is probably the most complicated I'll be showing. Basically, I think the message here is that like, packets to the root server system have gotten larger over the years. That top line, that purple line is the 32 to 47 byte packet size query. These are all over UDP. So you can just see it's been going down and the ones that have been going up have been bigger packet sizes. So, again like, kind of interesting to look at. Really probably not all interesting operationallyse but kind of neat.
This is what I can do now with the new dataset which I could just show the number of instances over time. I can also show like IPv6 available instances. Or dual stack enabled instances. There is a few kind of staggering like IPv4 only instances out there, but there are no IPv6‑only instances out there. That's pretty stead. It just continues to grow every year in the number of instances out there.
This is a pretty map. You can zoom in on the the pretty map and find more detailed pretty maps with, you know, more detail on each of the instances and find the one closest to you if you are interested in that and whether it supports v6 and v4 and all that and you can also do it by letter.
.
That's it. And I think I have three minutes left for questions.
ANTONIO PRADO: Any questions for Andrew?
.
Okay, no. Thanks Andrew.
Next presenter is Mikhail. Ten minutes with Mikhail. Throttling YouTube in Russia.
MICKHAIL KILMAREV: Hello. First, I would like to take a selfie with you. I am the founder of the VPN generator. I hope you can survive bacause I will test your patience for the next ten minutes.
Okay, in Russia, I have also about labelled for an agent, have several criminal cases against me. I have lost count but it's more than two, and in August, they even added me to the terrorist list. So sometimes I have to explain in the airport that it's all political motivated from Russia. But now, I have a selfie with all of you. Do you remember what I did? And I am so, sorry for my terrible English.
I would like to show you the timeline. First, it's clear that the Russian government was preparing for war and military censorship. Over the past decade we are passed for Internet blockage, the most important host was low 90‑FZ on the Internet which required every Russian operator to install DPI on main lines. This installation was founded by the State budget, and completed buy the end of 2022. After the war began, independent media and social network were blocked. Next, popular VPN dividers started to be blocked by connection points and gateways. In August of last year VPN blocking by a protocol started as WireGuard began working very poorly, and finally, in August this year, the slow down throttling of YouTube began.
This traffic decline YouTube in Russian, that YouTube traffic in Russia has dropped this data from Google transparent report link, I forget the link.
The decline in traffic became noticeable in August 2nd. All reports and user compliance started appearing two weeks earlier.
I simply averaged Google's data to make it cleaner for the presentation. It turned out to be around 30%, which matched the official statement. I decided not to show how does the throttling work. There is only one answer: DPI.
But, I would like to show what pro link actually works like. Theories open data on two popular YouTube channels in Russia, one is political and the other is purely for entertainment. We can see that views of these channels have not decreased. In some cases, they have increased. Why?
.
I checked the interest Russians have VPN technology. To do this I use Google trends and refresh yen counter part, Yandex word stat, when there was something important blocked in Russia, the number of VPN searches goes up several times. Here is, here we see a big increase in search in March 2022. And August 2024.
Compared to the MSK usage, the rise of VPN usage in Russia much of the data shown above, I checked, I checked traffic statistic on the MSK‑IX traffic chart page, and indeed, despite vacation season, traffic increased by around 20%, compared to August last year. I couldn't get the data of MSK‑IX, this is a slightly outdated chart.
This updated data, August '23, is missing, but we can see that the traffic growth has not stopped. In October, traffic exceeded the new by an additional 20%. This is currently the highest record growth which can only be explained by user massively switching to VPNs, they bypass global Google cache servers and connected directly overseas.
Conclusion:
You can read this conclusion on the screen.
VPN usage growth model. Based on the above, I created a model for VPN usage growth, some Russians, we cannot measure exactly how many people use VPNs, service is ineffective in war time, as people are afraid to answer truthfully. Technical measurement methods, we are also unusable as protocol, are no use that hide from the DPI blocking. Remember, that the main protocols, WireGuard, IP Sec and open DPN are blocked in Russia. Currently the most popular VPN protocol are shadow socks, and verification of Chinese solution based on V less or reality, according to this model, we can reasonably assume that about half of Russia is least know how to use a VPN. This makes the censorship ineffective. Quite the paradox.
How ‑‑ I have listed in the, all the challenges faced by telecom operators in Russia. I don't mean to excuse them of everything is going wrong, I'll give you some time to read. But do you know what the Russian government did to help the operation? ‑‑ operators? What did they? Right. They banned any increase in tariffs. That's called absolutely, absolutely stum maybe.
And I would like ‑‑ I can meet the opportunity to show statistic from our project, which also demonstrates growth. We have tripled in size over the past three months. And yes, we are looking for investors. Send me an e‑mail.
This slide.
We use a multiprotocol approach where protocol complexity increases with each escalation of blocking measures. And so shortly, briefly explain how it works.
Additional products. For greater clarity, we position our products as a middle ground between VPN provider and self hosted VPN. We have simply created a ready to use self hosted VPN and learning how to manage them. And a large scale.
What we need right now: IP addresses, maybe you are too lazy to pull from Iraq, we can buy. Outdated system and just fine. We need more ...inaudible to find it harder to block us?
In conclusion. I want to show what the community is already doing to reduce censorship worldwide, especially with the launch of ECH. Do you know about ECH on CloudFlare and ‑‑ also the publication of RFC 9620 deserves recognition. A big thanks to the author of this document. Please take a look at it.
And I am sorry, I won't be able to answer your questions right now. Here are my contact details if you want to reach out, the QR code to link this presentation. Thank you.
ANTONIO PRADO: Thank you Mikhail.
(Applause)
So, any questions to the e‑mail to the contact of Mikhail.
Last lightning talk for this afternoon on stage, August, for the Cyber Resilience Act and OpenSource projects.
AUGUST BOURNIQUE: Hi. I am going to talk about the Cyber Resiliency Act. So let's see how this works.
So, obviously I am not particularly technical. I am an attorney who is California licensed, working out of Amsterdam so we have to start with a disclaimer and that is this presentation is general information. I am not giving you legal advice, and I'm not your lawyer because I don't know what your specific situation is.
So, the Cyber Resiliency Act. It's an EU wide digital products regulation. That is brand new. And it has some friends and I'm not going to talk about these other ones, you might hear about them in the same kind of conversations, but the main one here is the CRA.
So what is the CRA? So, as I said EU wide product regulation. Every digital product in the EU will theoretically be covered if it has a network component. There is a few exceptions but they are already highly regulated. The point of it is to ensure that manufacturers of digital products, those products are secure throughout the lifecycle of the product and that consumers will have more knowledge about how secure their products are. Also, to sort of harmonise regulation, product regulation throughout the union.
You may have heard about this before because it goes back to 2022. But only now in 2024, has it been adopted, which means that 21 months after adoption, it's going to come into play. It's going to be active. At least some of it. Three years after enforcement, so October 31, 2027, the full array of regulations will come into effect. So what are those regulations? And why might this be a problem because it sounds like you have a lot of time.
First of all, it's a product regulation. So, your development time on a product might be more than two or three years. So maybe start thinking about it now if that's the case.
How will it regulate? First, the 2026 requirement is going to be reporting. Manufactures need to report breaches, vulnerabilities that they know are being exploited, report them to ENIS and the CSIRT. That's the union wide information security agency as well as the national response team. It's going to be a bit of a problems, but it should be fairly simple, fairly straightforward even if it takes a while with all of the reports.
Then in 2027 you get the standard certification. That's going to be a lot of work because there is going to be technical documentation of your whole compliance process and throughout the product lifecycle. Ending up in a consumer mark system for digital products, or network enabled products. This covers things like Internet of Things stuff. And it's all backed of course by fines and penalties.
But let's also look at some of the unstated realities of laws in general and the CRA in specific.
The goal here isn't necessarily to watch and see what everyone is doing but rather, to be seen to be acting on security breaches. And to force obviously manufacturersers to start caring about product safety and consumer privacy. Without doing too much harm. So, the problem I see here as an attorney, litigator, primarily formerly, is that this is a vast regulatory scope. Every product in the EU with a network connection. And that the primary regulator only has been 100 employees. So, that's a lot of work for these people so. What instead the CRA relies on is self‑assessment and certification. And self‑regulation can be useful, good, it can work well, but not always. It's not always that well followed. And I think that's going to be a hurdle for the CRA. But, what I want to talk about, now that I have used up most of my time is will the CRA apply to we'll say my OpenSource project?
.
And the answer officially under the CRA is probably not. But as with all these things, as a lawyer I have to say it depends. So, first of all, we have to look at the definition of FOSS under the CRA. And that's software that has a publicly available code, that is offered under an open licence. In that case, theoretically the CRA shouldn't apply to you unless you're commercial. Well, what does commercial mean? We'll get to that. But first, you should be asking would it apply to me anyway? The first question is my product isn't a product, it's not a physical good, it's not software, or it doesn't have a networking component, and a lot of things aren't products that people do in the OpenSource space. You know, contributing to someone else's project, it's not a product, you are not a manufacturer. Similarly software as a service even, while it's regulated under other laws is not regulated under the CRA.
So, your protection B is: Am I a commercial OpenSource project? We don't know what this means exactly because OpenSource has been talking with the regulators and they haven't gotten a clear answer. But we do know what it's not. We know that it doesn't mean receiving some donations. We know it doesn't mean having a contract or two for maintenance or new features being added. It's something more than just getting some money.
And then of course there is a question what if someone using your product commercially? It doesn't apply to FOSS it's another commercial product unless it itself is commercial. The manufacturer of a commercial project use as FOSS project, they have to make sure that the whole thing meets CRA's requirements unless it was sold to them with commercial intent. You make a FOSS product to include in other people's things you have got to follow the law.
So, even if your project is commercial there is still protections because I have haven't gotten into all the details of the CRA, but it has multiple levels of regulation, and the most standard level is the general product allowing self‑assessment. And no matter what project you are doing, if it's OpenSource, it will be considered a general product for the purpose of assessment. Which makes is much easier. And honestly, if it's commercial you recollect afford a lawyer, which is probably the other important thing here.
But there are even protections after that. First, you know, if you are a small business in the EU, a micro enterprise, that is pretty big, there are much lower fine requirements. So you look at the fines of 15 million euros or something like that, that's scary, but if you don't have that money, up probably aren't going to be fined that much.
And then of course, there is the OpenSource software stewards, which is mazing and interest, but basically NGOs that will be looking at FOSS projects or involved in them. Like, Linux foundation, and they'll work with regulators to create a set of standards. If you are working in that domain, there may be a set of standards that you have to follow that's much easier to will to.
So, again the text versus reality here again comes into play. The legal code that is involved here is not self, you know, it's not self enforcing. It's not a black box. So, whatever it says there is going to be interpreted over time. And ultimately probably without absurd results.
So, basically what I want people to get out of this is that is an OS project going to be the regulatory target? And think about, you know, does it even apply to me. So, you know, mostly the things you need to be concerned primarily are are you making money? Are you collecting private data that you are then selling and making money off of? And are you attempting target for someone to steal that data and put it out there and annoy the regulators? If not, you are probably okay. Hopefully you can afford an attorney. You probably should be able to. Then watch out for what happens with this. I only have 30 seconds left. So a very quick question, and this is just some resources if you want information. My website, the current act and the NSS standards that will be used to actually implement the CRA.
ANTONIO PRADO: Thank you August, perfectly in time. Questions.
AUDIENCE SPEAKER: We all understand what to do with this if we are a software designer, just move all outside European Union, establish a company in, let's say, Seychelles and work from there. The question is what is for users of this software? So ‑‑ if I represent Internet service provider and run for example BIRD for routing, should I ‑‑ can I use this product that will be not from European Union?
AUGUST BOURNIQUE: I am trying to understand that. First of all, there is elements. CRA that do apply to anyone selling a product in the EU. You don't have to be in the EU. Whether those regulators will be able to go after you, debatable. So moving to a friendly island might not help you. Two, if you're an Internet service provider, unless you are selling something with that you have a branded router that comes with your service, then you are, then you have potentially it's under the flu box thing.
AUDIENCE SPEAKER: The question is can I continue to use BIRD for Internet project and when the BIRD will be based in the Seychelles?
AUGUST BOURNIQUE: Yeah, presumably, you are not manufacturing anything. ‑‑
AUDIENCE SPEAKER: But there is no limitation for use of this?
AUGUST BOURNIQUE: Not that I can parse from my understanding.
AUDIENCE SPEAKER: Okay. Thank you.
AUDIENCE SPEAKER: Maria, developer of BIRD once again. So, I have a question. How is the CRA complementing with an AS2 because what I can see here is you are saying that the ISPs IXPs are not actually under CRA you are right, but they are under an IS2. And I would say if they are under an IS 2, they are basically required to follow CRA, or something like that, on those products they're using.
AUGUST BOURNIQUE: Well would I say this is a ten‑minute lightning talk on the CRA. That's why I said there is a whole bunch of other regulations that may come to interact with it. But, you know, we don't quite know what that other ‑‑ sorry, the name escapes me ‑‑ but the other regulatory structures that are around this are still also evolving, and again, I think what I'd like people to take away from the presentation is that a lot of this stuff can look really scary when it's written out in the legal code. But the question is how will it be implemented and who will be implementing it and who will they be implementing it on? And I think that's something people need to think about a little bit more.
ANTONIO PRADO: Doris? Nothing from remote. Thank you.
(Applause)
ANTONIO PRADO: Well the second session of the afternoon ends here. I just remind you to rate the talks, let us know what you like. At 6 p.m., there is the BCOP for who is interested, remain here. Thank you.
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
