Thursday, 31st October 2024. 9am. Side room.
Connect Working Group
WILL VAN GULIK: Good morning everyone. I think we are slowly getting to the hour, we could start. We'll leave like one or two minutes because we are not in a rush today, leave one minute for the people to come and wake up and get their coffee and so on.
If you are here for IPv6, that's not the good room, you are at Connect.
OK, welcome to Connect, your second favourite working group, first one being RIPE NCC services obviously.
OK, so we got ‑‑ so I am with my fabulous co‑chairs, Stavros and Paul who, it's their first Connect session as chairs so welcome! (APPLAUSE.)
So we have got an agenda and so we are quite sure that you all read the minutes of the last session, at RIPE 88, that's on the website, if you didn't see them and as we didn't remind ‑‑ well, maybe you can go and have a look but I suspect that everything is fine like that.
Then we have got like a bunch of presentations that we have for the session. I would like to remind everyone that we really appreciate if you rate the talks because that helps us know whenever you liked what we were able to produce as content.
I also know that we have got Tina, our scribe, here and I can't remember, Angela is the doing the chat monitoring, just ‑‑ and with that I think we can start with the PeeringDB update, Leo the stage is yours.
LEO VEGODA: Hello everybody, good morning. I have an update for you about PeeringDB as an organisation and a community product and also some product features and changes. So first things first, Shawna is the new secretary and treasurer for PeeringDB. If you are a PeeringDB sponsor, Shawna is the person who will be contacting you about renewing your sponsorship. Secondly, there are three seats that come up for election in 2025, if you have data in PeeringDB and your organisation is not a PeeringDB member, you can become a member at no charge and there is a how‑to on becoming a member and voting on our documentation website. We encourage you to take up your membership and cast a vote in next year's board election.
Key thing here is the admin committee has grown since I last gave an update and there are more opportunities to volunteer on the admin committee if you are interested. You don't have to commit set hours or anything like that; we recognise that sometimes day jobs take precedence and having a large admin committee is very valuable.
Similarly, the outreach committee has grown, you might recognise some faces and find some faces new here. In this community Lynsey Buckingham from links has been doing some really good work for the outreach committee. If you have noticed changes in the way that we do social media recently, that's Lynsey so worth noting her contribution. And the product committee now has Terry from AP NIC as a member, this means we have people on the product committee equally spaced around the world, we are changing the way that we are doing work to be less meeting‑driven and more asynchronous but if you want to, if you want to join any of these committees or stab for election or whatever, write to stewards at PeeringDB.com and we can help you.
Now on to the actual product. We have been introducing a new search and it is very popular, it gets most of the search queries that people conduct on PeeringDB. We need to retire the old search V1 or legacy search, it's based on some code which is no longer supported and updated so we are going to be deciding when to retire early next year. If you have searches which you would like to do and they are not working the way you want, it would be great if you would write to me and let me know and we can go and incorporate tests for your searches in the improvements that we are going to be rolling out to V2 search in January and we want to make sure that the search works as well as it possibly can.
In order for a search to to work, it's also important to be able to log in. Some people mentioned to me last week they thought there was a "them" error not being able to log in, we had a problem with sending one‑time passwords by mail, that has been fixed; it was an us thing, not a you thing. We recognise that if you can't log in, that's not good. But we want to improve security, we want to make multi‑factor authentication mandatory, it needs to work for everyone. We now have three methods, passkeys, hardware tokens like a UB key and authenticator apps. We hope it's enough, if making two‑factor authentication is mandatory is going to cause you a problem, it would be great if you would let us know how it would cause you a problem and why because if you can't log in to update your data then that's a problem, not just for you but also for the people who are searching for your stuff.
So we will be doing more communication about this, but this is a first effort saying please let us know because if we don't know, then we can't do things the right way.
We have had a couple of minor security issues, one that we found ourselves, a couple that were reported by a user. We are rolling out a fix to them in November and we will be publishing a blog post about those issues. This is more to let you know, hey, we have a security issues how‑to and if you report an issue to us, we will do our best to get it fixed and we will recognise you publicly and give you a token of appreciation, a hat or something along those lines. You can request the specific thing. But please if you notice something, let us know and then we'll go and fix it.
And finally we now have support for the location of an IXP switch. I wanted to point this out so that you know that it's there so you can either look for it or make sure you are including it in your export because if you don't know then you won't be looking and you won't get the benefit.
We do a retrospective blog post twice a year, October and April. This is our most recent one. We do a retrospective rather than a looking forward because of the way the product committee works, the product committee votes rather than a business setting a plan. So it makes more sense to go and say this is what we have done then than we guarantee this is what's happening, we have objectives like rolling out multi‑factor authentication, improving search, that kind of thing but we can't guarantee things the same way a business might be able to. And that is my update.
If there's time I am happy to take questions but you can always write to me and all the product committee or stewards and we would be grateful for your feedback.
WILL VAN GULIK: Thank you Leo.
(APPLAUSE.)
I know it's the morning and we are not all awake yet. Do we have any questions? I don't see anything in chat. No one in the queues. Then I guess that's a thank you very much and yeah. Happy to see that things are moving the right way there.
Next up I'm really excited about that, we have got an update or like a new mention or like what's happening with IXP manager in the new version, thank you very much Barry for being here and explaining these to us. Thank you. The stage is yours.
BARRY O'DONOVAN: Thank you very much for having me. It's great to be back, I haven't been to RIPE in person since before Covid, it's good to be back and give you an update on what's happening with IXP manager.
Just a little context for anyone who doesn't know me, I work with INEX, the peering point for the island of Ireland founded nearly 30 years ago, we are coming to our 30th anniversary shortly. We run three exchanges essentially in Ireland, we have INEX land one and INEX land 2 in Dublin and a smaller exchange in INEX Cork and in a particular context here, we are the home of IXP manager. As one metric as to why exchanges need two alike IXP manager, across throws thee exchanges for IPv4 and IPv6, we have 30 BIRD instances, a quarantine route collecter, a production route collector and two route servers, that's a lot of provisioning you can't do my hand.
A little update on the manager itself in terms of the overall global view of it.
This is a map we maintain and every green dot here represents an exchange that we know is using IXP manager. So when you install IXP manager you are encouraged to go to IXP manager.org and to register the fact that you use it. We do some validation, we make sure you are actually using it, there's good data in there and you know we can do with the IX‑F member end point so we do validate it and then once we are happy, we add it.
Once it's added, IXP manager.org website every night will run around all of the exchanges that are registered and gather some stats. So for about 140 of those we use the IX‑F member export directly on that instance of IXP manager, we get pristine complete data. For the other 100 odd where where they haven't published exposed the member export, we use PeeringDB, so these stats very much are the lower bound of numbers for IXP manager usage.
What they tell us of the 237 exchanges that we know of that use IXP manager, there's over 12,000 networks connected and of those 12,000 networks, 7,000 are unique.
We can tell again on a lower bound that the edge peering capacity is 320 Tbps and that's over nearly 18,000 connections.
And as a very particular lower bound of the exchanges that we can query directly, that's the 139, we know there's traffic, there's accumulated traffic bits of nearly 90 per bits per second. You can see across the exchange that use manager, 10 gig is still the predominant port. One gig is still very popular, some exchanges have started removing it as a product including INEX, INEX only legacy members have, it was a legacy products, if you have it, you still can keep it with you but you can't buy one gig ports any more at INEX.
The other number is interesting there, there's 17 four hundred gig ports which is very small but it's growing. There's about 600, 100 meg ports and 3,to 210 meg ports, wing that's an anomaly, we are quite sure it is. When you provision a port in IXP manager in the drop‑down menu for port speeds, ten is the first option. So we are guessing that a lot of people using it maybe provision their route servers don't care what the port speed, just picking the first option and moving on.
And then just the last slide on this, we have a most‑connected networks table, packet clearing house top of the list, anybody here from Cloudflare? Tom around? Tom, congratulations, the last time I looked at this, you were level on 80 with Hurricane, you have added four and they have dropped one. And then it starts falling off.
You top the table.
So as I said, I haven't presented here in quite sometime, we never presented IXP manager V6, I want to highlight one or two things of interest, the top line item here from IXP manager version 6 was our new user interface based pre‑filtering, if you are peering with the route servers and you want to implement some kind of routing policy where you say, look, it's fine I am going to peer with everybody at the rout servers except one or two ASs, if you want to do on your edge router you have got to tag the prefixess you announce them and you have to filter the prefixes you learn from the route server, if you are a very small regional wireless ISP, you might log in once every six months and talking about large community tagging might be double Dutch.
If you are a big network, you might have a very onerous change control process. So in the user interface, you can now do this, it will do 90 plus of the use case where you don't want to peer with certain ASs or you want to prepend to certain AS, you just, you can just do that and it does it the same way you would do it, rather than tagging your routes on egress, we tag them on ingress rather than filtering the routes you are learning on ingress, we filter them on egress, the mechanism is the same, it just happens at the other side of the BGP session.
And now because we are doing this, we can no longer reconfigure route servers maybe four times a day, if you want to do some prefix filtering, you expect that to go live within five minutes, so that means we could potentially be.net dating the route servers every five minutes so we need better resilience, we now pair route servers within IXP manager, we say route server one for IPv4 on LAN one is peered with route server two or IPv4 on LAN one, when route server one wants to update, it gets a lock from IXP manager, it downloads the configuration, if it does a dif, if it reconfiguration BIRD and makes sure BIRD is still running and sessions are established and it releases the lock and while all that's happening a second paired route server cannot get a lock, it has to wait.
So if the first one fails on that process, route server two will keep working in its kind of legacy configuration until there's human intervention.
Then there's a few other things from V6 listed there, the last one security enhancements. We had a third party that did ‑‑ that was using IXP manager and got some penetration tests done for their own reasons but they shared those results with us. So we did a lot of security fixes in V4, there was nothing critical but we represented an older internet so you could do things like user enumeration through forgotten passwords, for example, things like that were improved during V6's lifetime.
On to version 7 which has been in development now for most of 2024. That we hope to release this in the next few weeks an the plan is to have it released before URIX, we got a new developer during the year, we have a full‑time developer on the project and he is fully funded through the sponsorship programme and I was, I just realised looking at Leo's sponsorship slide, I forgot to put in a sponsorship slide in here, we really have one headline sponsor at the amount, APNIC but I know there's people out there and Richard is pinging me looking to give me some money, I will be chasing him up.
So what triggers a major release. Usually it's a new minimum version. PHP, for V6 the minimum version you had to have was 8.0, that will continue running on 8 one and 8 two and 83, you might see additional log messages about deprecations for but for V 7 the minimum is 8.3 released late last year and that's a hard requirement, which is why we now have V 7.
What else happens when we trigger a major release. Well we have a new recommendation for the server operating system so when you are running V6, we recognised Ubuntu 20./# 04 long‑term support, for V 7 we'll be recommending Ubuntu 2704 long‑term support and we update a lot of the foundational stuff, so you know when you are doing web development, you kind of know model, controller the MV F confirmation work, we use a framework called Laravel an the current version and next version will be compatible with A 3, we'll have full support in terms of PHP and our framework right to you to the end of 2027 for this and we update the third party libraries and there's improvements, both fixes and new features. So we are going to look at a couple of those now.
So I am sure anyone that's in the EU have had the headache of analysis two, part of that process was getting information security management system and getting IS 027001 certified, and as part of that process, we had gaps was identified while we did have unit testing, we didn't have static is good analysis, we added that during V7, when we turned it on, we had about 230 warnings. None of them were security related. Quite a lot of them were very esoteric in the way the framework works, even with the Laravel plug in, it had a lot of warnings about something called facades, it's a particular way you develop that framework. We had to fix all of those to get a new baseline. The static code analysing comes with eight difficulty settings from easy to hard, we are kind of in the middle of that, a lot of legacy code you can see there's 522 files analysed there or more than that in fact. That's a lot. So what we do is we fixed all those errors and base lined it and turn up the difficulty for all new development and that's part now of our continuous integration pipeline, so we pushed to the Github, static code analyseer kicks in and we need to see those passing before we release new code.
Another big feature that's coming in V7 which will be firstly for IXP operators but we'll start pushing this down into the front‑end as well for members, is a diagnostics suite. So when a member has a problem, there's a lot of things to check in an exchange, especially as the exchange gets more complicated, maybe they peer to LAN 1 and LAN 2 and there's a huge amount of things to check. We have introduced the diagnostics suite and what we have developed so far with the runs listed here, customer diagnostics is basically database validation. Is the member set up cell in IXP manner injury, are they set to connected rather than not connected.
Do they have an AS set, all that kind of stuff. It just does some basic database validation to make sure there was no user input errors.
The Irrdb suite says this member is a route server client, so they should have those entries and should have been refreshed at least in the last 24 hours, that checks that. The physical interface diagnostics does both database validation and it does SNMP request to the switch, is the port up, does the port speed on the switch match the port speed on the database.
The BGP sessions diagnostic is one that's already proved helpful, as I said if you peered at all three INEX exchanges on V4 and V6 and you used the AS 112 service and you are a route server client, you will have 24 BGP sessions to INEX.
That's a lot of checking by hand. So the BGP sessions diagnostics suite will check the sessions that they didn't bounce and you are not approaching or haven't exceeded your max prefixes, it handles all of that.
And then as we get down to the layer two and layer three, we'll do a Mac ping, make sure there's layer 2 activity. ACL is appropriate and do IPv4 and IPv6 pings and you can see they are all bunched into particular files so we can add more diagnostic suites, no problem.
This is just a screen shot of the diagnostic suite running for the transceiver. So you can see that we have pulled out from the switch the make, model, serial number of the transceiver and then you can see we are also pulling out from SNMP the light levels, this is a 100 big port, we can see the light levels for all four channels right there in IXP manager without having to log into the switch. It will pull out the warning and critical light levels, so it will warn you if you are low or there's no light.
Some additional new features, one thing that we have been asked for for a long time, if you are a member of an exchange that uses INEX and they have implemented peer to peer graphs and say at INEX you have that and there's 110 members, if you look at the peer to peer graphs at INEX, you are going to get 110 graphs, they are not going to be Y axe aligned and they are not going to be sorted alphabetically, it's going to be extremely difficult for you to pull out our ten ten peers or try to see big significant changes in traffic.
So we have long‑term plans that implement time series database functionalality but in the meantime we sort the graphs. So every night ‑‑ it's kind of similar to to how links does it, early every morning we'll look at the traffic from yesterday and put it into the database and we are going to sort the graphs for you based on yesterday's traffic.
We are also adding version documentation which is really going to be helpful for exchanges that have an older version and are reading about new features not realising they are not available in their version.
We have a very complex vagrant development environment, we'll talk about that in the next session and then I am just in the last five minutes, if anyone has any questions, certainly make your way up. But in terms of development plans, this might answer some questions. Before you ask them.
Once we release V7, what we are going to be look ago at is a complete redevelopment of the member facing portal, where the IXP members, it's where the networks log in, and some of the changes we want to make, as well as just a full rewrite, we want to add internationalisation so your members can log in and interact in their own language. We do have perennial plans for time series database support, we open BGP on the long figure, we commit to getting that working in early 20 at that, that's a fairly big project, people have often offered open BGP configure samples, that's the easy piece, it's the whole ecosystem that's the hard bit, it's having the configure generation, the looking glasses, prefix filtering, we show members what prefixes we are filtering and why, so the whole, it kind of needs to be on par with BIRD for the whole ecosystem to be useful.
Birdseye is our looking glass. People have been asking if we can use BIRD watch or other looking glasses, we want to support that. And then just as a URX project, we want to do for for RFC 8950. That's the end of my presentation, thank you very much.
(APPLAUSE.)
WILL VAN GULK: Thank you Barry, amazing presentation, looking forward for those new features, I recognise a lot of them, we use them as well in our own network, I see the value of those features, it would be nice to see other networks using them, in the meantime do we have questions.
AUDIENCE SPEAKER: Thank you very much for the presentation and the great overview, I was actually impressed about the footprint of IXP manager, I was not aware there was such a huge number of deployment. We do not use IXP manager ourselves but as you know, we offer IXP to interact with the IXP problematically. Do you have plans in your road map to do some kind of integration for the greater benefit of your users?
BARRY O'DONOVAN: I know exactly what you are talking about and if you were to join INEX today, you have to fill out a word document as your application form, which is very naff, so we do want to build in a provisioning pipeline manager, so how you add members, provision ports. When we do that, what we are going to do is look at the IX API, that work is already done and it could makes sense that we use that as the pipeline.
AUDIENCE SPEAKER: Thank you.
AUDIENCE SPEAKER: Will, so we are one of the users of the IXP manager in the sense that we couldn't upgrade because time and so we are running a terribly old version, 4.9 something. And so my question would be for a small IXP who didn't get the time to do the upgrades, would it be more clever for us to go ion the whole thing there from scratch because I expect that upgrading from an old version now that's seven is coming up, maybe it would make more sense?
BARRY O'DONOVAN: You know I don't need to tell you this, when you fall very far behind it gets very difficult because you may need intermediary versions of PHP, depending how many members you have, it might be easier to do it from scratch. If you want to talk to me afterwards, I might be able to help. The main thing you need is database migration, if we can help with that, we will certainly try.
AUDIENCE SPEAKER: It was mostly because I also heard some other IXPs were running older version and maybe they didn't pay attention as much as we did. So yeah, it's a good thing to know.
'And my second question is there's ways for like people to support that project also so we can go and do that, you have got the links on the website?
BARRY O'DONOVAN: Exactly, there's a sponsorship section, so yeah, please go there and we now have a full‑time developer that's being sponsored out of the existing funds we have but we do need to go and find new funding.
AUDIENCE SPEAKER: Thank you very much for that.
BARRY O'DONOVAN: Thanks very much.
WILL VAN GULIK: Thank you Barry. So for the next topic, we are going to have Max Stucci telling us about blogging and censorship on the Italian internet for football reasons and that's a presentation I am personally looking forward to.
MASSIMILIANO STUCCI: You are setting your expectations high!
Good morning everyone, so first of all, this is something I decided to write because I was a spectator on the side, I don't live in Italy, I haven't been living in Italy for a while but I am I keep following the IT NOG telegram group and people kept complaining and so I built my own view of the thing and on the other side, I am also a very big football fan so I'm unhappy on that side. I used to be a football referee, I was part of the whole game.
And we all love it, right? We all love football, yes we do. Actually, one day I should join Freddy at one of his matches. Yes. OK. So. This is a story, that starts from what people have reverse engineered being as very small group, very influential people and one of them was really called out, he is actually the owner of a football team. And when I was starting this story a few days ago to someone, they mentioned why is it always with Italian rich people who own football teams that something bad happens. I don't know. I have no idea. But things can easily go wrong all in the name of football and that's the part of football I didn't like, I like the game, I don't like the rest.
So I started imagining a conversation that might have happened sometime ago because the problem at the source is how can we avoid having people watching illegal streaming online, which is something I fully understand. So imagine someone sending a Whatsapp message or telegram, we have a problem, people are watching our streamed games illegally, they are not paying for the services, we need to do something.
So I have an idea. Give me ten minutes. I asked my IT experts here, we can force the ISPs to block those sites, we can do that, don't worry. People only be able to watch those matches any more, excellent looking forward to recover our losses and create more jobs because the idea they had this mind and we'll see that in a bit is is that it would add 1.7 billion euro to somewhere and 9,000 jobs.
So it all started, July 2023. I put some screen shots where you can see it was voted by basically unanimously by everyone, both chamber and Senate in Italy. So the law went through and what does it say. ISPs are forced to block fully qualified domain names and IP addresses as told by authorities. There's a key word here, IP addresses where the illegal activity is the sole activity. And you see there's an asterisk because, well, things will change later.
The blocks need to happen in 30 minutes. Sorry? Well, they need to happen in 20 minutes. So who is in charge of defining the blocks? Copyright holders. So. The AG come, that's the general agency for communications, is in charge of providing ‑‑ was put in charge of providing a platform to do this. So what is the platform? The platform was actually written by ‑‑ I put here a branch of, it's a legal counselling organisation, it's a lawyer's office, which had a small branch and I think with five employees I was told, and they built this website, this system, and then they donated in front of a lawyer to AG com, to the government basically, it was initially built to support 18,000 domain names.
And why, would you ask? Well, it was put in the ‑‑ it was basically a number they came up with. Random. We don't know where it came from.
AUDIENCE SPEAKER: ISPs requested that.
MASSIMILIANO STUCCI: Okay. They needed to build then their own solution to query this system, figure out what needed to be blocked. The problem was the specifications kept changing, so there were lots of discussions about, hey, which version of the specifications do you have? And people would wake up in the morning and say, I saw there was a different response to the AI request, what do you guys see? There was a testing platform where, apparently, not all the ISPs got access to, so some of them had to go directly production. This basically, this platform, is a ticketing system. And copyright holders have access to create these tickets, they can contain addresses and/or domain names. ISPs query the system, they build their blocks, blah blah blah, and don't forget this needs to be done in 30 minutes.
So you have this pressure all the time.
But the point was, can you remove these blocks? Well, not really. So the initial set up said, there's a block that gets put in, if no one complains in five days, done; that address stays on the list, we cannot remove it. We are sorry. So you can imagine easily what this could lead to: You have IP addresses that will never be able to be used again any more to provide service to Italian people.
There are some exceptions. In some cases, some of the blocks were removed at one point but very few exceptions. So we had some bumps in the road because of course you can imagine with such a powerful system that's in the hands of third parties, it wasn't even a month after it was put in place that, well Claude flare made is into the list, just an IP address and a few thousand websites basically were blocked in the whole of the country and the response from the agency in charge of the platform was like yeah but this is minor don't worry, it's OK. It's fine. Don't worry. It's all OK.
Then zen layer made it into the list, if a few months that grew to 17,000 entries. So it was close to, remember the limit of 18,000 and here we are. The problem is how can you complain about this list. You can't. Well maybe.
And yes, so people complained because of course imagine you were using Cloudflare as the CDN for your website and suddenly the most frequent used example is the Peadina shop in Italy, the a simple us customer with a simple website, it doesn't care who the provider is, they want the website to be online and suddenly the website is not online any more and how can you complain, there isn't really a procedure to do that so people started sending, there's a special system in Italy that certified electronic email, started sending them, but as far as people know, they never got a reply.
And there is no formal way to do. You can probably try going to court but we all know the small shop owner whose website just isn't reachable any more from the whole country will not take probably someone to court because they don't even know who to take to court so next step is they will move to another ISP probably. Is the list public? Can we look at the list? No, you can't, there were people who tried supplying freedom of information act requests and the response was denied.
No, we can't show it to you, we can't give you the list because of security matters. So no. We can't.
So it's there, ISPs can see it but they also have a mandate not to provide it further.
Now on the same thing the source code is not available for the main platform even though it's supposedly public domain. At one point one leaked on Github but, well there was a Github repository where they claimed it was that source code but there's no full certainty that it's that one because no one has ever seen.
AUDIENCE SPEAKER: It was.
MASSIMILIANO STUCCI: It was, well there's no full certainty, we know it probably is, most likely is. The next step was OK, as you can imagine, guys, this wasn't successful so people were still watching illegal streaming and actually the traffic was even going up so we promised 9,400 new jobs, it's not happening, 1.7 billion euro, not happening: So we can be more aggressive, I have a great proposal coming up and excellent was the response here so what was the new proposal. There was a new law and regulation in October changes now IP addresses need to be used predominantly for illegal purposes so it's not, it doesn't have to be the sole activity any more, if they are used predominantly for something illegal, then they can make it into the list. There is an official concept for the unblocking which was not present previously, now here the two most important things though.
VPN an DNS providers are also now subject to applying filtering and it doesn't matter if you a for entity, if you are a for entity, you have to nominate a legal person in Italy they can talk to a to apply this filtering and I can't imagine so I told you I don't live in it he /HRAORBGS I live abroad, I am not going to put my name on a list there too so whatever.
But here's the most important one to pay attention to, if an ISP has any suspicion of illegal activity coming from one of their customers, they have to go to the police or they risk one year in jail. And that's it.
So, but then this led to October 19th. I don't know if you can read it up there, can you read the URL on top? So on October 19th, someone, one of the copyright holders decided that Google drive was bad and so so it was put into the list of domain names that were blocked in the country so at 7pm on, around 7pm on a Saturday, that was the lucky part, on a Saturday, someone put Google drive as a bad actor into this list and of course no one in Italy because remember the concept that these blocks have to happen in half an hour so by 7.30, in that evening, no one was able to access Google drive, well without a VPN.
The speculation is that there was a list somewhere published on Google drive with the list of streaming websites you could use to watch the game that was going to happen soon after that.
The block was removed actually after midnight but there are reports so there's someone who made some RIPE Atlas measurements, get back to that in a moment to verify which IXPs were still blocking and some of them removed the blocks hours later, this of course led to plenty of discussions people realised wait a minute what do we have here. We have this system that allows someone, we don't know, because of course blocking Google drive made it into the main news papers in Italy and people realised we have something wrong and we need to fix it. Of course, on the other hand from the government the words we all heard were everything is fine, as you could see, the system works because it blocks stuff. No worries, yeah. So what is the situation now? There's an imper have an incaps /HRA, there's a list of Italian banks who make use of that, the list now is still about 17,000 entries long but the total counting even the unblocked ones is 25,000. And 6,000 IP addresses are on the list out of 8,000 total.
Now there's a tiny little detail. What have I not mentioned so far. There's no IPv6 that's been blocked!
(APPLAUSE.)
No IPv6 has been blocked. So well this might be a reason to implement IPv6 for Italian ISPs.
There were other small bumps in the road. I was told after, so Marco told me it's a couple of ISPs in Italy, imagine you have, you have to show a block page that has a specific series of consistents told by the government ‑‑ sentences ‑‑ but it turns out in a way the blocked page like the URL also made it into the blocking list and we called it inception blocking because the website is blocked and the blocking page is blocked, whatever.
There was a big question like how did no one figure out that GoogleDrive, service used by a large number of people in the country, made it into that list, it means that someone had no idea what they were doing and we don't know who these were. And on top of that, we would need to have some ‑‑ a list of critical infrastructure that is not going to make it into this list ever.
And one of the come is re from AG com said we do have, keep calling it a white list because it Alliance and we have a white list but Google never participated in any meeting with us so we didn't add them to the white list because they didn't communicate with us so whatever but now they are just trying to force them to join a working group and actually there are rumours that the government is trying to sue them and we are all holding our popcorns for that.
What is the shiny future ahead though? There is apparently a new platform in the works because this one works so well that we need a new one and this is something that I had not considered initially, I told you that there was a person that was doing measurements to figure out when our source were being blocked, but I told you in the new law there's a part says if you are ISP that sees someone doing something potentially illegal, you have to report them or you can go to jail. Now if I host a RIPE Atlas probe and someone is running tests to see if a source is blocked, potentially my ISP could be reporting me. So people have started turning off their RIPE Atlas probes saying I don't want to have that in‑house, that's a side effect I had not considered myself and now operators, of course, left with more questions.
So this is the story, I needed a little bit of help from a bunch of people because I don't have insights into being an ISP in Italy, here's a list of people that helped, it doesn't include Marco, he didn't want to be included but thank you Marco as well and this is it, we still have a couple of minutes for questions
STAVROS KONSTANTARAS: We are still good on time, please give a round of applause for Max.
AUDIENCE SPEAKER: Thank you for talking about this, it's probably the first time I have seen anybody talk about this, note, this is absolute fucking bullshit.
MASSIMILIANO STUCCI: I decided to do it because I don't live in the country, I have a little bit more freedom.
AUDIENCE SPEAKER: My question is you pointed this out earlier that first of all the IP address needed to be used only for illegal streaming and then predominantly for illegal streaming, how ‑‑ is there any enforcement there or is it literally just hoping that the right holder go /PW*EUPBGy software. This is only used for streaming or is there any form of validation done.
MASSIMILIANO STUCCI: No.
AUDIENCE SPEAKER: Cool.
MASSIMILIANO STUCCI: As far as I know but maybe Marco can tell you a little bit more but there's no, no. When I tell you that they put GoogleDrive there as a resource that's they needed to be blocked, how can you rely on the same people or the same infrastructure to give you more transparency. So no. There's nothing publicly that I know of that gives you any of that data.
AUDIENCE SPEAKER: Cool. Thanks for that.
AUDIENCE SPEAKER: I am wondering as this list is not public that you were saying could we probably reverse engineer this from providers looking glass servers, where they probably do some host routes and at least sounds like the ate al Jan software is not as bullshit because that block list is a PDF.
MASSIMILIANO STUCCI: Yeah I get that.
AUDIENCE SPEAKER: I can answer to that, there is a public chronicle run by I think some researchers at the university that you can use to check if a resource is blocked or not. Other than that, it's hard I think to get the full list but ask any friendly Italian ISPs and they will be happy to lick it I think.
AUDIENCE SPEAKER: Maybe go in the background.
AUDIENCE SPEAKER: Thanks Max for this great talk on the great wall of China, here the main problem is they keep saying an IP should be mostly used for piracy ‑‑ there's no way in the world to prove an IP is used for a single service, it's a law itself that is broken.
MASSIMILIANO STUCCI: It's broken in many, many, many parts. Like people have written posts and tweeted about how broken it is, on the other hand you get a wall from the government and the agencies saying well it's fine, we can, this is it, that's what we have and we are going to use that.
AUDIENCE SPEAKER: Thanks.
AUDIENCE SPEAKER: Hello. So the thing is obviously bad but why are you somebody from outside Italy doing this presentation, why isn't some Italian ISP, is it illegal at that point?
MASSIMILIANO STUCCI: No, no, it's not it's about taking initiative, it's just that and I think people just didn't think about it or didn't have time so.
AUDIENCE SPEAKER: Because since it's going on for almost one year, there should be massive ‑‑
AUDIENCE SPEAKER: There have been presentations in Italy about that. I did some, the operators did others, it's just what it is.
AUDIENCE SPEAKER: Because there is something to be done about it. Called maximum damage, for example. Turn the system against itself until it closes, there needs initiative to be done.
MASSIMILIANO STUCCI: When they were going to introduce the new regulation I think it was the CTO of Google Italy or some director of Google Italy, they posted like, if you push us to send all our suspicions to the police, here's the number we have that we are going through these days, and it's about 10 million cases just in Italy. Do you want us to send all of them to the courts? And then we can do that. But, yeah.
AUDIENCE SPEAKER: Thanks for the talk. I am not referring to the PDF in our country, which is kind of silly. But the question is what happens if an ISP doesn't obey? I mean, apparently this law is contradictory to the constitution which says censorship is certainly not allowed in your country and so ‑‑
MASSIMILIANO STUCCI: But now it is allowed.
AUDIENCE SPEAKER: Is the question is is the constitution above the law or the other way around like in Switzerland? If you, in Switzerland it's like this, if you don't apply the PDF that was mentioned, you can ask for a ‑‑ for an order, you can apply to the court and since they know it, they didn't send the order yet. So.
MASSIMILIANO STUCCI: As far as I know, there were, well Marco.
AUDIENCE SPEAKER: I want to say that this started in 2006, this is nothing new, it just has been an escalation. When we first started talking about this, it was obvious that this battle will be lost, broadcasting rights for football in Italy are worth about 1 billion euro per year. This is massive. And is much more than what ISPs can do. So this was from the start a losing battle. On the other hand I believe that ISPs should have folded and they did not, this is nothing new, they started this kind of censorship in Italy started in 2006 with gambling websites, then it was kiddie porn, then it was something else, something else again. Every time Italian ISPs said, OK, we will do that, it's annoying but we are going to do that. So that they won't bother us any more with other things. And guess what, every time there were other things. This started with DNS blocking and now we have near realtime IP blocking, what's next, we don't know but we should have said from the beginning to politicians that internet blocking, that's not ‑‑ does not work, cannot work, it's not working in China and Russia and it's not going to work in Europe either. Thank you.
STAVROS KONSTANTARAS: I would like to close microphones, if you guys have any last questions, this is your time, in two minutes we close the queue.
AUDIENCE SPEAKER: I would like to mention in this country we also have PDF vendor list, we have DNS blocking, and constitutional court decided that such a blocking was not a censorship.
AUDIENCE SPEAKER: Hello. I am Christian from Italy. So hello everybody. I just want to show you something that maybe we are not always saying enough. This was a law from the Parliament, not from the government and there's one law maker against the law. We had the occasion to speak with people in the opposition to say the principle there is right, it's just the application is wrong. And you know, this means that there is a huge whole of culture about internet, about our world in the Parliament and this has big consequences because law makers also the opposition think that piracy is something to fight and they think their principles are always right because if they say in front of the people that they are against piracy, they get votes, this doesn't matter if they are from the left or the right, we have to be more talkative to people from lawmakers everywhere because this is not finishing in a well, way, that Saturday was really a mess when GoogleDrive didn't work in Italy, it was during football matches of course, it was like Saturday night, but we had a lot of discussions with other people in Italy and hospitals that use IP addresses, there are a lot of important infrastructures it could be very wrong in the near future, we need the help of everyone to be more loud to this, to go to the European Parliament and to share this thing because it's not just like a technical thing, it's just like little thing. Thank you.
MASSIMILIANO STUCCI: Thank you
STAVROS KONSTANTARAS: A round of applause for Max please. So for the next session, we have Paul as a host because he is going to do a panel discussion with some very nice guys, one of them is Tom from Cloudflare and the other one is Theo from Interlink, I would like to invite the panellists here on stage. And Paul is going to moderate it.
PAUL HOOGSTEDER: Hello all. I'd like to to have a discussion with you and these people on stage about the communication between exchanges, their members, communication between the members because a lot of things have changed in the peering world. The first modern exchange started at around 42 years ago, at a time exchanges only had five or ten, maybe 15, members who all knew each other, often came from academic background, they used tools like IRC to talk between them, they often had each other's phone numbers, you could actually call someone if there was an operational problem on the exchange.
But nowadays the large exchanges has almost 3,000 members and they are not sharing their phone numbers any more and some of us run networks who are on more than a hundred exchanges, like one I run or even two or three hundred exchanges or more. So we need modern tools to communicate about things like out ages, planneded maintenances, new locations, new port speeds, ring numbering effect changes because changes do ring number, a /24 is not big enough for your peering VLAN at least for some exchanges, half of the exchanges in the world actually have fewer than 15 members still.
But big ones, most the big European exchanges are nearer a thousand members. So, there's a lot of communication that needs to be done.
Question for the guys on stage here. Outages and planned maintenances, are you happy with the way these get communicated to by the exchanges and do you see differences in the way exchanges communicate with you? Do they do good things or bad things?
THEO: We are pretty happy with that, at the end of the day from our perspective, we are on a lot of internet exchanges most of the time, we don't notice because we have automation in place so the automation takes care of most anything really. If there is an outage or planned maintenance, our systems will automatically disable the internet exchanges and reenable it six hours after whatever entrance we noticed, if it's an extended outage and we get yelled at for other members that our peering session is still down because there was a split horizon on the peering LAN, things like that, that's when we care but those are few and far between, at the end of the day any communication is better than no communication and that's basically it from our side really.
AUDIENCE SPEAKER: I think the challenge we see is the difference between IXPs and how they handle maintenances, they are very well automated, there's other IXPs that do not, so I think that's one of the main challenges that I think will come in a couple of more questions that you will ask is that there is no homogenous landscape around IXPss and how they handle maintenance, that's the main challenge we see across different regions.
PAUL HOOGSTEDER: It's interesting that you call automation of processing of authentications there is a standard for that called main note where you have a standardised email saying things like when the maintenance will be done, when it will be over in what time zone it is, what location it refers to. I did talk to quite a few networks that are on many, many exchanges and asked them in advance of this meeting to share their comments with me and one think I got from the large content networks which are all over the world was that the notifications often didn't clearly say in which country the thing occurred. And these large networks have got European and team and north American team and South American team and it takes them ages to find out what exchange actually, which explaining actually is that's come to do the maintenance and many exchanges do not use the same name for themselves as in registered in PeeringDB.
PANELIST: Also something I looked into this morning and one of the major IXPs connected in Europe who have exactly one port with one session that is apparently or the IXP is apparently fully automated, it's showing three services in a portal for us, three are live apparently and there is a lack of data of accurate data from our point of view in even IXP database or not IXP manager as a product but IXP port /ALS that's leading to inaccurate data when it comes to maintenance, we don't have the services but we still get notified for those three.
PAUL HOOGSTEDER: Weird. New member, sorry, the presentation doesn't go to the next slide. Yeah, some exchange actually do notify you when they got a new member or new member leaves, do they want to information, do you use it?
THEO: Yeah we definitely do, at least new member notifications, the old member or the member has disconnected notifications aren't super valuable because if the IXP is operating in a decent fashion, they will sink hole the IP address and our BGP sessions will go down and automation will go down the BGP session, at that point again don't really care, new members definitely do care, we are after all a network that wants to connect with as many networks as possible so knowing a new network has turned up on the inter exchange, even if we peer with them on another LAN in the same Metro, still important for us to have that redundancy to make sure we have those members. So yeah, the new member mailings are definitely appreciated, they are not always the most helpful, sometimes it literally just say well it's a new member, OK, what's the IP address. Be nice to know or what's the communication or how do we interact with this network, those kind of things would be very helpful
PAUL HOOGSTEDER: Do you think mailing list is the right tool to communicate this?
PANELIST: I think the mailing list, there's a good reason why it exists, however we get tired of reading that's on the mailing list because there's either way too much communication around topics that are not operationly relevant or way too little and also regarding kind of members that left IXPs, what we do more is just depeering or just removing the sessions when they are idle for two weeks, we just do not communicate any more because we see that most of the communication that we do goes into a black hole so there's no response, we try and signal on telegram or whatever and there's nobody answering, that's I think something where communication is not working out the way we would like to see that on IXPs.
PAUL HOOGSTEDER: Thanks, there's some other things regarding two mailing lists that I want to discuss with you. Do you want changes to provide you with multiple accounts, multiple roles, different email addresses to send you maintenance notifications, a different one for general chat between the members? Do you want to have role counts? Do you want to have personal accounts?
THEO: Yes, definitely, it ties into a wider ask really that would be really nice to have and that we don't really see in the ecosystem today, single sign on, it's not really relevant to the conversation we are having right now but for most big companies, they use an identify eprovider, be that Google or PeeringDB or any other kind of role.
PAUL HOOGSTEDER: Microsoft probably.
THEO: And having the ability to use single sign‑on with groups and everything associated with that would be a massive relief for bigger companies like us. So that in general is definitely an ask and that comes from roles and individual accounts, as far as failing lists is concerned, the more mailing lists the better, honestly, the main thing is having at least a separate mailing list between new members for example, tech mailing list and a general discussion mailing list, those should be the minimum three required mailing list an internet exchange have and active moderation to ensure that people stick to it and then the fourth requirement should be in a no ticketing system is allowed to subscribe to the mailing list.
PAUL HOOGSTEDER: That's a good one.
THEO: Can we stop doing that, that would be great.
PAUL HOOGSTEDER: Thanks.
SPEAKER: I think also to add to that is that I think the contact data that most of the IXPs have, it's not their fault, it's pretty inaccurate, so we see a lot of communication going to a dead end where the relevant operational people not being able on the mailing list. So I mean I am personally ask on telegram regularly, do we have a contact? And I think most of the IXPs have that data in their CRM system, it would be really nice if the XP could disclose to whatever GDPR grade is possible who are the contact people, a provider that we can contact and not just sending an email to the mailing list can someone from somewhere reach out to us, there's a lot of improvement in communication we could do and I think the data is there as well.
PAUL HOOGSTEDER: Thanks. One of the other things I wanted to bring up that in some places we still get emails saying we have got a new prefix, we got a new downstream ASN, do we actually need this in this day and age?
THEO: Absolutely not. I am aware for, I think, especially within this community, it should absolutely be no. But we are present in some other regions, especially within APAC for example or Latin America where the incorporation of RIR based prefix filtering isn't fully incorporated yet, let alone the RPKI. In those cases, I can understand why but I expect better from certain internet exchanges that won't be named that are head quartered in Germany that they would try to enforce a better policy, which they are currently not doing in India specifically. In India they are still very happily communicating, please update your prefix list or if you want the route servers to update the prefix list, please send us an email, that needs to stop or that needs to go away. But I fully understand why that's still very much a thing. We don't use them. They instabilityly like you said, they go into the bin, we don't use them.
PAUL HOOGSTEDER: We don't use them either.
THEO: But I get why they exist, we should probably make sure that they stop.
PAUL HOOGSTEDER: Especially in Japan, it's quite often that we still get all these messages but also I see it in Italy and that surprises me; people in Italy should know how to update the RIPE database and ROAs and filter sets.
SPEAKER: I think the IXP that you mentioned has a very positive thing, it shows rejected prefixes for example, but nobody takes care of that, there could be an option to just like hey, this prefix is being rejected, please look into it, there could be automation around prefixes not being accepted that help addressing the same issue like updating filters and making sure the prefixes are accepted and I think there is a database that is pretty good an the filtering at least in Germany works pretty well so that could be something to pick up.
PAUL HOOGSTEDER: Thanks. I would ask people in the room to come up to the mics and tell about their experiences with communication with exchanges.
AUDIENCE SPEAKER: I just want to say in Italy there's in an automation and routinely report filter prefixes and why.
SPEAKER: One thing I noticed this morning as well when I looked through the portals, not none, none of the IXs we are connected to, you check light levels in the portal, less than half of them allows you to check errors on interface, there's a lot of stuff you can put into the hands of the user as well to figure out hey, what's wrong with the IXP, not just prefixes rejected but other stuff as well.
AUDIENCE SPEAKER: Hello, an internet exchange not a member connected to, they always talk about the mailing lists and this is a strange topic because you are complaining that sometimes there's uninteresting chatter, I'd like to add that some of the big ones contribute to that. Reply to all with mail that went to the mailing list announcing a maintenance and then one of the big ones answering the ticketing system of the other big one, can you please confirm your IP address, that's exactly what we do not want to have. So to all the big ones that have their help centres in some low‑wage countries, please educate your people to not do "reply to all." Please. Thank you.
PAUL HOOGSTEDER: Absolutely.
AUDIENCE SPEAKER: I would have a question for IXPs as well, DECIX, for example you have the ability to download the IX‑F files, sort of JSON with all the members, you don't need actually email announcement that you have a new member, you can download the JSON file and work on it, usually those JSON files exist, there is a standard including in PeeringDB but as a member or someone who is in the in PeeringDB, you have no access to those files, D kicks is offering us a participant list but it's the same file, most internet changes you don't have access to that, I would like to see that actually from all the IXPs and about the ROA counts, we just had a conversation and they actually do not allow ROA accounts because this violates some local whatever rule, they disallow this do not subscribe to the mailing lists because this sucks. Now what's your circuit ID, whatever request and no one needs it any more.
AUDIENCE SPEAKER: Hello. I just want to clarify this is something we have to do and this limitation is related to that access to the customer portal, not the mailing list, in the mailing list we prefer to have NOC et but to access the customer portal because of the NIS 2, you cannot have raw emails.
PAUL HOOGSTEDER: Thank you.
AUDIENCE SPEAKER: Guilty of at least someone having replied all to mailing lists bigger than ten times today. Radical opposition to at least half of this. I think we should change, we should go away from mailing list. I would love to see no mailing list needed to this at all. If we can get the information just in a standard way where we can query this, let's get rid of them and use them for communication, for discussion, for events, for meetings, for associations, whatever. Not for announcements, please. And on a realistic side, we are not going to get rid of people reply all to them, it's like we hire more people in NOCs than I can educate on a daily basis and even if I would see it day job, it would not be possible, there's an easy fix, announce only for maintenance, there's no reason you need to do reply to announcement for maintenance.
PAUL HOOGSTEDER: I think when you can define a specific email address where you want to receive your maintenance notifications, then the mailing lists of the exchange should automatically drop any replies that it gets, all of them.
AUDIENCE SPEAKER: Absolutely, agree.
THEO: I'd like to add that I fully agree, but I am trying to be pragmatic in a way that it's possible for Akamai and Cloudflare and for Interlink and for a lot of the bigger folks to write systems and tooling around whatever automated end point that exists from an IXP, maybe something that IXP manager exposed, be that something else, I am cognisant enough that the internet as a whole consists of a lot of meow meow networks that may not be interested writing automation around these systems and for those the mailing lists don't make sense. So yes, it will be great if we didn't, but we probably should.
AUDIENCE SPEAKER: The idea of standardisation of maintenance notifications, all that thing, somebody did actually try to do this, I think it was 2002, I think we tried again in 2005, we tried again in like 2008, it was like you are never going to get the whole world to agree that this particular subject has to be ‑‑ nice idea, but I can't think how practically we are going to do it. A mailing lists and things like that, I think from our point of view, it's quietly evolved from what was a discussion group of one relatively small number of participants to now quite a big list which is not a lot of discussion happens on it but it's used for sort of announcements and that kind of thing. We have this every now and then should we split this up, make it announce only list. I don't know. Everyone seems to have forgotten how email is supposed to actually work and the headers are in there on the mailing list, your ticketer should actually respect the headers. I don't reply to header for a reason. I add a list header for a reason and if your ticketer is now doing the wrong thing and I have blocked pretty much most of these now anyway, because there's quite a lot of really quite stupid ticketers out there that really ‑‑ because people don't know how headers and emails and how it actually works. So I just filter out now if somebody does do the reply, yes just look at the headers, if you sent it to the only to the list and we know who you are, fine; if you sent it to a whole budget of other people, I am going to automatically moderate that now because it's kind of annoying I don't know, I would be interested to see what other exchanges do. I think culturally, it works differently at different exchanges, we use IXP manager and the way it's kind of set up is quite specific to INEX, that's the way they do things and it didn't work for ‑‑ this isn't the way Lonner does it, we did a different thing how we do things, IXP manager is great but some of the stuff seems to have become a bit of a global standard or you decide not to use that bit of IXP manager, we are not going to use the data for that but so kind of a hard one to get consensus on. Whether we get standardised maintenance calendars and that kind of thing, I think we did try it and we got exchanges but we are just blocking every single maintenance like oh, it's going to block every maintenance now every Wednesday for the next ten years so we can never do any maintenance. So it's kind of, well, OK, how does that work and how long do you get to ‑‑ how long a maintenance window do you get and does that mean we can't do any maintenance at the same time so don't know, it sounds nice but I don't know if we'll get there but stupid ticketers are stupid, maybe a bit of training in a NOC but about what an IXP is because we see quite often somebody looks it up by IP address and then he emails us and says, oh, the sessions with are down, why is it, go look it up by AS so that's maybe a training thing about what an IXP is.
STAVROS KONSTANTARAS: Let's not start with this discussion, I have some interesting stories to share and we are going to finish in two. I have seen other NOCs that have been outsourced and come back after a maintenance and say we see our BGP session, why don't you accepted us a full BGP table of the remaintainance, I mean, bro, as an XP connection, why do you expect a full table? But anyway, that's a different story. We can have a whole different discussion on that. Rob?
PAUL HOOGSTEDER: There's somebody in the back at the microphone.
AUDIENCE SPEAKER: So, is it possible to get all those wishes written down somewhere.
PAUL HOOGSTEDER: I am going to work on that.
STAVROS KONSTANTARAS: I think the local RIPE NCC are keeping notes in the background, maybe we can have those notes and work and that later.
AUDIENCE SPEAKER: Not only what has been discussed earlier but generally what people do expect, now we have Cloudflare.
PAUL HOOGSTEDER: I did interview quite a few people before this and I will watch back the recordings and write something up. Yeah.
AUDIENCE SPEAKER: I wanted to agree with what Robert said because he just hit the point, it's about the format, it's not with about the transport, I am against getting rid of mailing list, some people might want to have something transported on mailing list and some people don't have the possibility to implement some fancy end point, as a matter of fact a fancy rest end point is completely useless, it's not a perfect use case for static querying when it's going to happen a maintenance, there might be transparent, we are lacking proper specification to announce this information.
STAVROS KONSTANTARAS: I see a newing with coming up.
AUDIENCE SPEAKER: What I see is that we could probably I hear... that was really a good part and we should probably remind people that the thing moved a bit in NOGs and so on and say OK, we don't need to go to the length of like announcing new prefixes and so on, just inform people, by the way it wasn't the best, now we have got this thing automated, you don't need to do that any more and if we inform the people maybe the traffic on the mailing list will get a bit lower, we don't need maintenance, we can have a new way to push things with the people so we just might want to once we have got something aggregated, we could go and push that a bit like that that's maybe a consensus or something we can go and push to the NOGs and then we will ‑‑ we will be moving the community in this way an I think that's something we should follow up on the mailing list.
AUDIENCE SPEAKER: One more remark. I am not at an IXP, I think most of you on stage are not an IXP but there are lots of IXPs in the room, my question would be can you take those questions, suggestions, whatever, and take this to URIX for example, maybe you have working groups there and you can discuss things on a broader level because connect might be the place to discuss this from a customer and user standpoint but people from IXPs meet on a different level and maybe they can work on this together.
STAVROS KONSTANTARAS: I think this panel discussion from Paul is an excellent panel discussion to initiate this subject and people raise concerns and think, OK, maybe we need to move forward to something more modern but definitely we should follow up in a task force or something and come up with something, with some regulation, not regulation but something for standard, more than standardised way, some guidelines, I think we should go like that now.
AUDIENCE SPEAKER: Just to comment because I am the chair of the board of the URIX, I thought I would respond and just a general comment to start with because I also work for an IXP, I think it's always very useful to hear this feedback, but I think we also recognise that the communities are quite diverse, there are a lot of people with very strong opinions about how we should communicate with our members, but they are not always homogenous, it's not always consistent. So I think for IXPs it's also very much about listening to their membership and trying just to put the various views of the membership up and then just commenting on the URIX part, at URIX we very much welcome these type of discussions, we have these customer panels where we invite different customers of IXPs that come and provide that input, we don't see it as our role to regulate and come up withstand standards and say everybody needs to communicate that way. But I think those types of discussions are super helpful and interesting to have at your IXs but have them in your IXP community as well, right, if you go to those meetings and he can press your views there now absolutely you are welcome to come and provide your insights stove Ross: Thank you very much.
(APPLAUSE.)
STAVROS KONSTANTARAS: I would like to invite Will for the closure of this working group session.
WILL VAN GULIK: I think we are just at 20 seconds over time, something like that, so I will not retain you so much here long, much more long because you need to go back to your coffee. I would like to thank you, thank you everyone that worked this morning and that came to the session because we know that we had like a rough night yesterday. I would like to remind everybody that we really appreciate if you rate the talks because that will help us see what you did like and didn't like and I would like also to apologise because I think we had like a miscommunication with one of our panellists and so he couldn't be there, here and so on, that's fine, so apologies to Remco for the miscommunication. And for the rest, I think I will tell you that we will see you all in Lisbon. Thank you very much.
(APPLAUSE.)
(Coffee break)
