The plenary session commenced as follows:
This is a test. Rob: Good afternoon, we are run ago bit behind schedule, therefore we announced that the coffee break would be rather short. And we said that we would start again at ten to four and that is now. So if people could please find a seat, then we have an improvised voluntary chairman to wrap up the tail end of the last session, Patrik Faltstrom. So please be seated and he is chairing the next one so we won't lose time in swapping chairs.
I think we can start.
Patrik: Thank you very much, Rob. So, as you heard during the working group meeting of the cooperation working group, this morning there were two closed sessions where one of them was sort of extension or a continuing discussion and also initiating discussion among law enforcement people on Internet related issues as I was not there myself, I am as interested as you what that actually means, so a summary of that session, thank you very much. You can present yourself, please.
SPEAKER: We just /EP uploaded my presentation, it just came in. All right. So it's a little less crowded than before but then again if, I could choose to for a cup of coffee or listen I would know what I would choose. That is all right if people keep coming um my name I amifieser at the National High Tech Unit of the Netherlands and here to do a short report on to the session of this morning of law enforcement.
What we did, we gathered and here I set up the reason why we gathered because combatting cybercrime several law enforcement agencies come across issues and cannot solve that alone so many issues, so RBN is one big one which maybe all of you know but in general cybercrime fighting many issues which we can just not solve completely alone, a problem with bought net herders and other untouchable cyber criminals in general. As we cannot solve that problem alone, we need help and we actually need your help for that, and that is the reason why we gathered also this morning and also in previous session.
Who were there this morning, maybe interested for you to know, Serious Organised Crime Agency of UK, National High Tech Crime Unit represented by myself, national prosecutor on cybercrime of the Netherlands was also there, also really important because one of the reasons for that is that RIPE NCC as, you know, is based in the Netherlands and Netherlands and therefore if we need certain matches by law or court orders then that will be applied on a Dutch basis, on Dutch law so it's really important that Dutch National High Tech Crime Unit and prosecutor is involved in that. But we also had other representatives, representative of the London action plan which is an anti-spam cooperation and also the CIOT, central information point for telecom investigations, going to be a presentation later so he is going to explain what he is doing and also of course a representative of RIPE NCC itself, and also earlier occasions of these session were also High Tech Crime Units of France, Italy, Switzerland and cypress. They are actually not here today but also represent them because we know a little bit also what their issues was and basically all law enforcement across the world who are combatting cybercrime have all got the same problems which they need help with.
So, we discussed earlier with RIPE NCC what could be a good solution and they suggested maybe there should be some sort of cybercrime task force, maybe if you have such a task force which not only involves law enforcement, also other parties together you could come maybe to some good solutions for your problems. So this morning we discussed that options, so it's just an option for now, call it cybercrime task force and what focus should it have? We were debating a little bit and I hope I got it right, get the information, but from a law enforcement perspective, establish ways in identifying and denying the ac -- access to crimeinal misuse of Internet number resources. Should the focus be of such a cybercrime task force but key issue in this is working together with RIPE NCC and its community in discovering the solutions because we know what the problems are as law enforcement but it's hard for us to pinpoint the exact solutions because the Internet is very complex and involves very much expertise and we cannot solve that alone. So we can then through such a task force we could suggest policies to existing RIPE NCC channels, address the right experts and working groups for helping, for example the Co-operation Working Group or database working group or whatever specific topic requires that. So that is what we discussed. We thought well who should be in that in such a task force? You should keep it manageable so we could put in there all law enforcement agencies all over the world but you can manage that so maybe if we could put in six law enforcement agencies, for example SOCA, our high tech crime unit, the national prosecutor, somebody from the anti-spam representative, somebody from euro poll and euro just who also based in -- both in the Netherlands, in the /TPHAEUG and they can maybe help us out on European level when laws have to be adjusted or something but also of course RIPE and RIPE NCC also said we can support that with certain tasks, with experts we can get you into contact with specific experts who can help you out and the community itself. Who of the community that would be, we industrial to figure out and if anybody has ideas of that, or says well, I would be the right partner to participate in such a cybercrime task force we would be happy to hear that from you. So, that is basically it. Two minutes, I already have finished so that is basically what we discussed this morning. It's still in orientating phase but I can -- I have really good feeling about the session so I guess it's to be continued, and if anybody -- we don't have task force e-mail address right now because it doesn't exist yet of course but if anybody has any good ideas or tips or maybe questions or remarks now, feel free to speak up.
Rob: Chairman of RIPE. I think this is a very good initiative. Now, the way procedure question -- not question, Word of advice: The way task forces are created, they are created by the community. When do they do that? They do this on Friday morning of RIPE meeting. So you are in a lucky circumstances that on the Monday afternoon, you bring this idea to the community and you have the whole week to put your seeds there. So are you present for any time, are you available this week here so maybe you and I can come together and I can help you with what the right procedures are. Basically, what we want is is a proposal on Friday morning, definition -- I mean, task force definition, goal, milestones, time line
SPEAKER: Yes. Well, I have some appointments this week but I am sure I can reschedule some of them Rob. F could you produce something like that on Friday morning then we can install this task force.
SPEAKER: Great. Thank you.
CHAIR: I also think to a certain degree maybe it's overloading of terms the word task force. Is very formal here in the RIPE community. I think he was regionally only talking about we have a small group where we are talking about these issues and reporting to the Collaboration Working Group and you brought up the idea of creating a RIPE task force which is maybe one step further than you thought. And Rob: That is what I read on the slide. Chair. We had a question.
Bill: I am guessing you are aware of the ARIN government working group which is very similar but in the north American and Caribbean region, what are you doing to liaise with them?
SPEAKER: The issue was actually raised also this morning, I don't know who in our group it was, but they also -- so we are aware of that and we certainly said that we should, somebody assimilate activities, learn from each other, maybe they have done some work we can learn from. Otherwise we have new creative ideas they can learn from so we are certainly planning if everything goes in that direction, to work together with them very closely, yes. And not only the one in North America but all of the other ones around the world.
CHAIR: There is also explicit coordination and liaison between that working group and the collaboration working group, for example there was a meeting today at lunch to kin /KROPBise this, need to synchronize with the working group but the coordination is going on.
AUDIENCE SPEAKER: Chair of the RIPE anti-abuse working group. It wasn't mentioned on your slide there and I think that just to tell mention that there is an anti-abuse working group in RIPE and the session this week is Thursday at 2:00. So that may well be something that come along to and contribute to at that point in time.
SPEAKER: Yes. It was actually mentioned this morning also indeed, but it's good that you mentioned it. I didn't put it on there indeed but yes it would be logical to cooperate with them.
CHAIR: Last person.
Matt Ford ISOC I just wanted to say that I fully support this initiative, I can it's a great initiative. There was a programme on BBC Radio 4 over the last week called hacked pieces and for neighbour heard that it was very apparent that an organise like ISOC, the serious crime agency in the UK, badly needs some education in this space and I suspect other similar LAA agencies throughout Europe similarly need help so I think it's a great initiative
SPEAKER: Yes. And I can say that many High Tech Crime Units are open because we have of course some technical guys working around but the Internet is so complicated and we are really open for any expertise or help or ideas which can help us achieve our goal. So yes.
CHAIR: Please short comment at the back.
AUDIENCE SPEAKER: On behalf of SOCA. The write out 4 programme we do very much welcome everybody's help, it was misrepresentative of our abilities and what we are actually know about this area of crime. I think one programme doesn't make our agency but certainly as far as this area is concerned we would really welcome some technical help how around we achieve goals with this because you are the experts in this so any help would be much, much appreciated.
CHAIR: Just because it's Monday I am very nice, Jim gets five seconds.
Jim: That BBC Radio 4 programme is downloadable from the inter web, W W BBC you will find a link to hacked pieces it's called if you listen to the audio.
CHAIR: Thank you very much for this report and if nothing else, I am -- I hope that we see more of the work you do and also in the collaboration working group if nothing else. More discussion on Friday I hear from Rob. So thank you very much.
And now the -- over to the original plan for this afternoon, wave presentation from Rene Bladder, from Ministerie van Justitie here in the Netherlands.
RENE BLADDER: Good afternoon, my name is Rene Bladder, I am from the small organisation from the ministry of justice in the Netherlands. And first, I want to congratulate RIPE with this 20th anniversary, see it only exists -- we could have been co-operating ten years more but see at is is a small organisation and it's often seen as a part of the big brother concept. We gather information from Dutch ISPs and Telecom operators and we provide this information to law enforcement agencies like high tech crime unit N this presentation, I want to show you that although I am big, I am into the big brother, and that we have an open and transparent process, helping the security in the Netherlands.
I want to mention that this is also the first time I present in an audience like this, so perhaps I am a little bit nervous but I will get over it. And please keep track of the time schedule because I talk a lot.
Why was the see and the invented? Well there was increase of interests for telecom data and Telecom users from the law enforcement agencies, Internet and Telecom and I think also banking data is a very good indication for somebody's behaviour, so LEA is interested in the identities behind all the numbers, telephone number, bank numbers, IP numbers but there was a big problem in the Netherlands with the liberalisation of the telecom market, first we had only one company to go to and have the telephone questions but now we have various so you get end to end relationship.
The existing communications was a process, was not quick, law enforcement agencies had to wait days or weeks to get their answer, it was not really cost efficient because the number of questions raised so all telecom providers had to have very big administration process, and the police forces had to have the same administration forces, just to send faxes. (Just to) and it was not at all complete because, well with ISPs it's even bigger problem and ISP doesn't have to register in the Netherlands before he can start up. You just start up and then you look what laws are applicable.
And other thing there was no track record and there were no standards in exchange information.
So, a group of wise guys came together and thought we have to deal with this and the /SAO*E and the concept.
Who are concepted to the Seat information system which I will show you later in the presentations, 42 organisations in the Netherlands dealing with telecom investigation or Internet investigation. I can name them all but they are here, also the intelligence servicings and some special investigators are connected to our system which is fairly unique also for the Netherlands because normally everybody has its own system, and now being part of the justice department, we provide this information to everybody here on the sheet.
Who are connected to the system on the other end? , the providers of data for me. Those are all telecom providers for fixed lines, mobile telephony, MVNEs and Os, carrier preselect services but also since the 1st of September of 2006, all Internet providers who provide Internet access or e-mail services have to connect by law to the CIOT information system, and it's a nice job for me to find out who are Internet service providers providing Internet access and e-mail services because web hosting for example is not part of the deal. We started with the Telcos in 2004, legally before that there was a -- we started in a project form, and in 2006 there was a new act where ISPs also had to connect to the see and the system. Right now, we have connection with 110 providers of telecom or Internet services.
I see the layout is messed up a little bit but there is also some central organisations involved, so we have to opt at that for number distribution mainly. The telecom agency as a holder of the telecommunications act. We were together with COIN and Central Bureau for Privacy because basically wave system with all customer data of all providers in the Netherlands and that is -- there is a privacy concern in that.
What is the goal of the CIOT? We collect current customer data from providers on a daily basis. We deliver specific information to investigation services and to the secret services. And we manage our system or develop it. I am responsible for the development and for the relation management. And we are information broker or intermediary for the law enforcement agencies and the providers. We like our -- to be the man in the middle. And we are part of the Ministry of Justice.
Now, just to make a distinction of the various types of data and not to get confused with data retention or things like that, we only have customer data. So the Telecommunications Act in the Netherlands has a distinction between customer data, traffic data and the content. Getting information of content that is legal interception getting traffic data you are talking about data retention and there will be Data Retention Act in the Netherlands very soon, it's now still under discussion in the government, but that is another story. We only deal with customer data, trying to find out who is behind a number.
So what is the service we developed? Well, on the right-hand side, you see the public prosecutors, on the left-hand side you see investigators and they are investigating a serious case. And they do that with specialised teams, I have acronym here BOIDs which is really just investigate and the dotted line is the fax they normally send to the providers to ask information. The dotted line took days to weeks to get answered and was very expensive so we invented the CIOT how to make this process and it's nothing more than that.
So an automated fax procedure here.
But to do that, we have to take care of a lot of arrangements, agreements. First of all we have Telecommunications Act, obligations for the Dutch Internet Service Providers and telecom providers are stated and where they are obliged to deliver on a daily basis their customer data. On the right-hand side, in the yellow, you see the laws for the intelligence services, because there is written down what they are allowed to ask to the telecom providers. On the left you see the Privacy Act, which of course, is applicable for the data of your customers. So all this together put down, we have put down a system with service levels agreements where we guarantee the privacy of your innocent customers and where we put investigators to have a warrant to question the system and in this way we can come with a solution where they have within seconds an answer to their question instead of weeks.
OK. So, in our model, I always try to explain what we are because if I am looking at the fora, we have a centralised database where every investigator can look around but that is not what we have; that is what you over, if I go to I can do backward searches and telephone numbers but that is something the government is not allowed to do. What we are allowed to do is written down in the investigation act and we have only automated the request for information to providers. We automated the comparison of all -- of the questions with the data that you have given to the CIOT, so asking does somebody know this telephone number, then I - -- almost the same like here somebody would raise his hand yes I know this number and I do that in automated way. We make this whole process auditable and transparent. We can report on a yearly basis or daily basis, even, what the behaviour is of the investigators in the Netherlands, how many questions they ask, not what questions but how many questions. And it's clear and process monitoring. (Precise) it's not somebody looking around and picking in the database; it's based on Dutch law.
So how does this generic question and answer process look like? Well, on the right-hand side, we have what I call it a black box but it's not one it's various black boxes for every provider in the system, so right now I have 110 black boxes. Where off daily update of data and only way to get an answer is to put a question in there. And the question is submitted by a LEA on the left-hand side and they would write down their question, put it in an envelope in, a digital envelope, we take off the envelope so the anonymous question will get into the black box. If we have a security breach we only see questions and answers not who is asking and we give back the answers to the LEA and it's not nothing more to it.
The request that the LEA can do, has to be -- well, there has to be a warrant, there has to be a legal base, a functional competent authority has to ask the question and the name of the investigation has to be in place, also, there. We audit the police forces every year to see if they make a rightful use of our system and every year we audit the providers to see if the quality of the provided data is what we expect it to be in there. And also the CIOT is audited every year to see if we put up a system that is very safe on privacy side but also on investigation side. What type of questions can you ask in our system? Well, you can ask -- you want to get the identities behind the numbers so you can ask questions with the number in it, a telephone number, e-mail address, IP address, even IPv6 or account ID which has to do with e-mail address or IP address and some cable ISPs have hardware ID for identification purposes or you can turn it the other way around and do a poster code question to pinpoint the person who lives on an address and see what means of communication he has on that address.
Well, the question is very straightforward because you get the name, address or city, even foreign addresses. The type of address, is it a billing address or connection address. The type of service, the type of network services name of service provide he is and network provider, there is a distinction in there and the date of creation, not of the database but of the data of course.
Here I have an example of the system. This is what an investigator sees when he logs into the system. And he gets his request in preparation, request in progress and completed requests. Now, if he is here, he wants to ask the question. He gets this screen where you just fill in the envelope, National Prosecutor is the authority that authorises this request. We have an investigation, we put in on what law we want to do the investigation and then you can just add questions on the lower side, a telephone number, e-mail, IPv4, IPv6, account ID, hardware ID. Well, I cannot make it any more difficult.
And this is -- the answer to a request -- sorry, this is the overview of the request where you see, again, the investigation and the questions you have put in in this example, there is a postal code for e-mail, IPv4 and telephone number investigation. And this is what big brother sees. Just a screen where the answer in the lower part where you see a telephone number with the name in the first part, the address and information of the service and the provider. Now, this is not very astonishing, but it's very important because I have been speaking to Spanish colleagues and what I just show you here with this speed, that is the speed that a investigator gets his answer also and it takes in, Spain, sometimes up to two or three weeks just to get a name behind a telephone number because, first they go to Telefonica, then move /PHO*B and star and every time another warrant, we cut down on cost and increased efficiency in this way.
And it's also my problem, always I always go ahead with the sheets with increased efficiency is something that we had not only on government side, also on provider side. I was once with KPN investigating why they still get 600 faxes a week for missing information, and one of the workers there said well, 600 faxes a week is just peanuts compared to what we had before CIOT, so the cost in fall for personnel is cut down there on the legal side. It's volume and location independent, meaning it's a web-based application and we don't pay per question but pay for the deliverance of data. It's a generic application with standard equipment and it's 24 hours, seven weeks a week available, very important for investigators. And we standardised the whole system, so government and industry have an agreed way to exchange data. And we always have some security audits on our system and some process audits by KPMG or FOX-IT in the Netherlands so we try to be a very safe system.
The production highlights in this sheet are the data provide by the providers, which is now up to 95 percent, so the last year we connected more VoIP providers so we get a higher hit rate. We have about 250,000 questions a month, 42 organisations are sending in the questions, 110 providers giving the answers and, well, roughly, 50 million telephone numbers and 31 million Internet identifying numbers or items are accessible through our system.
OK. This is my presentation. I don't know if I succeeded in being open and transparent in what CIOT does and what we do for the privacy of your customers. I can talk for two, three hours, but just give me a question and perhaps I can have a try at it.
AUDIENCE SPEAKER: Owe laugh, concerned Dutch citizen. Not really concerned but if it comes to openness and transparency the only question that remains after this presentation if those other three reports are publically available?
SPEAKER: What reports?
AUDIENCE SPEAKER: Well the system is audited from two ways, from the provider side but also from the side of the legal enforcement units, if access is being done in a regularised fashion, so to speak, and I wonder if those audit reports are publically available?
SPEAKER: OK. The audit is conducted by the Ministry of Justice and since I am on operational level I am also part of the audit; I am not giving the audit, so the Ministry of Justice is the one to answer that. What I do know is the audit reports are discussed at the advisory committee of the CIOT where providers are also invited to join, so, yes, they are available but in which extent, I don't know.
Randy Bush: IIJ. Europe is known for its concern for privacy and you are currently processing a quarter of a million of these a month from 42 agencies.
Randy Bush: What was the rate before this was automated and you had warrants and the usual protections of private citizens that some of us are used to and thought Europe was so good at?
RENE BLADDER: I think there were no numbers before the CIOT, when we started off there were 200,000 questions a year, so what we had in a year we do now a month basis, but I think that then the mobile phone was not introduced in the extent it is introduced right now and telecommunications was not used in a way that is used right now. If I see everybody here with a laptop sitting, I think in 1999 it was not in this extent. So -- what you really see is the information need of investigators.
Randy Bush: What I really see is is a radical increase in availability of data without much supervision and -- I think of Geoff Schiller's quote that law enforcement is not supposed to be easy, when it's easy it's known as a police state.
RENE BLADDER: Well, I am very glad I am not from the police but from the justice department.
RENE BLADDER: And that is basically I think the problem with this issue, give the system to the police and they will abuse this information, so I am not a fore fighter forgiving this system to the police; you have to audit the police and they are audited. I do not tolerate any (police) question in my system that is not guide with a warrant and the authorities do know that, so, yes, privacy, I think in this way you can see the behaviour of the police, I think that is better than we had before where we didn't see is the behaviour, and yes you can ask questions about the numbers, but at least we have numbers to talk about.
CHAIR: Next question, please.
AUDIENCE SPEAKER: Also -- Europe talking about 1.5 million requests a year, that is roughly 10 percent of the Dutch population. How many of those requests do you think are duplicate?
RENE BLADDER: Oh, that is not to answer for me because I don't have any (duplicate) I don't see the content going through my system. It can be that investigators work -- not work together and have the same questions and sometimes that is a plausible way of working and sometimes it's just a little bit inefficient. But, I don't see why people ask questions. The only thing we do is provide an answer to questions.
CHAIR: Let me, I insert myself in the queue here. The first thing is, the follow-up on what Randy said, you said now with laptops people are, there are so many more people using telecommunication but before we had laptops and chatted in the chat rooms that we are doing now, people were chatting in the hallways. And the law enforcement couldn't really gather the information there very easily, so I don't really agree with the argument that things are moving from one place where police could do something in law enforcement, to another one, just as a comment. But really question, two of them: First, this data, of course, is coming from the providers to a large degree and the two questions I have is, the first one, do you follow any kind of standards like the ones that ETSA is working with regard /-BG the data retention collection? And the second one: How is this funded on the Service Providers' side in the Netherlands?
RENE BLADDER: Well, let's talk about the standards, first. We adopted the standards that the local government, you call it, have connected, so it's IRS is in place -- data retention standards are not yet implemented in the Dutch government because we are still talking about data retention, there is no law in place but of course our system is adoptable. But saying that, I now have a connection with 110 providers and I don't want to change the format in /RA drastical way because I will (drastical) increase costs on their sides. It's a nice bridge to cross the Dutch government base Dutch providers astonishing A 26 or 27 euro a day for delivering their customer file to the CIOT ways 10,000 euro an year's basis. Chair so that is flat fee which is not connected to the number of requests?
RENE BLADDER: Yes, which is a flat fee for having their data available for questioning by LEAs.
CHAIR: Thank you. And for the information for people in the room, there are, for example in, Sweden, the model we use there is Service Providers have to make investment themselves but then they charge per request that comes to them, in the UK there is -- there are contracts between Service Providers and the Home Office where the Home Office pays for the investment, so it seems that your model is closer to the UK model. Thank you. So please.
AUDIENCE SPEAKER: Denis Walker RIPE NCC. Do you store historical information, if an IP address is reissued and somebody asks you the question who had this address in 2007 do you give them the answer of who had it then or now?
RENE BLADDER: The current situation is we only have the data from the past 24 hours so, the current customer database is in the CIOT system, no Data Retention Act in the Netherlands and it was filled in this way because of privacy concerns within the Netherlands, so only the last 24 hours, which was very sufficient for telecom operations but going to the Internet and having IP numbers changed /EFRP second or ten minutes or whatever you want (every), the providers can provide me data with all the during the last 24 hours and the system is set up to have a larger extent data retention is -- if the data retention law is in place in the Netherlands.
CHAIR: OK. Last person microphone Randy.
Randy Bush: IIJ, pain in -- could somebody please do the calculation that assuming is judge can issue one or more -- one warrant in an hour and there are -- 250,000 of them issues in a month, what percentage of the Dutch population are judges? (Applause)
RENE BLADDER: Those are really nice questions and I just want to say that the judge, at the end, always is looking at the righteous use of the information, and if I see how the police is working to analyse data coming from telephone poles or coming from -- like in /A*PL Doran last Queen's Day there was a -- of course they want to know who is in the neighbourhood and they are just investigating all data they have available but it doesn't make everybody a suspect, just part of analysis.
CHAIR: It seems to be we will have something from the Jabber.
AUDIENCE SPEAKER: Has made a comment that the 24-hour period is Lar retention period (already a). I don't know what he implies further with this but I think he is trying to say that despite there not being a retention law, you are retaining data for 24 hours.
CHAIR: Yes, I think the -- to end this, I will try then a quickly: According to what I know in each country today already you have the ability to save the data as long as the need for invoicing or to solve technical problems but no longer than six months so data retention legislation is not to enable data data reretention but retained long enough to police enforcements to get the data. Do you want to make one last statement before we end?
RENE BLADDER: No, but I find it very interesting to be in front of this audience because lots of questions are raised by not understanding the system and I think we made an open and transparent system where you can even visit the CIOT and see for yourself how it's working and I think it's -- we have to get rid of the big brother idea but just make law enforcement more efficient so they can get the -- real cyber crimes in your case. With that, thank you very much.
Rob: Remco, are you going to chair the next session as well. And if you run over the allotted time, you are shooting yourself in the foot as sponsor of the drinks tonight.
REMCO VAN MOOK: That can go two directions, I am over spending, I won't I promise. Well, good afternoon, ladies and gentlemen. The next hour-and-a-half will be dedicated to the fantastic subject of capacity planning, don't run for drinks just yet.
So what is capacity planning? Definition:
Wikipedia says the following and this is going to make you run to the bar even faster. Is the process of determining the production capacities needed by an organisation to meet changing demands for its products. In the context of capacity planning "capacity" is the maximum amount of work that an organisation is capable of complete not guilty a given period of time."
So here is some food for thought. So what is the production capacity of the Internet and who determines it? And for the individual pieces, all of your individual networks there might be a simple answer but when it comes to linking it all together who decides and based on what?
Background on this is actually, this is, if you look at the historical documents, this is the first subject of the first RIPE meeting after saying hellos and saying yes we should do this more often." So, it was an early subject at RIPE meetings, it pretty much died or at least I couldn't find it in minutes any more, we have -- advent of the commercial Internet in early 890s so this session is all about that very ill subject and see whether we should put it back become on the agenda again or not.
So, what is the agenda for rest of the afternoon. I will give a bit longer introduction and then to give you a bit more of an idea about the subject, I have got three other speakers invited who are going to tell about capacity planning from their perspective, Hank Henk Steenman from AMS-IX, Haro Busker V P corporate development for Europe he can nick, unfortunately he won't be able to make it so the country manager for the Netherlands has stepped in the last minute to do that presentation. I will find a bottle for you.
Capacity planning, another presentation by Christian Kaufman of Akamai comparing carriers and after that I have -- I hope to get a bit of discussion and some question and answer with you guys. So pay attention, there will be a test afterwards to give you a fair chance, these are the questions I will ask.
So the questions are -- should we discuss this and if so how and if not why not.
With that, enjoy the presentations and I would like to ask -- I am first. Good morning. So, I was nearly inviting the second speaker over. With that, here is my first presentation on capacity planning and enjoy and I will make sure that after every presentation there will be some room for quick questions and answers relating to that single presentation. If you have got wider remarks please save it to the end of the session thanks.
So capacity planning, a brief history of time.
Long -- I am about to burn up the projector, going to have to make some changes. This is interesting. So it's now gone dark. Here we go. Long, long time ago, way back in time, so in 1989 you will hear a lot more about there was a meeting about connecting and sharing European Internet networks and that meeting was later called RIPE 1. From the minute of that meeting, and don't think a lot of you have read those, there is some interesting things in there. This is literally copy from the minutes. The aim of the meeting was therefore defined as follows:
Collect as complete as possible all information about ongoing IP wide area network activities. Take initiatives to set up, coordinate and support IP connectivity within Europe. Take initiatives to coordinate European - USA IP connections. I think we have got that sorted by now. And act as nucleus for a more formal European IP coordination activity. I think we have also got that one covered: There is one more line in this and I thought this was very intriguing, different organisations explain their international connections planned for the near future. So they were explaining about the connections they had planned, or, in other words, they were explaining their planned connections. They were talking about what they were up to, what they were going to do and how it would affect the others.
So this is all of course a long, long time ago. RIPE 1 was a meeting of non-commercial networks, cooperation reason was, amongst other things, minimise cost, not maximised returns, which is what we do nowadays, so since we have gone commercial we really stopped talking about this, at least in here.
So, well, if you look at the discussions in 1989 and those in, well, 2009, how different are they, really? So what were the key issues in 1989? Would you believe net neutrality, do I allow these even Swedeses to counsel -- there is only so much important that fits on a connection to the US (porn) how much is this going to cost us or key issues now: Net neutrality, not a big surprise. Capacity constraints, there are only so many 10 gig interfaces you can put into a router. And minimising the individual costs. So this is really the single change. But has the subject really died? Well, I don't think it's dead. We have taken it outside. I can't imagine all the discussions we are having during coffee breaks are about Address Policy. So should we take this inside again. Is this a subject worthy of discussion here at this meeting?
So, with that, enjoy the other presentations and I will get back to you after those.
So, with that, I would like to invite Henk up. This is gone a bit wrong.
HENK STEENMAN: So this capacity planning at AMS-IX for those of you don't know AMS internet exchange we are this exchange point near Amsterdam, an Internet platform interconnecting over 300 ISPs, content providers and a lot of other Internet related businesses. Our network more or less works like this. One big switch in the middle interconnecting all the extra switches where we connect customers. What you see here, we essentially have two of these, a red one and blue one and at one particular moment in time either the blue or the red is active and shipping traffic.
So what is the capacity we need to plan on that? We have of course, a limited resource, especially on the access switches, the number of 10 Gig e-ports is limited towards customers but also the number and ratio of ports and in the core of the network this big core switch interconnecting all the access switches, and the number of 10 Gig e-ports is limited and also limits the total bandwidth of the Linx.
So, what do we do each year when there are we make model of how much traffic and ports we expect on a platform for the coming year. Based on that model we know how many new extra switches we need to add, how many ports we expect, how we need to grow the core of the network and we can budget it in and plan to buy these switches.
So the model we use is of course mostly by definition a historical data so it's total traffic coming into the platform, typical traffic per customer, traffic per site, and the 6 sites we have traffic ratio between the total traffic and the traffic that goes over the core or that stays local on the access switches. We have sFlow data showing us the traffic that goes between the the different peers on network that we can discuss to put big ones together. Next to that historical data on the port growth, how many 10 Gig ports did we grow over the last so many years. And we have something what we call life cycle of a typical access port which I will come back to later.
Also we can see there are effects when we introduce new access ports on the platform like introducing Gigabit link bigger effect of growth on the network I will show later. We talk M6 members at meetings like this and we have AMS-IX getting a a lot of answers and expectations. And we get a lot of information from vendors unavailability of switches and we have high density 10GE, new technologies. So all of this mixed together gives us an idea what have we can expect in the near future.
What you see here is what I started mentioning with, the historical traffic data. On the left side is what the five-minute and peak traffic grew over the last 10 years or eleven years or so since 1998, the scale there is logarithmic. On this picture is the amount of traffic in Terabyte per date we shipped since January 1998 which is a very nice logarithmic growth curve although slowing down a bit. The sync sheer typical summer period which starts earlier and earlier this year in March already for the growth. Here is more or less the traffic, the traffic, it's not that the traffic goes down but gross, becomes stable, which used to start usually in May or maybe even later but now that has been shifting towards April (grows) and march, even. So we have seen that for the last couple of years and very strongly this year. (Seen that) anyway, it gives us a good period during the summertime to work on the platform because we are not hampered by immediate needs for growing traffic.
This also traffic growth rates, interesting in itself, this is the yearly traffic growth for every month, we calculate how much the traffic volume grew compared to the same month a year before. Late '90s, early 2000, traffic growth rates in the order of 250, 300 percent, going down to the order of 60 to 70 percent now, but interesting here is what I mentioned, the introduction of new access technologies, introduction of gigabyte aggregation towards customers really make traffic growth on the platform go up very fast and we have 10GE so even though that is not a guarantee we might expect something similar, or not, when we introduce 40 Gig or 100 Gig or something.
Historical port growth: Total ports on the AMS-IX platform, customer ports, combined of the whole mix and 10GE, what you can see here is for a long time already the total amount of gigabits for Internet ports is more or less stable but the growth is driven by 10 gigabyte Internet only. It doesn't mean people only buy 10GE internet ports but there is a shift new customers coming in on gigabyte Internet and other ones upgradeing from gigabyte to 10GE but that is what keeps the net growth is in 10 gig. You can see a little bit more clear in this picture where we only show fast Internet big bit and 10 big. What you can also see here is what we call the typical life cycle of AMS-IX access port, phase of initial growth here which is more or less the same for a big byte Internet and 10GE with rather slow uptake compared to period later where the up take of that port type increases and then that typical port type becomes stable at the moment new and new access technology becomes available. When we started introducing gigabyte Internet the growth in fast was more or less zero, when we introduced 10GE the growth became more or less zero. And this is something what we can use to estimate what will happen when something like 40 or 100GE becomes available.
Also an important thing at the moment is the size of the 10GE link aggregation, so people started to connect to go the platform, they are now aggregating 10GE to bigger sizes and to show the changes in that, the red bar here show the link aggregation distribution in October last year, we had 24 link aggregated connections consisting of two 10GE ports and we had four connections or five connections consisting of four 10GE ports and in half a year time you see especially at 3 and 4 the amount of that kind of connections increased quite large. So we now have eleven connecting routers connecting at 40GE.
So, how do we use this to predict or estimate what we will need in the future? It's not -- essentially we do a regression analysis on historical data, logarithmic on -- because it gross logarithmically, we don't do that too far in the future, we typically use one-and-a-half to two years up to the end of 2010 A linear fit on the port growth, again up to 2010 and then using the life cycle model we try to get an idea what have will happen when we introduce new technology like 40GE. And then based on that, we do -- we get to a required size on ports and access and links.
So when we afly to for example the traffic, we get something like this the green and blue dots is the measured traffic on which we fit a logarithmic curve and the yellow bars an estimated arrow on this and this is the peak traffic we expect on the exchange until the end of 2010. And similarly, for the 10GE ports, so we are now at around 250 10GE customer ports and we expect to grow that until the end of last year up to around 400 or so.
And this is what I mentioned with the life cycle model, when you go a little bit furtherer beyond 2010 you grow the 10GE ports more or less along the same linear fit and then I have estimated that we might see the first customer ports either 40GE or 100GE showing up end of 2010 or maybe later, I don't know and it will follow a similar growth curve like there, so the initial slow growth rate here and a flattening out of 10 gig, so using this model we get an idea of where we are going with the platform in next two to four years.
Some conclusions we drew on this, as I said in the beginning what we usually do is estimate the amount of ports we need to in the core switches and on the access switches. This year we have come a stage where we no longer can grow the platform as I showed you on the first page, because especially the core switches are full and at the moment certainly not before the end of this year, core switches with more than 128 or -- will not be available so we had to take other measures and grow the platform in a different way.
So this year is is a little bit different than other years.
So no larger core switches but what we did is we designed a whole new network with more core switches, we no longer will have two networks but one integrated MPLS network, four core switches and all the LSPs between the access switches will do load sharing. And I am not going into that because next Friday we have to have AMS-IX technical meeting and that you will will be explained in lot of detail. But that is how we do capacity planning at AMS-IX. That is it. Any questions?
CHAIR: All right. Any questions for Henk? If not, Henk, thanks very much.
Up next is Michael - country manager for Equinix in the Netherlands. (Manager)
Again thanks for coming in at the last minute and thanks for that at least. And here
SPEAKER: I really look forward to the wine. My presentation will be slightly different, because, actually, we are not into technology planning but we are much more into real estate planning which is a whole different game than to technology planning and actually, we have to bridge the gap between the technology that still grows at a very fast pace and real estate planning which is the base of our core product.
Some general remarks about capacity planning. As I said, we are in real estate planning or real estate business and that has much longer cycles. Our cycles are measured in years, our planning cycles. If we have to deliver a new data centre it usually takes us between one and three, sometimes four, years, to deliver one. So it's very important to plan ahead and with environmental restrictions that governments put on new buildings and new build outs, it's increasingly difficult to get the building pyramid into plan.
We start planning on the basis of our customers, which is quite hard, because not always our customers have really well-defined planning and technology is different, they are really plan in terms of technology so their planning cycles are usually in terms of months and not years.
Of course, we do communicate with our customers, we have over 2000 customers and we are communicating more and more with our customers but we are trying get traction there, but it's still hard. Our customers when they have capacity plans they are not 100 percent reliable. If you look at economic downturn, the last year, you can understand why, and still there are customers that are growing a lot but flexibility is not really as solid as stone.
Still, we believe that capacity planning is one of the most important things because customers tell us their capacity planning and need within our data centre is very important for them. Actually it's one of the top three. I think the other is location, so location and the capacity, flexible capacity is together the most important for our customers. Of course the number one for our customers still quality, why they choose Equinix.
So, how actually do we do capacity planning?
First of all, we have an ambition in capacity planning, we have existing customers in our market, for our primary core market we would never want to go dark so our ambition is never go dark for network customers and eFX customers which we mean with financial markets customers. Our second ambition is never go dark in any of our core markets, and they are defined as the main cities in the markets we are into so if you look at for instance, Europe, we would consider London, Amsterdam, Frankfurt and Paris our core markets. And second of all, well never go down in any of the markets we are into. (Dark). We also want to go with our customers to new markets, and that is also part of our ambition so if you look at this ambition it is a lot but we still said that this is the base for our complete capacity planning.
How do you prevent going dark? I mean, the prevention is, it's much more difficult to achieve. First of all, we have a long-term strategic plan. You need financial strength. Technical flexibility and a customer allocation philosophy.
OK, our planning process is really there, it's something as a company you start, you learn and you grow in, well we do have a quarterly planning cycle where all the data into our planning cycle is revised and adjusted on the basis of the actuals and of the economy around us.
We have a bi-weekly updated where teams from all over the world do have audio conference, they revise the plans, they take the actions that are needed to achieve the plan. If you look at the -- as I said at the beginning, typical real estate planning is one to three years, at least, so our planning cycle and capacity planning goes ahead from five to eight years. And actually we act too upon the real estate planning we do, so it's not evenly plan but if we need to take lease on a building we really pay for the building which we would like to build out or pay for the land which is, later on, developed into a new part of real estate. Of course we search for location and searching for location, I am responsible for the Netherlands is really, really difficult. If you look in Amsterdam, customers don't want to go to Schipol, they really want to go in Amsterdam and the space available here is really tight, so -- spot location is really a very important part of your planning process, to go for location, location, location.
Second of all, we put all the data into a real plan. This is how a plan looks, for a region for us. The bars are the individual data centres we have, and this is not the real region but a fake region but the bars there are the data centres, are the -- are the amount of space, inventory we have available. Based on the fill rates we have per data centre are when we see problems and when we say that we go dark. That is defined as 95 percent full. If you look at the dots, the blue dot is the approval of the new IBX, new opening, monthly or two-weekly review we do, and the yellow dots on the screen are the planning of the new data centres so again if you look for the Netherlands, we are now building the last phase of Amsterdam one. We do plan for approval and plans for the Amsterdam two and we are already planning with legislation, with environmental pyramids for Amsterdam three. So that would be in such a plan in 2012, so that is a real plan, per city, per market that we have and discuss.
If you sum this up it will lead to ongoing expansion and more than 50 percent of our markets and it would lead to an expansion of 61,000 cabinets. Of course this is depending on the quarterly progress we make, the quarterly forecasts we get in and the real actual economic situation.
It all looks like a very nice plan to have 61,000 extra cabinets but one of the most thing is when you need to plan you need the financial means to pay for it and luckily we have a lot of the customers committed to Equinix so we have a very strong financial basis and we are able to really from our operational income provide financial means for this expansion. If you look at the promise first, our ambition in our core market is never to go dark. We see it as 20 percent increase in our growth rate in that market and we can fully provide from our operational financial cash flow for that market.
The second ambition, never go dark in core markets, 30 to 35 percent growth in those markets with our financial means we can also provide for those ambitions.
The hard part is never go dark in any of our markets that. Means we have to continue to grow in all of markets and we with our financial means cannot provide to do that, only with some financial leverage from banks or other institutes we can provide for that. So all in all if if you look at our financial strengths it provide a real good basis of really doing what we promise and delivering the capacity that is needed for the future.
Part 3 of our technical planning is to technical flexibility because the technical flexibility really provides us more continuous growth rate and growth then a step one. It is very important for customers and for us, as well. First of all, we plan to build campuses, if you have a campus you can build out and have multiple buildings on a site and grow more naturally.
Second of all we phase our building. By phasing the building, phasing the capital needed to build, we have a much more gradual growth, we can grow with our customers and we can see what happens so we can also direct the effort where the capacity gross fastest to the main markets in this way, allocate resources needed to our really important market.
We have a model or expendable design, that means we do not build out for the maximum the first time, but we look at building out in a flexible way. That means, when you have to take decisions, in systems, you really provide actual resources or extra finance for (extra) technical designs that are more flexible. Difference, air conditioning is very hard to be flexible, what we provide extra financials into those systems to later build out and, for instance, grow with the power needed per square metre and also increasing the air conditioning. That means we can build out the facilities we build after ten years, after 15 years and also provide for growth for the customers that are already in the building. Again, we would rather go for simple designs than of a very elaborate designs because they are more flexible in a long-term view and that is the trade-off we take.
We also have a flexibility within the design, so if we design floor space in our buildings, we have it in a flexible way so if one customer leaves that has private, we can change it to more shared co-loc space etc., which gives us the flexibility to reuse the building and use the building for longer period of time. We have a customer allocation philosophy: We know that in the previous years, we have not always had enough space to provide for customers, which is very hard for us but we commit to our core customers to provide capacity for them so. For networks, and for financial exchange customers, we promise ourself that would always be space in the future and we actually give them priority over other customers.
And we also focus on our ecosystems. We have customers within a certain, we provide space and reserve space for those customers. So we rather reserve some space and not give it to other customers outside of the ecosystems and those are negative financial decisions we have to take but we still think it's more important to go for our core businesses and grow the customers that do need each other in those ecosystems than to go for a short-term profit.
Last point is entering new markets. Again, there, we are very selective and grow with our customers towards new markets.
We do our customer investigation every year, and we really act on it. The investigation of 2006, 2007 has shown the first priority worldwide should be Amsterdam and here we are, today in Amsterdam with first data centre and we are very proud that have so we do look into the investigations of our customers. We also have a global approach which is not rale solid approach right now but we are start to go talk to global accounts, see where their needs takes them and try to plan the new markets we would like to enter from that respect.
And third of all, we have our own business development, we have our other means of internal communication with customers where we get the information, which markets should be next.
All in all, that provides us with a planning cycle which seems very easy but it's still very hard to accomplish. It's all dependent on the quality of data you have, it's dependent on the quality of the people that are executeing this planning, we are confident, after a few years of using this planning cycle, that we are on a good route and we would like to continue that route to the future and build a more and more customer interaction to get -- to refine it, to get it more sharp and to go ahead and to provide actually the whole purpose why we build it, to provide customer space and grow in a very controlled way for Equinix.
This gives awe little bit of insight (you a) in how we do, how real estate planning in a technical way fits into technology. I know I do -- I realise it is completely different from a normal technical planning, but this is how we do it, how we start it, and how the last years this has been very successful for our customers and for our markets.
Are there any questions on how we do our planning, how we plan our data centres and growths for the future?
CHAIR: Any questions? This is your chance. No. Thank you very much.
SPEAKER: OK. Thank you very much.
REMCO VAN MOOK: So now it's up to Christian. I hoped you recognised in the presentation he just gave that we -- Equinix dropped their pants a little, shown some things about planning normally would stay inside of board rooms. I thought it was important for you to see that there is not just network capacity planning; I mean, everybody can say well, we need five more racks in Amsterdam or Paris but somebody needs to provide those and that is all part of what we are doing as well. So with that said, Christian, are you up for it.
CHRISTIAN KAUFMAN: Yes. Thanks. Right. We have another 46 minutes to go for the drinks and again, I am ending up with the last speaker slot which is interesting. I should think about if this is personal or not. But whatever, it doesn't make sense to run now because the drinks are not ready so you can stay here and listen.
Well, that morning I was looking for the green agenda, I have to hold my presentation and when I find my slot I saw capacity from a network perspective. And I thought, oh oh now, I am in trouble. For the last years I tried to tell everybody we don't have is a network, so I was really wondering how I can talk myself out that have, but thank God he changed the agenda and I guess I am fine to go now.
When he asked me if I can talk about the capacity planning for CDN because one of the network guys couldn't hold the presentation, I thought let's see if we can combine that and use parts of my dark history as peering manager for a bigger network and mix that with the experience from a CDN and see where the differences are rather than just bore you.
Most of the parts and the comparisons work for Akamai, they might work for other CDNs as well, depending on how they build and the mileage. That is the only marketing slide I have, so stay with me, it's over in a second. To see where the differences come from. So, we have global CDN, the interesting part, we have 48,000 servers right now deployed. That is interesting because you will see the figures for the years before. We are in 1500 POPs, 950 networks, 660 cities and so on and we peaked at 2 terabyte so a little bit of traffic I guess.
When we have or when we plan for capacity, we have to take into account that we actually have three different kinds of connections. Most of you guys have any kind of transit, if you are not transit-free, certainly some peering and what comes on top of this in our case is what we call the Akamai accelerated network programme where we put servers into unit, we use IP space and UIS to serve your customers and downstream customers so this is traffic which you actually can't see from the outside because it's not delivered from our normal peering ah this is traffic that comes from the service in your network that so that is an additional (servers) thing we can play with when we have to deploy servers. So let us see what the differences are between the carriers and Akamai.
Most of the carriers have any kind of fine or at least wavelengths to connect cities and regions together. Akamai itself, as mentioned before, we don't have a network, so there is no fibre or wavelengths, besides some 12 kilometre long DWDM part in London by I wouldn't call really a backbone. You guys normally connect the cities together also with IP because that is what most of you sell, I guess. We don't have an IP backbone. You are using IGPs, BGPs, MPLS some kind of traffic engineering, probably. We just use BGP to feed our internal self build system, and then we basically distribute the traffic from there so this is something home grown, priority tarial. You guys have to take into account when you plan for co-routers, aggregation routers, peering router, switches and all this stuff, our main concerns are actually servers. Because all this content we serve to you guys is coming from the different servers. Then we have to connect them together with switches and if the peer and if they are in exchange and you peer with one of our ASs then we need a router that is but a that is not alter case. In most case in your network in the ANP version we wouldn't even have a router there.
Carriers are using all kind of different methods to interconnect. POS, ethernet, probably in some row moat connections, E3, DS3 whatever. As we are in data centres or have short reach we go for ethernet. Multiple 10 gigs.
Job is to transport traffic from A to B to, hand it over to, send it to the next city, next region, whatever. So all the capacity when you plan for, you have to, somehow, I guess, or figure out in a model, how much traffic stays locally, probably on the router if you are lucky, in the same city, dame data centre or how much you have to transport over the ocean, to Asia, America, wherever. And this end of the day you fact near a price and into your planning. That is very different to Akamai. We basically have no bit mile whatsoever because the idea about us is that we bring the content as close as possible to the eyeballs so normally the traffic to, a certain eyeball, is generated, if everything works, in the same city or at least in the same region. Where we then hand it over to an ISPs carrier.
The next interesting thing is the ratio, the ratio from the customer to customer traffic or the customer to peer traffic where you get money on one side and you can offset it on the other side for a fixed cost and the difference between customer and transit traffic is very important for us because that tell us, at the end of the day, what the costs are and you want to have as much as possible customer to customer traffic. In our case, very different. All the traffic which we have is going outbound, it's leaving our network. And stays in the same data centre or in the same city. Other, well most of the ISPs and carriers if they are actually not looking into value added services they just transport IP, they don't care what the transport, they get it on the left and transport it to the right. That is quite different with a CDN because you are not just transport the bits, as far as -- as /TPO*S as possible out of our non-network, we /SHR-S to look into which kind of service we need for which service. Because a flash server and HGDP server and WMS, all of them run on different operating systems, different software and probably even on different hardware types. So whenever we make capacity planning we have to look into that, as well.
A good thing for the carriers is the price per transport and ambit -- M bit goes significantly down with every new router. If you compare what you have these days, then your M bit or -- what an M bit costs you and how you plan for it, is different. As unfortunately serve remembers not as efficient as routers because they just get a couple of percentage faster, we actually if we want to double the traffic or quadruple have to put in much more servers so. This bring us back to the 48,000 from the beginning, you see we actually get not just every more and more servers which we have to put in data centres, the percentage is also growing from year to year. Which makes it then, as you can imagine, actually, difficult to find data centre space and enough power and enough cooling to grow exponentially when ISPses and carriers on the other side when they have removed it from the box, the power requirements doesn't grow so much.
Right. If you look, the whole stuff from geographical perspective, then North America and Western Europe is actually relatively easy to deploy. You have proper data centres and more or less equal price in transit and fixed costs for the A Xs, that changes dramatically if you go in let's developed regions in parts of Asia Africa and South America. Things which were relatively easy before become quite complicated. Because the available space and power you have in these kind of data centres is not just lower; the price is also much higher. So the -- the M bit you can transport for the same money is actually much lower. The next point is your transit price is much higher and the bandwidth you get for that is much lower. You also have some interesting behaviours from incumbents in the region, when it comes to market protection. And which kind of value their eyeballs have. Also, shipping into such of these countries is actually not as easy, either you have problems with customs or you have to pay extra tax to import servers, so that all makes it from a planning perspective, A more expensive and B, much more time. Also once in a while you need a licence or you have problems with regulations, which at least costs you money, but certainly also more time than if you put something out in a more developed country.
So, this said, how do we do the capacity planning? First, we figure out how much traffic we send right now for a certain region or for a certain network because at the end of the day, all the traffic we send out is coming from requests from eyeballs which are in your network. Then we look which kind of traffic we actually have, what is the traffic split and how high is the traffic. Right now we deliver mainly HGDP, flash, WMS, but quick time streaming or LSL certificates and other stuff. How much traffic I have and what the traffic split is for a certain region, city, then I actually look, how do I serve net today? From which city? How close am I? Then normally we look back into the last, six, 12 months. We don't have a fancy model where we put in the figures and you get a number, you need X servers and that is what you have to deploy, depending on what kind of peering session we have, depending on how the network truck structure and topology changes, we can reach a certain destination completely different from before. Factoring that into a model is actually quite difficult. Normally we look for bigger networks, we look this up manually, one by one. Then we see if we actually can upgrade an existing network. And if that makes sense. We see other eyeballs we want to reach in that network or part of the downstream networks. If yes, can I upgrade the location, do I have enough power, do I have enough space, what is the IP transit in that region cost right now and do I have any kind of political or commercial issue which I have to take into account? Because it doesn't really make sense if I can upgrade somewhere and it either costs too much money or the build out is something ready in 12 months or so when it's already too late.
If the eyeballs are actually not in that network or the latency is not good enough because at the end of the day we want to go as close as possible to the eyeball, I have now different options: I can either speak with the end network which have the eyeballs which brings us back to the ANP, if they don't, then I still is:try to peer with them. If they agree for peering, then again the usual stuff, either I can go for private peering but then they have already to be a quite big network or I go for upgrading an existing IX and put more servers, probably routers into that. This brings me to points like space, power, transit and all the points I had before.
We actually not just randomly upgrade, obviously; we look into when it makes economical sense. If there is not enough traffic for a certain server type like WMS or any other kind of traffic, we actually not upgrading in a certain region, so this is the exception from the rule. So even peers who have -- even networks who have an ANP which means the service in their network might see exotic traffic coming from a different location. So that means they might get it from the next peering or from the next transit because the bar to deploy these kind of servers into the network doesn't make sense for 10 or 20 Mbytes. Which I guess is something also better for you because the time and effort this takes to role that out is certainly not worth it for just a couple of Mbytes.
So, after we have figured all this stuff out, then we actually have the most important part, we talk to the networks. Because at the end of the day, all this just makes sense or all this just works if you find a common ground and help us to upgrade or peer with news a certain location and at the end of the day nobody knows your network better than you and we want to try to make it as easy and economical, useful for you as possible, so before you get the transit via some strange peering or some transit, I guess it makes really sense to talk to us . Good.
REMCO VAN MOOK: Right, any questions for Christian? I see on the microphone. Go ahead.
AUDIENCE SPEAKER: Bill Manning. One of the things that is of interest in capacity planning is, let me ask the question: How long has Akamai been in existence.
CHRISTIAN KAUFMAN: That is a good question. I guess since '89
AUDIENCE SPEAKER: How many of those 48,000 servers are 1989 vintage machines? The question; how do you -- -- what is your churn rate on pulling out old hardware and putting in new hardware in or do you just cut your losses and leave them in the racks and let your data centre people worry about auctioning off old gear?
CHRISTIAN KAUFMAN: No, even with the current data centre prices I guess that is not a good idea. It depends a little bit because it doesn't really make sense if you have as I was saying some exotic service like WMS streaming to replace this serve when you know this is a fading out S certainly in the points where at exchanges or data centres where we pay per rack like everybody else, it is interesting for us to get as many Mbytes out of that rack, so there is a certain threshold where older technology can't surf enough any more and when we replace them with new ones.
AUDIENCE SPEAKER: What is your churn rate on replacement? That is part of capacity planning. All I have seen is these up to the right graphs from everybody but I know that my servers tend to fail about, after about four or five years and I have to replace them and there is a churn cost and there is upgrading operating systems and security patches and maybe I can downsize a little bit how much virtualisation is involved, so as part of the capacity planning exercise look at those numbers and I am thinking I don't know how you manage all those servers.
CHRISTIAN KAUFMAN: Right. I guess three to five years is is a rough -- a good figure for us, a point. How we manage them, I mean if a single server goes down or has a problem and you can't reach him remote and you have a couple of hundred in a cluster you are probably not sending out a technician for that. So this is more handled on, if you have a certain percentage of servers down in a rack or in a cluster you replace them or replace them, whatever. But I wouldn't say we don't have really servers which are much older than four, five years. I would be surprised. But most of the servers, especially as you see as the serve amount is growing, the total percentage of these kind of old servers is really low.
REMCO VAN MOOK: . OK any other questions for Christian? There is one.
CHRISTIAN KAUFMAN: Still have 27 minutes for questions.
CHAIR: I have got some more stuff to do for them.
AUDIENCE SPEAKER: Thanks for nice presentation. I would like to ask a bit about geography. How traffic coming from some region to your servers, are you using BGP or DNS or maybe some other technologies?
CHRISTIAN KAUFMAN: You mean --
AUDIENCE SPEAKER: The second question is about the statistical, how you -- using your network to do the content could see where is the traffic come from and so on. For example, I am - /TKEUFRB supermarket and I have my images and I want to serve them via City M, I want to know all statistics, from which country my traffic is and other things, IP addresses and something like that. How do you solve this?
CHRISTIAN KAUFMAN: Right. At the end of the day, we know really good which kind of traffic we have served out because we are the owner of the service and we have the log files so every single little - sending south accounted for. So this we can use, A, for billing, if we put that all together in big files and give these kind of statistics to customer so you know whatever, Australia is really interested in your shop this month for whatever reason. That is relatively easy and works much better than any kind of NetFlow where you have, whatever, a sample of 10,000 packets and you have a rough idea in which direction it goes. We know it down to every byte. Going back to the first question, if I really understood it, how does the cache get filled?
AUDIENCE SPEAKER: No, how does content explore, for example -- how the shortest path is selected, yes, from client to your content --
CHRISTIAN KAUFMAN: Ah, right, right. We work with several methods to figure out or to map how we call it, the eyeball to the closest server. One of the stuff we really use most is the DNS, so wherever the DNS requests comes from and then latency and throughputs and stuff like that, we then figure out -- which is the closest cluster to your eyeball and then we serve it out from there. If you pick me up later I have a couple of presentations which actually go with 20 slides or so into more depth, how it works. But roughly, like that.
REMCO VAN MOOK: One more question.
AUDIENCE SPEAKER: Mark tansly. What is Akamai's readiness level with respect to connectivity to IPv6 networks?
CHRISTIAN KAUFMAN: Well we are big fan of IP version 6, no question about that. But seriously, we started to roll out, we have 3 or 4 exchanges now on IP version 6 as well, more are coming. But to be honest, right now there is in kind of a test phase because right now there are not so many IP version 6 eyeballs requesting it. So I guess, that may take a while. But if you want to have an IP version 6 session, you can actually find peering DB or Akamai, you should find of version 6 facilities where we are now and I am happy to set up a session there.
Daniel: Follow-on question on that. We will see this also tomorrow in the IPv6 Working Group but and we saw it now in the Co-operation Working Group, the governments are big into IPv6 promotion that, kind of stuff and one of the things that we run against here is that they set some goals, you know, political goals and say hey, 25 percent of all websites by some date in the EU should be reachable by IPv6 and things like that. The real problem here is actually measuring whether we reach those goals and things like that. Actually, where the websites are, I am come to go you Lorenzo, just wait -- is actually some, is relativelyeesty do from DNS and other sources but where the eyeballs are -- he is escaping -- no, no, that is unfair, stay here -- where the eyeballs are is much more interested in the other side, and obviously it's very difficult to actually get to that data, and I have asked the organisation, My Friend Lorenzo is working for for this kind of data because they know where the eyeballs are because everybody is looking at them. Many people are looking at you so one -- now, I am coming to the question: In order to promote IPv6 and also have a little bit of a handle on what happens in IPv6, would Akamai be willing to share some, however anonymised or aggregated data, that we can actually have a reasonable handle on how many v6 eyeballs there actually are? And the same question goes to Google but they always say it's too difficult.
AUDIENCE SPEAKER: We are happy to share our methodology and have presented our data on numerous occasions I saw it.
Daniel: Only very spotty and aggregated. But OK. I asked him.
Lorenzo: I have better data tomorrow I think.
CHRISTIAN KAUFMAN: Let me answer the question before your question comes, or no question. Right now we are not serving any investigation 6 package really, we are building up infrastructure at the exchanges because parts of the system as I mentioned before is home grown, we have to change all the code and all this blah-blah-blah. I guess we will not serve any version 6 traffic in the next circumstances nine, months, but in general we should talk later to see what kind of data you had in mind.
DANIEL: Planning and getting the lawyers to agree.
AUDIENCE SPEAKER: I am going to say what you are going to saying, home grown code is not an excuse it's an advantage.
CHRISTIAN KAUFMAN: Certainly.
Lorenzo. It didn't take us -- it didn't take a huge effort to put a simple redirect and in a very low percentage of client requests that would go either to v6 only hosts or v4 only hosts or you get a very fair idea of how many your clients do IPv6. All you really need is a signal of this host can do IPv6 and all you really need is one host in the world that acts as a beacon. So I mean, I am happy at that to take that off line, we are share our methodology. Wikipedia are doing similar work. It would be great if you could have numbers to add to the pool. This latches on to other things that I have been, other papers that I have been seeing recently where there is tremendous lack of information regarding this IPv6
CHRISTIAN KAUFMAN: One point regarding the beacon but that means you have one or a couple of version 6 servers in one location in the world and you use them as a beacon to see how much traffic you have and for statistics and blah-blah-blah.
Lorenzo: If your objective is to find out how much use verse IPv6 one is enough
CHRISTIAN KAUFMAN: The goal is to bring the content as close as possible to the eyeball. Having a version 6 connectivity somewhere in the world and which is so bad.
Lorenzo: Just to find out the dimensions of your target market before you start serving that is way to do it. I am not suggest having --
REMCO VAN MOOK: I am looking forward to continuation of this discussion tomorrow afternoon during IPv6. Any other questions for Christian? If not, thank you very much Christian.
I am not done yet, unfortunately for you. So, here is the test:
With the presentations you have just seen, I have tried to illustrate how wide the whole subject of capacity planning actually is, on linking pin throughout all three presentations is that a lot of the input for planning ahead comes from the outside, whereas the outside means you. And this is -- I think this is fundamental and this is something that we are not addressing at RIPE meetings right now. So, keeping that in mind, here is the test: Question:
Should we in some way, shape or form, start talking about capacity planning or discussing how we connect networks together, how we are going to upgrade, what are we going to do, at RIPE meetings? I will throw you an example, and an anonymous example, I heard a story yesterday from somebody who was started to look for de/TPHO*F service attack on their networks when all that happened was one of the broadcasters in Netherlands had up the bit rate on one of their streams. That is the information that could help out a lot and help you plan ahead. Is this something that we should discuss? And I don't see anybody by the microphones just yet. So, should we? This has gone very, very quiet.
Daniel: I am not running a network for a very, very long time, so I am probably not one of the people you want to ask but I think if it was done right I would enjoy it it, would be like the good old days but certainly I would want to avoid marketing presentations.
AUDIENCE SPEAKER: I agree, maybe the second question is -- well, yes, but the main question is are we allowed?
REMCO VAN MOOK: OK. So help this out a little. If yes so, how should we do it? Where should we do it? What should we discuss there? How do we handle constraints? I think constraints are a reality in the world today. So what -- should we have, just as an idea, during the EIX working group there is a slot dedicated to status updates of internet exchanges. Should we turn that around and have a slot dedicated to short updates from networks? Is that an idea? I don't know. It's gone so quiet I am about to shrink off the stage.
Jim: Another suggestion to do what we did when the ENUM working group got created, we got a couple of BoFs, the idea was to see was there a enough momentum and interest behind this to turn it into a working group, we felt this was worthy of becoming a working group and let's progress it that way. Maybe that is an alternative approach. It would be unwise to take away time from one of the existing working groups unless -- I would suggest that might be a better way forward, and then we can then see the -- see it fail on its own merits.
REMCO VAN MOOK: OK. But, thank you very much for your suggestion. That still leaves the subject of what we should discuss in mind. Yes, absolutely.
Rob: Yes, I had two remarks to make, and my first remark was already stolen by Jim. So a BoF is I think a mechanism to give some more time to people to think this over and discuss it.
My second remark is that we are here under the umbrella of RIPE, which is technical coordination of IP networks and I have listened just now to three very interesting presentations and I basically have not -- I had missed that element of coordination, three different segments from the market explaining how they basically based on their own historical data do a straight line or other curve fit for the next two years. A little bit of waving and gazing in crystal balls. And that is it. That is fine. That seems to work. But that is not much of interaction with either the peers or other segments of the market. So, I think it is probably that aspect that we should try to hammer out a bit more. I think how fascinating it can be to have some deep insight of how Akamai, Google, ISPs, whatever in, splendid isolation, do their internal planning but that is not really what this group is about (ISPs). I think if we can identify some common problems, somewhere where a better interface between peers and/or other segments of the market can play a role, and there the format of 1, 2 or 3 BoF sessions, it gives people the opportunity think this over. I think it's a little bit optimistic to think that would you get an answer to this question right now.
REMCO VAN MOOK: OK.
Rob: I think this this has been an honest, a useful awareness-raising exercise already.
REMCO VAN MOOK: Thank you, thanks for the suggestion and I think the coordination aspect is, as you said, something that is, well, not highlighted enough or at least not used enough right now and I think -- it is also my opinion that we could usually benefit all of us benefit from better coordination on that, in that aspect.
Malcolm: Thank you. We are not going to turn the clock back to 1989; it's not going to happen. The thing that you identified -- there may well be mileage in this proposal as you feel out through what Rob was saying, what kind of things might be interesting and useful to discuss. Certainly, the presentations that we have just had were interesting, as for our interest and learning and understanding what other people are doing but it wasn't really coordination that was going on there. The kind of coordination that you were describing going on in 1989, and that was well before my time at this put my hands up to that, was a collaborative capacity planning. Now, we do collaborative policy making here within the field of that which needs to be coordinated, Address Policy and so forth. Capacity planning for Europe, Europe and Asia, the RIPE area, the world, is now done commercially, meaning not through the profit -- the profit motive and the maximising return elements as being the most aspect that have; but actually, through market signals as being the most element of that. So the coordination of us making decisions about what kinds of investments is required or something through fields that we get either from presentations here (feel) or for that matter, coffee outside, is really never going to be a significant nor a major input into that. It's the identification of market need is going to be the main driver for investment in that kind of capacity. So I would suggest that we stay away from that side of things, there may be other elements of it and I think certainly the technical learning side of it is potentially very interesting and useful to many people, but it's not to replace the economic mod that he will we have now adopted because that is just not going to get replaced.
REMCO VAN MOOK: All right, I agree with you for about 75 percent, I'd say. In the sense that where I disagree is that not everything we do is in the market -- is put into the market model. The example I have just used with some broadcaster, well, putting up the bit rate of their video streams and an access provider not aware of that and started looking for DDoSes, that is just one of the examples where the market, apparently, doesn't really work or has its sincere imperfections.
AUDIENCE SPEAKER: I am not sure I would actually accept that. Just because there is a shortage of a resource that the market supplies, doesn't mean the market is failing; it just means the market waits needs to wait for that signal of availability and demand to transform into that drivers that deal with it. It means the market doesn't clear instantly.
REMCO VAN MOOK: OK, I will accept that.
Keith: I think that when you suggested updates, EIX format, I think one of the experiences of the EIX working group is danger of -- people have same presentation every time with new numbers and I think what keeps it important and worthwhile is that -- is operationally relevant and potentially that the talks have something which is novel or, you know, otherwise notable in them. I also think an important part is not just about tracking the demand in terms of capacity, also about tracking the technology and standards, I think if you are going to have any kind of forum for talking about capacity planning it's very important to track. When are going to get around to giving us 100 Gig and things like that. I also think if it's novel and operationally relevant, as an infrastructure operator I believe that valued form of coordination for me is fining out other people's operational experiences,, the E in EOF, so I think this is worth doing if it's operationly relevant if the talks are novel and not just routine.
REMCO VAN MOOK: Thanks.
Randy Bush: The loud mouth from IIJ. The level of coordination in 1989 might best be captured when Al Whyse put his hand roughly on the shoulder of Rick Adams and said "I am going to eat your lunch." Al Whyse is now a yoga ininstructeder andal -- providers don't coordination in that way and never have and aren't going to. What made this interesting was not exchange point numbers of how many gigabits I transmitted at 3:00 in the morning, it was the actual content describing the network, the philosophy or whether it was a network or not, the philosophy of the service provided, etc.. this is operators talking about operations and I agree very much with Jim and Rob, we should try to experiment with a forum for it and I have a name to suggest, why don't we call it the European operators' forum, some place where operators actually talk to each other about their operations?
REMCO VAN MOOK: Thanks, Randy. I am not sure if anybody else is willing to comment on that. I am sure as hell not. Anybody else? With that, it's 3 minutes to 6:00. So, I thank you all for your attention. I'd like to invite you upstairs in the Amsterdam room at 6:00 for drinks. Rob, do you have any wise words to add for today?
Rob: No. Maybe in half an hour.
REMCO VAN MOOK: OK. We will get back to you in half an hour. Thank you.
The plenary session then concluded.