DFN

X-WiNThe NeTWork INFrasTrucTure oF DFN

The Network Infrastrukture of DFN | Page 2

X-WiN: The Network Infrastructure of DFN

H.-M. Adler, P. Eitner, K. Ullmann, H. Waibel, M. Wilhelm (DFN-Verein)

X-WiN Services

X-WiN users (universities, colleges and other insti-tutions of research and education in Germany) may have access to the technical X-WiN platform via two services: the DFNInternet Service and the DFNVPN Ser-vice. The services are realised on two platforms, the optical platform and the Internet platform, which will be described in the following paragraphs.

The optical Platform

From an economic point of view, the lease of dark fibre links became feasible with the liberalisation

of the German telecommunication market. Econo-mic advantages of an „own“ fibre infrastructure but much more the high level of flexibility in technical provision of network services were the trigger for DFN to open these opportunities for DFN’s users. First thoughts to develop G-WiN into that direction were already made in summer 2003. The new network in-frastructure X-WiN ought to be based on fibre link technology and equipment to operate several multi-plexed links on top of the fibre. It was also elaborated that in regions, where this would not be economically feasible, carrier links should complement the design.

X-WiN is the most innovative generation of DFN’s network infrastructure and has re-placed the Gigabit-Wissen-schaftsnetz (G-WiN) in 2006 after G-WiN had been in service for four years. DFN now can look back on an almost 25 years tradition to develop its net-work infrastructures and adapt them to the most advanced technologies available on the market. While all predecessors of X-WiN have been realised with significant financial support of the German federal government, X-WiN is the first generation of network infra-structures, which has been financed through DFN’s own financial resources.

Figure 1: X-WiN fibre links (as of January 2009)

The Network Infrastrukture of DFN | Page 3

In 2004, a Europe wide tender was issued for four dif-ferent lots (fibre links, equipment for operating fibre links (DWDM), network monitoring and carrier links). This tender process yielded the results which are re-ported here.

Fibre links are in almost all cases being conducted from user site to user site, terminating in the respective lo-cal IT centres, predominantly the computing centres of universities and research institutions. In those centres, these fibres are connected to the DWDM equipment of the DFN PoP (Point of Presence) at this location. The technical standards of power supply and air con-dition are high – power supply and air conditioning in most cases are provided by the hosting site – and are contractually fixed. There are about 70 DFN-PoPs all over the country (see also figure 1).

As of early 2009, about 9.500 km of fibre links are available, more fibre links will be added if this is tech-nically reasonable and economically feasible. The exi-sting links are forming several interconnected rings. Only a few carrier links complement this fibre link structure. Fibre links are being delivered by three ma-jor suppliers (Gasline, KPN and Global Connect) and a few smaller local suppliers. In general the basis for fibre link delivery are long-term lease contracts, with appropriate service level agreements (for example mean time to repair (MTTR) for a fibre) to guaran

tee a suitable level of operational quality. Another very important item is also contractually agreed bet-ween fibre provider and DFN: the fibre provider has to agree on basic rules for maintenance and trouble shooting processes, which enables DFN to set up a co-herent and widely automated maintenance system.

With X-WiN DFN users now for the first time have an „own“ coherent network of fibre links for their data communication purposes. The fibre technology to-

gether with the DWDM technology enable very high data rates across the network and nearly unlimited bandwidth. Moreover, X-WiN can be designed not only for the DFNInternet Service but also for the DFN-VPN Service in a very flexible way.

The wide area coverage of fibre links for the X-WiN optical platform does not only offer economic advan-tages to DFN and its users but in addition also stabi-lises financial planning processes. Moreover, the high degree of fibre link meshes is highly beneficial for the overall design of the optical platform. Wavelengths (i.e. data links) established for the DFNVPN service can be put in place on quite short paths, which influences positively the propagation delay of optical signals.

The high degree of meshes also plays a major role for the network resilience. The DWDM equipment (presently from HuaWei) apart from up to 160 wave-length per fibre link offers optical protection per link. For such an optically protected link, a so called wor-king path will be set up and another resilient and path diverse protection path is established. In case of dis-ruptions, the protection path will automatically take over the working path’s functions. In general such protection paths are longer and transmission delays for optical signals usually will increase. This doesn’t matter for applications like email but may be a pro-blem for more sensitive applications like Voice-over-IP

or Video-Conferencing, where even the end user may see such delays. A high degree of fibre link meshes reduces this problem significantly, as the protection paths can then be implemented on shorter paths.

Internet Platform

The platform for the DFNInternet Service is mainly built through routers and the links between them.

Figure 2: X-WiN router technology

The Network Infrastrukture of DFN | Page 4

The requirements to the router technology for X-WiN were the ability to serve

• core network links up to 10 Gbit/s, later 40Gbit/s, and user access links from 2 – 10 Gbit/s (GE) and

• through inherent design, high availability of the most important components by keeping time to repair (TTR) and maintenance costs low.

With those specifications in mind, the central core of the Internet platform has been built with Cisco rou-ters CRS-1; the other routers are Cisco 7609. The CRS-1 routers were being sponsored by Cisco and are set up as 8-port-chassis systems with redundant processors, switch fabrics and power supplies. For special appli-cations, this router type provides means for imple-menting logical routers.

The Cisco 7609 was already tested and used in the pre-decessor network G-WiN. Figure 2 shows the general layout of an access to a user site. The Cisco 7609 is being characterised in particular by providing cost-ef-fective and performant 10GE interfaces. In the X-WiN core network, 10 Gbit/s links are therefore implemen-ted using 10GE interfaces.

Through minimising the number of routers, mainte-nance costs could compared with G-WiN be signifi-cantly reduced and the mean-time-to-repair (MTTR) of router equipment could be lowered to three hours for any DFN-PoP. Spare parts are being kept in nine deposits all over the country.

Considerations for an optimal core network topology

had the following design goals: a topology should

• be robust and fail-safe,

• be scalable and flexible and

• support minimal packet transfer time.

For the design of the core network topology, fibre pa-ths between the DFN-PoPs have been known through the results of the tendering; though most DFN-PoPs, which are in general user sites, were always linked in two directions. For DFN-PoPs, where this was not achievable, carrier links were being used (see figure 1 but without carrier links).

The basic principle for designing the logical topolo-gy for the DFNInternet Service was “follow-the-fibre” when establishing a link. At four locations, name-ly Frankfurt/Main (FRA), Erlangen (ERL), Hannover (HAN) and Potsdam (POT) are DFN-PoPs, where many fibre links and wavelengths meet. These DFN-PoPs form the inner part of the core network, the so called super core.

The rest of the topology design then follows the abo-vementioned principle “follow-the-fibre”. The most natural way of connecting DFN-PoPs was to thread them on a chain from one of the four DFN-PoPs of the super core to the next DFN-PoP of the super core. This has the nice side effect that local traffic will not be moved across the whole network. Through this me-thodology it is also clear that any DFN-PoP has two connections and a single cut of one link does never isolate a DFN-PoP. If the capacity of such a chain is not

Figure 3: X-WiN core network for the DFNInternet service (as of January 2009)

Telekom

xr-goe1 (c7609)

xr-adh1 (c7609)

xr-zeu1 (c7609)

(c12410) xr-han2

xr-wue1 (c7609)

xr-bir1 (c7609)

xr-aac1 (c7609)

xr-dui1 (c7609)

xr-mue1 (c7609)

xr-bie1 (c7609)

xr-zib1 (c7609)

xr-dre1 (c7609)

xr-jen1 (c7609)

xr-lei1 (c7609)

xr-che1 (c7609)

xr-bay1 (c7609)

xr-gie1 (c7609)

xr-mar1 (c7609)

xr-kas1 (c7609)

xr-bre1 (c7609)

xr-ham1 (c7609)

xr-han1 (c7609)

xr-pot1 (c7609)

xr-gre1 (c7609)

xr-ros1 (c7609)

xr-kie1 (c7609)

xr-gsi1 (c7609)

xr-hei1 (c7609)

xr-fzk1 (c7609)

xr-stu1 (c7609)

xr-gar1 (c7609)

xr-reg1 (c7609)

xr-kai1 (c7609)

xr-saa1 (c7609)

xr-ewe1 (c7609)

xr-ilm1 (c7609)

xr-ffo1 (c7609)

xr-aug1 (c7609)

(CRS-1) zr-fra1

(CRS-1) zr-han1

(CRS-1) zr-erl1

(CRS-1) zr-pot1

gr-ham1 (c7609)

xr-bra1 (c7609)

xr-fra1 (c7609)

xr-erl1 (c7609)

xr-mag1 (c7609)

2x10GE

10GE

1GE

Geánt

DeCix

DeCix

GC

Telia

xr-bon1 (c7609)

xr-fzj1 (c7609)

xr-boc1 (c7609)

xr-dor1 (c7609)

xr-wup1 (c7609)

xr-pad1 (c7609)

P2

xr-pik1 (c7609)

Telekom

xr-hub1 (c7609)

xr-des1 (c7609)

xr-tub1 (c7609)

(c12410) xr-lei2

STM-16

The Network Infrastrukture of DFN | Page 5

sufficient, the chain can be easily upgraded to almost any capacity necessary and the topology can be adap-ted to the real needs.

This topology has then been checked with measured and extrapolated traffic patterns, and through this step it has been optimised. The final result is a ful-ly meshed super core with 2*10GE links, easy to be upgraded to 40 Gbit/s. All upstream accesses with 10 Gbit/s are implemented at the super core DFN-PoPs. The logical topology, the core network, of the DFNInternet Service is shown in figure 3.

Since January 2009, any user access is implemented with two path-redundant access links. Thus, user sites are offered maximal redundancy for their Internet ac-

cess, a method which is currently very rarely offered in commercial environments. This was possible without increasing the access fees, because offerings of the-market achieved through a tender brought the costs significantly down.

With the X-WiN Internet platform, users of DFNInternet have access to one of the most perfor-mant research networks, which is prepared for new innovative services and which is able to react fast to new user requirements. Access capacities of up to 10 Gbit/s and almost free scalable capacities in the core network turn X-WiN to one of the most capable and performant research network worldwide.