Slide 1

Janet Update GridPP Collaboration meeting Sept 2012 David Salmon

Slide 2

Topics Janet6 Janet & e-Infrastructure Issues & futures....

Slide 3

Janet6 Neil Shewry Rob Evans

Slide 4

SuperJANET4/5 Getting us out of the cycle of networks that were bursting at the seams at the end of their life Dedicated fibre backbone, allowed us to grow. Subject to contract SJ5 was originally due to finish October 2011 Extended to October 2013 Still no congestion Originally 10Gbit/s, but have upgraded through 40Gbit/s to 100Gbit/s

Slide 5

SuperJANET5 Greater visibility of the optical layer than previous networks Managed by Verizon Business Janet NOC had read-only access to (most) transmission equipment Regular meetings with Verizon on implications of capacity requests Good communications between Janets senior engineers and Verizons. 40G and 100G trials before deployment However, still some distance away from the laser-face Deployment issues that were contractual rather than technical or operational

Slide 6

Janet6: Building up ideas Thinking & reflection as SJ5 progressed Greater visibility of optical layer New features of transmission systems ROADMs: Colourless; Directionless; Contentionless (more later) Direct experience of transmission equipment TVN Developments in Janet and other R&E networks Use of point-to-point circuits Dynamic provisioning

Slide 7

Requirements Gathering

Slide 8

Bandwidth, bandwidth, bandwidth... Flexibility Agility Cost control Service delivery Delivery of third party services Cloud services Reliability and resilience Partnerships Public/public and public/private Information assurance Off-net Anytime, anywhere access Internationalisation of education Management of costs Funding environment Costs of change

Slide 9

Requirements Gathering Functionality Responsiveness to additional network service requirements Flexibility Protection of interests of teaching and learning and research sectors Separability Ability to increase bandwidth at controllable cost Scalability High reliability by minimising single points of failure; flexibly coping with major breaks; assisting in resilience to customer organisations Reliability With agility End-to end Across JANET Support for the provision of wide range of services: e.g. High information assurance; Unified comms

Slide 10

Procurement Strategy

Slide 11

Options appraisal Bespoke arrangement Procure dark fibre infrastructure Procure optical transmission equipment Management in-house by the Janet NOC competitive dialogue procedure

Slide 12

Procurement Strategy Fibre Procurement Lot 1: Fibre Infrastructure in the UK Options: Scotland, Wales, Ireland, Aurora, London, Research Lot 2: Fibre Infrastructure in Ireland Framework Agreement(s) for 4 years Initial order from 5+2 years to 10+5 years

Slide 13

Procurement Strategy Transmission equipment procurement Supply, design, delivery, installation, commissioning, maintenance, training, specialist technical support Framework Agreement for 7 years Initial order 3 + 2 + 2 years

Slide 14

Project Timetable

Slide 15

Slide 16

Procurement Update:

Slide 17

Fibre Procurement Prequalification phase Launched October 2011 13 responses Shortlist 6 bidders

Slide 18

Fibre Procurement UK Government approval to proceed Cabinet Office July 2012 Contract Signed July 2012

Slide 19

Transmission Equipment Procurement Prequalification phase Launched November 2011 17 responses Shortlist 6 bidders

Slide 20

Transmission Equipment Procurement Contract signed last week September 2012

Slide 21

Current Status

Slide 22

Slide 23

From requirements to services Capacity Exponential growth continues Commodity IP networking R&E requirements Must be able to scale the network Not just more channels, but faster channels over the networks life 100Gbit/s to start 400Gbit/s? 1Tbit/s?

Slide 24

From requirements to services Ubiquitous connectivity Access to Janet from anywhere Not all in the scope of Janet6 Other activities in the company Feeds the capacity requirements Feeds the external connection requirements Feeds the reliability requirements

Slide 25

From requirements to services Distributed campuses Inter-site, intra-organisation connections VPNs Separacy Centralised firewall

Slide 26

From requirements to services Outsourced services Compute, data storage, applications Low latency for interactive services High bandwidth for bulk data services Reliability, reliability, reliability.

Slide 27

From requirements to services Research National and global collaboration Interoperation with GEANT and other R&E networks worldwide LHC, SKA, ITER, Bioinformatics, Climate Data deluge Services Dynamic provisioning of layer 2 circuits

Slide 28

Similar topology to SJ5. Two north-south paths New east-west path from Birmingham to Nottingham Erdington to Lowdham No backbone PoP at Reading Still have a regional PoP STFC RAL as a backbone PoP New PoP in Acton, West London Janet6: Core Fibre and Backbone Resilience

Slide 29

Illustrative only The details are subject to change Similar to SJ5 Collector arcs Spurs Regional networks connect to two different routers Over 70 optical nodes Amplifier sites, add/drop sites, and router sites Janet6: Regional fibre

Slide 30

Lighting the fibre DWDM transmission equipment Still some way to go in the procurement Theoretically 80 channels of 100Gbit/s per channel Roadmap for 400Gbit/s and 1Tbit/s with better spectral density ROADMs Native capacity on the core to be 100Gbit/s 10Gbit/s multiplexed over that where needed Muxponders or OTN switching Coherent technology required for 100Gbit/s doesnt sit well with dispersion compensation required for 10Gbit/s RNEP fibre may still carry 10G native

Slide 31

10G Circuits: muxponders or switched? Muxponders Switching

Slide 32

ROADMs Will be more reconfigurable than SJ5 One of the collector arcs in Scotland required 13 engineers to be in dotted along it at the same time to commission Get away from the air gap between RNEP fibre and backbone fibre that exists on SJ5. Different transmission systems on collector arcs and backbone Requires two client cards and a patch cable to bridge a circuit from one to the other

Slide 33

Building the network and the NOC What will this give us? Janet NOC will have a view from the fibre up the stack to the routers We will see when fibres/lasers/receivers degrade Lower signal levels Increase in error rate Forward Error Correction (FEC) working harder We will see where faults are Fewer administrative boundaries to cross Directly translate the communitys requirements into engineering

Slide 34

IP service As you were Finish moving away from SDH access towards ethernet 802.1ad bonding for 20Gbit/s, 30Gbit/s etc. 100GE if you can afford the optics (have seen a dramatic fall) Still IPv4, IPv6, unicast, multicast In London, retaining Telehouse and Telecity Harbour Exchange as major points of external connection Continuing to expand Telecity Manchester Content providers and content delivery networks IX Manchester operated by LINX

Slide 35

VPNs VPNs over primary access Inter-site connectivity Low bandwidth layer 2 circuits Other services? Statically configured VLANs (site/RN boundary, perhaps RN/core) BGP signalled VPNs (RN/core, perhaps some sites) Bandwidth management between IP and VPNs These are layer 2/3 VPNs Not IPsec If you want to be sure it is encrypted, best to do that yourself Would you trust this man with your encryption keys?

Slide 36

Lightpath service Good question Ethernet over MPLS network will remain Upgrade the MX960s to 100GE interconnects Move 10Gbit/s circuits from dedicated wavelengths to EoMPLS Investigating OTN switching as part of transmission procurement May be too early to do much especially as well also have MX960 platform Dont need guaranteed bandwidth? See VPN service

Slide 37

Migration 66.666666666666667% (ish) chance well be moving core PoPs Moving routers from PoP to PoP overnight would be challenging Requires all regional networks connected to a PoP to move simultaneously (no link between old PoP & new PoP) Cant move anything until all the regional network fibres to a PoP are ready One delay holds up everything Remember the newts! Need to upgrade the switching fabric anyway T640 to T1600 T1600 to T4000 Investigating options & costs

Slide 38

Migration Aiming to have the network installed and ready by the end of March 2013 Migration between March and August Post-Clearing until end of October is contingency only End of October is a hard cut-off.

Slide 39

Janet6 summary Extensive fibre footprint Direct management of optical transmission On-schedule Very capable strategic foundation 10 years+

Slide 40

e-Infrastructure Networking David Salmon

Slide 41

Topics e-Infrastructure funding What has Janet been doing ? Emerging Issues

Slide 42

Janet e-Infrastructure networking 26MWith HEFCE Breakdown 10MContribution to Janet6 backbone 12M Network provision within classic Janet community, including key locations which may need additional fibre access 4M Industry connection

Slide 43

e-Infrastructure programme support CTO Bob Day on ELC Paul Lewis secondment to BIS programme Office 8 months STFC Michael Wilson secondment just started funded from Janets e-Infrastructure allocation

Slide 44

e-Infrastructure Strategic Sites Potential fibre / capacity requirements Work with organisations to understand requirements Build cases for provision If approved, integrate provision with deployment of Janet6

Slide 45

Patterns of use Data movement working patterns How much data volume ? Beware Bytes vs. bits ! How quickly ? How often ? Where to ? National & International Network engineers need bit-rates ! Working with research communities to understand emerging/evolving requirements 45

Slide 46

Norwich Bioinformatics Cluster The Genome Analysis Centre - TGAC The Sainsbury Laboratory John Innes Centre Institute for Food Research University of East Anglia Norwich Research Park

Slide 47

Hinxton Genome Campus European Molecular Biology Laboratory - European Bioinformatics Institute The Sanger Centre Babraham Institute

Slide 48

Exeter & locale Met Office University of Exeter Weather & Climate research Monsoon HPC NERC & M.O.

Slide 49

Broaden scope of Janets fibre infrastructure Complementary & integrated with Janet6

Slide 50

e-Infrastructure projects funded HPC 5 regional consortia National facility Hector / Archer Dirac Particle physics NERC STFC RAL Daresbury will engage to discuss requirements & evolution

Slide 51

Outcome Outline proposals made Approvals discussions with Research Councils Must converge soon to optimise Janet6 fibre layout Some budget remains Likely to be a further proposals Academic & Industry locations ?

Slide 52

Some implications 52

Slide 53

Throughput challenges Infrastructures can deliver very high capacities Realising high-throughput end-to-end will take some effort Cant just take systems out of the box and expect to get full line-rate data-transfers This needs to be communicated & understood more widely Janet & NREN community needs to work more closely with research communities to understand and solve issues. 53

Slide 54

Achievable Throughput What is realistic ? Challenge: 1Gbit/s, 24hr ~= 10TBytes Holistic approach End systems - Hardware & Software - Both ends ! Local networks Wide area networks Highlight successful examples Technical case studies Janet role ? Co-ordination/collaboration Advice & troubleshooting support ?

Slide 55

Network structure implications ? Large research communities International/global overlays or private networks Particle physics - LHC OPN (LHCONE) Radio Astronomy eVLBI, LOFAR, (SKA...) Future Bioinformatics ?? ELIXIR Campus implications ? point to point services (lightpaths) more complex multi-homing routing/switching issues

Slide 56

Industry ? Existing Janet policy Supporting collaborative R&D Business and Community Engagement (BCE) Evolution - more needed ? e-Infrastructure Open and accessible Janet

Slide 57

Connecting Industry meaning ? Network connections ? Access to community systems & resources connected to Janet ? Access to Janet community knowledge & expertise ? Mix of all three... BIS & HPC Consortia views ?

Slide 58

Issues & futures Good planning & dialogue between Janet & GridPP continue ! understand evolving requirements address these as they emerge Involved in LHCOPN & LHCONE meetings Rob Evans (Janet) Technical input Balancing UK requirements, GEANT & other NRENs in Europe, and US & global NRENs

Slide 59

GidPP networking archaeology....(2000)

Slide 60

Conclusion Janet6 on-track Very capable strategic platform supporting UK Research and Education for 10 years+ e-Infrastructure Approvals soon hopefully Challenge of delivery & deployment Industry ? Using the network Achieving higher data throughput

Slide 61

?

Slide 62

Janet Aurora

Slide 63

Last presented at CEF in 2010 Very good results produced High profile papers at conferences But, funding issues have held back full development

Slide 64

Context - Funding History e-Science and JISC Funding 4.35M 2007-2012 Fibre infrastructure only Fibre access (single pair) ILA rental space for researchers equipment 1 rack basic telecomms for management access Tax lit fibres are subject to business rates in UK NOT Research equipment Staff effort

Slide 65

JANET Aurora Intermediate equipment co-location point University / JANET access point Fibre spans International Lightpath access to other NRENs via JANET & GEANT JANET Lightpath access to other UK Locations University of Cambridge University of Essex UCL University of Southampton Telehouse London 550km of fibre-pairs VM Brentford PoP VM Camberley PoP VM Crawley Court PoP VM Chelmsford PoP VM Enfield PoP VM Ipswich PoP VM Bury St Edmonds PoP 31km 52km 58km 43km CAL483331-1 (28km) 30km 55km 57km 54km 50km 72km

Slide 66

Current position Janet will maintain Aurora in current form until Sept/Oct 2013 Aurora / Dark Fibre facility WILL need to evolve over next 12-18 months Can not extend contracts for current infrastructure any further (procurement law) The Essex research group is moving to Bristol (Prof. Dimitra Simeonidou) Research Council (EPSRC) funding for a Dark-Fibre Facility is still being considered Equipment and staff effort to exploit dark-fibre as a test platform

Slide 67

e-Infrastructure New funding form UK government 100M+ into HPC National Regional 26M to Janet to continue and extend capability Extend Janet6 to key locations Biosciences Climate and Weather Research HPC Dark-Fibre facility / Aurora MAY also be possible Exploring with Research Councils

Slide 68

Recent Results ECOC next week Dimitra Simeonidou & colleagues will present interesting new results Collaborations with NPL continue Ultra-stable lasers NPL and UCL Optical comb transfer NPL and Southampton Janet in discussion with NPL about participation in Aurora NPL has independent funding, and plans to deploy fibre for comb- transfers Following slides from Radan Slavik UFE, Prague ORC, Southampton

Slide 69

JANET Aurora Dark Fiber Link Intermediate equipment co-location point University / JANET access point Fibre spans International Lightpath access to other NRENs via JANET & GEANT JANET Lightpath access to other UK Locations University of Cambridge University of Essex Aston University UCL University of Southampton Telehouse London 850km of fibre-pairs University of Southampton Telehouse London 40 km 60 km 40 km Southampton to London part of the link: 200 km, dispersion-compensated, 6 double in-line flat-gain CLA EDFAs with maximum input/output powers of -5/15 dBm. Experiments: (A) Southampton-London roundtrip for phase regeneration; (B) Southampton-Essex roundtrip for optical wavelength conversion.

Slide 70

Phase and amplitude all-optical regenerator All-optical wavelength conversion Taken from University of Southampton, U.K. (V. Rancano; results under progress) CLA amplifiers used Taken from University of Southampton, U.K. (R. Slavk et al, Nature Photonics 2010 and JLT 2011)