The University of the South PacificCampus Network Expectations and Challenges

PACNOG 25 - Fiji

WelcomeEdwin SandysIT Services

Agenda

• About USP• LAN & WAN Infrastructure• Systems & Networks Infrastructure• Cloud Services• Challenges• Future

About USP• USP established 1968 to serve 12 member countries.

– Fiji, Samoa, Vanuatu, Solomons, Tonga, Kiribati, Tuvalu, Nauru, Marshalls, Niue, Cooks, Tokelau.

• Unique Regional Scope;– Covering thousands of islands over 33 million square km of ocean– Around 2 million people and hundreds of distinct cultures– 25,000+ students; 5-8% growth p.a. & 1500 Staff

• Connectivity– 26 Active Sites over 12 Countries– Satellite C & Ku Band (16 Sites – 28 Mbps)– Undersea & Terrestrial Fiber (Fiji Sub Sites, Tonga, Marshalls, Vanuatu &

Samoa)– Upstream via Research & Education Network (AARNet)

Connectivity Cont….• Fiji eXchange Point (IXP) Peering (Early 2018)

– Better service access for Students & Staff– Better VPN Access (Work from Home)– Planning Stages: Livestream lectures for Fiji

• Future for IXP – Later Slide!

LAN & WAN Infrastructure

Roads & Bridges

Satellite Platform Revamp• C Band Dishes Installed 1999 - 20 Years Old

– EOL 15 to 20 Years• Regional Dishes

– 7 New Installs: 2 Non Penetrating & 5 Penetrating Mounts– 3 Refurbished plus non penetrating mount dishes

• Fiji Hub– Refurbish current– Build new antenna

• Enhanced IP Satellite System (iDirect)– 32% efficiency gains of current outbound Mbps– Better enhancements (DVBS-2X with ACM)– Enhanced Modem performance (32APSK Modcods)

Fiber Ring Journey• Business Case Approved - 2008

– Fiber Purchased Approved• Trenching Works Completed - 2009• Cabinet Cleanup & Auditing Completed - 2011

– 70+ cabinets– 35 fiber terminals

• Fiber Pulling and Termination Completed - 2013– Documentation Update & As-Builds

• Cabinet Electrical Wiring & Grounding - 2015• Fiber Ring Switches Purchase Approved - 2017• Fiber ring Online & Operational - Oct 2018

Fiber Ring Infrastructure

FJ VSS Core

ICT – CoreVSS Switch 1

Comms – CoreVSS Switch 2

Sup2T Port 1 & 2Te 1/5/4 to Te 2/6/15Te 1/5/5 to Te 2/6/16

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

SOH Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

Library Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

SMT Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

CELT Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

FSTE SPAS Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

Aus Aid Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

Land Management Building

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

FBE SSED Building

Te 1/6/4 toTe 1/0/12

Primary Path

Te 2/6/4 toTe 1/0/12

Primary Path

Te 1/0/11 toTe 1/0/11

Secondary Path

Te 1/0/11 toTe 1/0/11

Secondary Path

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

Statham Campus

CONSOLE

MODE

STAT DUPLX SPEED STACK

SYST ACTV XPS S-PWRUIDCatalyst 3850 12S

1 2 3 4 5 6 7 8 9 10 11 12

MSP CampusTe 2/6/9 to Te 1/0/12

Te 2/6/7 to Te 1/0/12

Fire

Comms Fire Fiber Redirection

Systems & Networks Infrastructure

Layering on equipment!

General Networking• Scale

– 10 / 40 / 100 Gig• Reduced Footprint

– Less racks = Less Power– Smaller rooms– Less cooling

• Wireless Services– Centralized Management– No Cabling No Problems

(Mesh)• 15000 Feet = 4572m

– Over 350 AP’s– 2000+ Connected Users

• Design is Essential– Redundancy & Resiliency– NSRC Engagement

Wireless Dashboard

Design - Wireless

WIRELESS DEVICES

WIRED DEVICES

ALL SERVICES

USERSSSO, Certificates, Policy

Servers

NAC Server, MDM Server, Provisioning Services

Monitoring & Control

Push Services & Policies

Send

Enforce

Management & Monitor

Management & Monitor

Management & Monitor

Use Register & Authenticate

Design – Layered Approach• Reference Point• Endorsed Direction• Easy Equipment Choices• Application Provision

– Business Critical– Faculty / Departmental– IT Services– Development

• QoS Tagging– End to End

AARNet ISP

10 G Fiber10 G Fiber

2 x 40 G Fiber

HA cluster

TFL Fiber SuvaPrimary AARNet Link

TFL Fiber VatuwaqaSecondary AARNet Link

2 x 10 G Fiber

FGT-3200D-Sec

HA Active

FGT-3200D-Pri

HA Passive

10 G Fiber

Data Center Central Core

ISP Peering

NGO Peering

USP Fiber Interconnect

USP Satellite Interconnect

IXP Peering

USP Campus WAN Sites (Trusted)

Federated Peering

USP Affiliated Sites (Semi-Trusted)

ISP Sites (External)Edge BGP Equipment

VPN

BGP

BGP

SS

Catalsyt 9500 Series C9500-48Y4C

RFID

...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

C9500-48Y4CSS

Catalsyt 9500 Series C9500-48Y4C

RFID

...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

C9500-48Y4C

GPS

1PPS 10MHz

CLASS 1 LASER PRODUCTGE2GE1GE0 GPS

BOOT

1PVDM0

LINK

CON GE3AUX

MGMT ETHERNET

CRIT

MAJ

MINTAT

WR

1

0

GE4 GE5

GPS

HDD

BITS

BITS

Cisco ASR 1002

21

3

ASR1002-X

GPS

1PPS 10MHz

CLASS 1 LASER PRODUCTGE2GE1GE0 GPS

BOOT

1PVDM0

LINK

CON GE3AUX

MGMT ETHERNET

CRIT

MAJ

MINTAT

WR

1

0

GE4 GE5

GPS

HDD

BITS

BITS

Cisco ASR 1002

21

3

ASR1002-X10 G Fiber

ETHERNET

USB

CONSOLE

AUX

RE

ONLINE/OFFLINE

SYS OK

PS1PS 0 MX80-48TFAN

LINK LINK LINK LINK

0 1 2 3O

I

O

I

0/ MIC 0

1/4

1/5

1/2

1/3

2/8

2/9

2/6

2/7

1/0/

00/

1

1/3/

103/

11

USP to AARNet Rack Link

Internal BGP Equipment

Data Center ServersCISCO NEXUS N9K-C92160YC-X

53 5451 5249 501 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

BCN

STS

ENV

N9K-C92160YC-XCISCO NEXUS N9K-C92160YC-X

53 5451 5249 501 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

BCN

STS

ENV

N9K-C92160YC-X2 x 40 G Fiber

QoS

Business Critical

Faculty / Departmental

IT Services

Development

Server Hyper Converge Infrastructure (HCI)

• Technology– Smaller– Faster– Compact

• Reduced Footprint– Less power– Less racks– Smaller rooms– Smaller cooling– Better Electrical

Server Room Cleanup

HCI Server Compute

# Technology CPU (>2.0 Ghz) Memory (TB) Storage

1Dell VX Rail G410 (6 Nodes)

5 Nodes x 14 Cores x 2 = 140With HyperThreading = 280 2.5 65 TB Effective

(All Flash)

2

Cisco HyperFlex HX240c(3 Nodes)

2 Nodes x 20 Cores x 2 = 80With HyperThreading = 160 3 120 TB Effective

(All Flash)

3IBM ThinkAgile HX552x(8 Nodes)

7 Nodes x 20 Cores x 2 = 280With HyperThreading = 560 2.8

480 TB Raw(Hybrid)

250 TB Effective

• HCI Node Failover N+1

Cloud Services

Determine services that should be placed in the Cloud.

Student Gmail• On Premise

– Google Cloud Directory Sync

– Users & Groups• Cloud Security

– Anti-SPAM– AV– IPS

• Students– 15 Gig mailbox per user– Unlimited Cloud Storage– 30,000 active users– 450 Terabytes of Data

Backup

# Service Classification Backup Frequency

Recovery Testing

FrequencyRetention

Period

1 Business Critical (includes development) Every 6 Hours Every 3 Months 7 Years

2 Departmental & Faculty Weekly Every 6 Months 3 Years

3 IT Managed Weekly Every 6 Months 3 Years

4 Development None None n/a

Recovery

• 3rd Backup Repository– Cloud storage services.– Amazon AWS / MS Azure /

Google / USP– Cost vs Features vs Ease

• Service Recovery– Spins services on the cloud– Business disaster options– No need for secondary data

center

Challenges

What keep the cogs turning!

Challenges for Team• Documentation

– Maintaining records (meticulous)• Standard Operating Procedures (SOP)

– Process driven– Flow of events

• Automation & Central Management– 4 Systems Engineers– 5 Network Engineers– 3 Infrastructure Techs

• Infrastructure Patching vs Application Updates– Patching security platforms, networking gear, etc.– Updating application code and open source

development.

• Testing Tools– Fiber & UTP– Wireless

Challenges Cont.…• Construction / Landscaping

– Fiber Cuts & Pit Damage– Redesign of contractor plans

• Technical Expertise– Constant training renewal– Recruiting appropriate personnel– Internal recruitment process– Maintaining market value

• Security– Securing people (Trust but Verify)– Securing infrastructure– Securing services

• Weather – No Control!

Bridging the Gaps• NOC Monitoring

– Interns with supervisory staff.– Weekday operations from 7am to 10pm– Weekend operations from 10am to 6pm– Other Hours: Automated Notifications

• Detailed Provisioning– Network Operations Center (NOC) – iCinga– Bandwidth Validation – Perfsonar– Network Provisioning – Cisco Prime– Systems Provisioning – vCenter & Prism– Security Monitoring – Firewall Analyzer

Future

What's next on the roadmap?

Future• Connectivity

– Stateful IPv6 Re-Deployment (2019) – Partially Complete– Regional Fiber Connects (join Fiji eXchange Point)– Eduroam – Under Technical Trials– Additional Satellite Providers– Extend 10 Gig (Office Uplink) – 30% Complete– Reduce Firewalls

• Cloud– Office 365 & Disaster Recovery

• Infrastructure– Small is better so consolidate– Localise Content to remote sites (caching)

• Technology Trends– Keep track on current technologies– Extensive Automation & API Integration

WAN Optimisation

VOIPSecurity VPN

Regional Designs - Server

Regional Designs - Wireless

Thank You & Questions

We do things right, our people stay connected!

Email: Edwin.sandys@usp.ac.fj