network operations centers training programs quilt noc workshop panel discussion: cenic indiana...

Network Operations CentersTraining Programs

Quilt NOC Workshop Panel Discussion: CENIC

Indiana University/Global NOC WiscNet

Pacific Northwest GigaPopOctober 3, 2007



October 3, 2007

NOC Training ProgramsWiscNetKika Barr

9/11/2007 Network Operations CentersTraining Programs

Who We Support

• 53 Higher Education Institutions • Over 300 k12s • 12 CESAS – Cooperative Education Service Agency• 18 Technical Colleges• 13Library Systems• 36 Government Institutions • 1 museum• I hospital• BOREAS-Net

WiscNet


Staff Workload

• 100-200 tickets per month

WiscNet


Our Staff

• Size– 2 full time tech support specialists– 1 ¾ time tech support specialist– 1 Network Architect

• Mix– The Tech Support Specialist are strong generalists– Yet, the staff tend to divide itself on the level of the

expertise by service (Jack of all trades, master of some)

WiscNet


Recent Training Content

• Training offered to staff in the last 6 months – Seminars about Security– Fiber training– Fiber Optics training– Firewall training– Cisco related training– Juniper training

WiscNet


Training Resources

• Recent training was designed and provided by;

– Vendors– Cisco– Packeteer

– Engineers on staff

WiscNet


Training Budget

• We allocate training budget from the organization budget

– Try to stay local to save on expenses– Take advantage of vendor training

opportunities

WiscNet


Training Logistics

• Methodology for scheduling training– When training is out of the office, we normally just

send 1 tech support member– When training is in house, our NOC is made

aware that staff is on training. Tech Support specialist have their laptops on with IM so NOC can reach us if there is an emergency

– Voice mail to Tech Support goes to [email protected] mail list

• Where do you train?– In house using conference rooms. No training

rooms available– travel

WiscNet


Lessons Learned

• If you don’t use it, you lose it

• Consistency and relevance

• Delicate balance between overdoing it and not doing it enough

WiscNet


Training Program Example• WiscNet Support Staff if Tier 2

– Training as needed– Have been discussing with engineers about internal

training and a more structured planned• Competencies:

– Troubleshooting– Ticketing– Tools– Customer Services– Service Expertise– Optical networks around the corner

WiscNet


Demonstrated Competencies

• Excellent phone skills

• Excellent understanding of WiscNet services

• Technical proficiency in several areas of technology – router configuration, DNS, troubleshooting, firewall configuration, WAN infrastructure, and Optical networks

WiscNet


Keeping the Program Current

• Challenge for us because of time constraints

• Hoping to work with WiscNet engineers to design a more predictable training train

WiscNet


October 3, 2007

NOC Training ProgramsWiscNet

Kevin Schmidtke


Who We Support

• Number and type of sectors your organization supports

– UW Madison Campus – network backbone, wireless

– WiscNet – statewide educational network– BOREAS Net – regional optical network,

research focused

WiscNet


Staff Workload

• # of incidents opened per month

– 300 Clarify cases opened on average• WiscNet cases – 68%• UW Madison cases – 29%• BOREAS Net cases – 2%

WiscNet


Our Staff

• Size of Staff– 10 staff

• 7 Professional Senior• 3 Professional Junior• 1 Process & Procedure coordinator

• Staff Mix– All Junior staff scheduled with a Senior staff

member. Seniors perform in a lead capacity.

WiscNet


Content

• Examples of training offered to your staff in the last 6 months – Infinera DNA application training– Infinera hardware configuration training– Modified WiscNet process and procedures

WiscNet


Training Resources

• Who designed the content and provided the training for your staff in the last 6 months

– Most training was developed in-house– Most training was delivered by subject matter

experts on staff

WiscNet


Funding Sources

• Two Sources– New network support contracts are

expected to deliver training to all staff as part of the service rollout and acceptance

– Training budget allocation for each staff. Traditionally this is used for specific professional development

WiscNet


Training Logistics

• Scheduling training– Training is typically presented in a

minimum of two sessions to ensure all staff can attend. Rarely do we have an all-staff training session

• Training facility– Several training rooms are available in

data center complex

WiscNet


Training Program Example

• Knowledge Database Repository– All formal training results in an article or a

series of articles– Many departments have instances of the

knowledge database and articles can be shared from one instance to another

– Only the originating instance administrator can modify the content of the article

WiscNet



• Knowledge Database instance administrator– Creates all new knowledge database articles

or recruits a subject matter expert for content– Reviews all articles originating from another

department– Is responsible for making all NOC staff aware

of new process or procedure– Ensures all staff review and accept

responsibility for content of articles

WiscNet



• Knowledge Database Lifecycle– All articles start out in draft form– All articles are set to team review– All staff members ‘sign off’ on content of the

article– All articles are reviewed at a minimum of 6

months for accuracy and relevance– Articles are reviewed by Instance

Administrator and/or subject matter expert

WiscNet


Lessons Learned

• Transition from hardcopy to online knowledge database required trust in materials.

• All articles are keyword searchable. Review failed searches to determine better ways to define articles

• Keep a copy of knowledge database on stand alone server on separate UPS

WiscNet


October 3, 2007

NOC Training ProgramsPacific Northwest GigaPop

Linda Hornung


Who We Support

• PNWGP

• University of Washington Campus

• University of Washington Medical Centers

• Washington State K-20 network

• TransitRail (4/2007)

PNWGP


Staff Workload

• Our NOC opens 900-1000 tickets per month across the various customer bases we handle

PNWGP


Our Staff

• Full time staff in the NOC and Computer Operations, who provide 2nd and 3rd shift coverage. – Four tier 3 – Seven tier 2 – Five tier 2 for off-shifts.

• New NOC students – 6 students work 4pm-8pm M-F, and 9am-6pm Sat/Sun. Considered tier 1.

PNWGP


Recent Training Content

• Several of our staff have been to Cisco router classes and Cisco ONS classes.

• Several others attended Juniper ONS classes provided on the UW campus.

• Professional development courses such as leadership, communication, project management, team building.

• Many presentations by Network Engineering and Network Architecture teams on new services and network design that have recently been implemented.

PNWGP


Training Budget

• No defined budget, but this past year has seen a tremendous increase in training opportunities!

• We have been able to have training from outside companies, our vendors, the UW Professional Development program, and many in-house experts.

PNWGP


Training Logistics

• Because of the different shifts and the need to provide continuous coverage, we will always offer at least two different training sessions for NOC and other interested staff.

• Usually train in local conference rooms.

• Vendor classes have been held in larger facilities on campus. Always in well-connected facilities.

PNWGP


Lessons Learned

• The staff that usually does our NOC training must have taken a Train the Trainer course.

• Have defined training goals.

• Use mixed media – don’t just talk.

• Engage the audience through occasional questions or scenarios.

• Provide take-away reference material.

PNWGP



• Tier 1 staff: the focus is on escalation within the NOC and being able to gather appropriate initial information.

• Competencies:– Troubleshooting– Ticketing– Tools

PNWGP


Training Topics for Tier 1 Staff

• Architecture overview

• Services overview

• Customer information

• Monitoring

• Opening trouble tickets

• Handling and issuing outages

• Escalation

PNWGP



• Basic phone skills (answer, hold, transfer)

• Searching tickets and email for previous information

• Using tools to find out information on equipment location, customer contacts, circuit ID, etc.

PNWGP


Keeping the Program Current

• Continuous project to keep the content updated and accurate

• Reliance on an internal wiki for procedure and information

PNWGP


October 3, 2007

NOC Training ProgramsCENIC

Cindy Abercrombie


Who We Support

– CalREN Community (-approx site count)

• California Community Colleges - 112• University of California - 11• California State University - 25• California K-12 System - 74• California Private Institutions - 9• Associates and sites outside California - 11

– NLR• Layer 1 Services• Support equipment / deploy services across 250 sites

– Pwave, CIIX, TransitRail, AARNet• Support equipment and services

•

CENIC


Staff Workload

• CalREN Monthly Ticket Stats– Avg 670 tickets opened

• Community requests for support and outages• Deployments, upgrades, maintenance

– Avg 375 support tickets resolved • Community requests for support and outages

• NLR Monthly Ticket Stats– Avg 105 tickets resolved

• Deployments, outages, maintenance

CENIC


Our Staff

• 15 NOC Engineers– 1/3 of our team have > 5yrs. Network support

experience– 2/3 of our team have 2-3 years combination LAN/WAN

and end-user technical support, with 1-2 years large-scale network support

• Other Network Operations Support Resources– Our senior engineering team works with us daily on all

types of request– Our Sys Admin team responds to NOC needs quickly

and provides support to our community too

CENIC


Content

• Examples of Training Provided– Cisco 15454 MSTP Test and Turn-up– Configuring BGP on Cisco routers– Cisco CRS-1 Essentials– DNS in Practice

CENIC


Training Resources

• Cisco

• Global Knowledge

• CENIC Senior Engineers

• Outside Specialty Organization

CENIC


Training Funding & Logistics

• Methodology for scheduling training– 2 training sessions on same topic,

sometimes weeks apart or months apart

– Reduce upgrade, maintenance and deployment activities

– Reduce NOC coverage and include senior engineering resources

– Purchase bulk training hours and use Cisco Learning Credits

CENIC


Lessons Learned

• Discussing training opportunities during recruiting process

• Coordinate training with at least two people at a time, the more the better

• Purchasing bulk training hours is more cost effective than a couple of courses at a time

• If course feedback for a particular trainer is good, request the same trainer for future courses

• Make training and professional development a regular part of communications and schedules

CENIC


Day1 Training Program– Start of a Shift– Tools– Process– Escalations– Schedule Maintenance– Outages– Announcements– Contact Information– Where– Updates and requests for info

CENIC


Non-Technical Training Topics

• High visibility sites• Education acronyms• Our role in other network support organizations• Preparing turn-over work for the next shift• Standard signatures and greetings• How we do lunch• Program your cell phone• How to report you are delayed or absent

CENIC



• Using tools to find out information on equipment location, customer contacts, circuit ID

• Searching tickets and email for previous information

• Document and articulate the known facts of the current issue

• Know how to use our vendors’ processes

CENIC


Keeping Network Operations

• Everyone owns network operations

• Sharing is caring = keep wiki current, send email

• Ask questions to understand b/c there’s too much to memorize

CENIC


October 3, 2007

NOC Training ProgramsIU/Global NOC

Steve Peck


Who We Support

• 17; Campuses, University System, StateNet, GigaPoP, Regional Aggregation point, National Backbones, International Connections (total includes separate networks supported by the national backbones)

IU/Global NOC


Staff Workload

• 906 incidents a month

IU/Global NOC


Our Staff

• Size of Staff– 25 Service Desk– 17 Network Engineering– 7 Systems Engineering

• Staff Mix - Service Desk• Shift Supervisors (Day & Night shifts)• Senior Technicians (at least one on every shift)• Technicians (including hourlys)• Off Front-Line support personnel (5 people)

IU/Global NOC


Our Staff

Staff Mix Network Engineering

Network Engineering Team- Day-to-day problem assessment / resolution, - After Hours on-call rotation.

Network Planning Team- Longer term network planning tasks.

Systems EngineeringSoftware DevelopersSystem Administrators

IU/Global NOC


Content

• Service Desk– New Internet2 Network– Connecticut Education Network– Automated reporting tool– IU Core network transition to GRNOC Eng– Indiana GigaPoP changes– CIC OmniPoP follow up– NLR L2 VLAN / Circuit determination– New database circuit schema

IU/Global NOC


Content

Engineering• Internal training: Ex: Ciena CoreDirector training from

resident in-group expert.• Training for new staff as necessary: Ex: commercial

Cisco BGP training for a new Network Engineer.Systems Engineering• One-on-one training of new staff• Ad-hoc training from in-group experts on various

technologies and systems.

IU/Global NOC


Training Resources

Service Desk• Senior staff picks topics and focus, utilize

internal expertise to provide and design the content

Engineering• Internal formal and informal training sessions by

in-house experts.• Commercial training for individual engineers as

needed.

IU/Global NOC


Training Resources

• Methodology for scheduling training

• Training Facilities

Service Desk

• Definitely a challenge! Once a month from 4:00 pm to 7:00 pm; two techs man the SD; use conference room; meetings are video recorded;

IU/Global NOC


Lessons Learned

• If you have access to excellent internal training resources within your organization, you *HAVE* to utilize them!

• Must have a clear picture of staff responsibilities to ensure that appropriate training takes place in a prioritized manner.

IU/Global NOC



From I-Light Service Desk Training Session

• There are three main components to what

we call I-Light:

• Layer 1 fiber optic network

• Layer 2/3 network on the fiber

• Copper based circuit network

IU/Global NOC



From I-Light Service Desk Training Session• The I-Light NOC will be monitoring and

supporting TDM border routers at universities, and taking calls from their primary technical contacts. However, calls will sometimes be from end users.

• The dmarc for all of I-Light is the customer facing port on the last piece of equipment I-Light owns. TDM customers are used to IHETS helping troubleshoot beyond that dmarc.

IU/Global NOC



From I-Light Service Desk Training Session

• Most of the universities have Cisco 7206 VXRs, Cisco 2620 XM routers, or possibly 1700 series routers. These routers use several different versions of IOS. Some have QOS, some don't. However, all commands should be uniform.

IU/Global NOC


Panel Contact Information

Cindy AbercromieCENIC

[email protected]

Kika BarrWiscNet

[email protected]

Linda HornungPNWGP

[email protected]

Steve PeckIndiana University/Global NOC

[email protected]

Kevin SchmidtkeWiscNet

[email protected]

network operations centers training programs quilt noc workshop panel discussion: cenic indiana...

Documents

scheduling training

internal training

staff wiscnet slide

corner wiscnet slide

month wiscnet slide

optical networks wiscnet

network architect

tech support member