swan technology users group · • microsoft teams • for better organization and tracking of our...

SWAN Technology

Users Group

Thursday, March 21st, 10am

RAILS Burr RidgeSteven Schlewitt, SWAN Information Technology & Systems Support Manager

1

Meeting

Agenda

•Updates to the IT Team

Introductions

•Azure/Infrastructure Migration Project Debrief

•Downtime Report

•Volume Email Service

SWAN IT System Updates

•Transit Slip Project

•Update on BLUECloud Mobile Implementation

Other Notable Projects

SWANx2019: The SWAN EXPO

Open Forum Discussions

Updates to the IT Team, SWAN Reorg

Infrastructure Migration: Recap

October 10th – Symphony 3.5.3 upgrade

• Mitigated database slowness from Spectre/Meltdown patching that became apparent in upgraded Red Hat and Azure environment

November 6th/7th – Red Hat Operating System upgrade

• Ensured compatibility with Azure environment along with OS security and stability needed moving forward

January 8th – SendGrid mail service rollout

• Ensured high-volume emails to patrons would not result in blacklisting

• Permitted greater insights into email delivery

January 13th/14th – Production server a migration to Microsoft Azure

• Eliminated dependence on aging server hardware

Progressive dates throughout – VPN and firewall migration to pfSense/Microsoft Azure

• Eliminated dependence on aging firewall hardware

Infrastructure Migration: Migratory Tools

7

• Visio• Timeline Planning• Pinpointed conflicts and parallel

interactions with other project• High-level focus• Easier to demonstrate to other

staff

• OneNote• Shared documentation and

“knowledge base”

• Skype for Business• Necessary for constant

communication

• We also could have used…

• Microsoft Project

• As a more comprehensive project-planning substitute for Visio

• Microsoft SharePoint

• As a substitute for network mapped shares and file sharing

• Microsoft Teams

• For better organization and tracking of our communications


Virtual Machine Handling• Microsoft Virtual Machine Converter 3.0

• For migrations from VMWare environments

• Veeam Backup & Replication• For simple extraction of virtual machines

Performance Metrics & System Tools

• Microsoft Azure Analytics Tools• Graphic metrics of CPU, memory, and IO

consumption

• HTOP (System-Monitor Process-Viewer)

• Open Source, Linux CLI-based comprehensive process monitor

• FIO (Flexible I/O Tester)• Open Source, Linux CLI-based I/O tester

• StatusCake• External uptime monitor with SMS and

email notifications

Volume Email Handling• SendGrid

• External volume email relay service

• Postfix• Open Source mail transfer agent


Membership Communications Out

• MailMan• Membership mass-mailing list for fast

communications

• MailChimp• Membership newsletter

Membership Communications In

• OTRS

• Support ticketing system

Web Presence

• Drupal Support Portal

Infrastructure

Migration: Highlights of

Issues Along

the Way…

• Addition of 19 libraries in parallel with the project

• Posed a significant risk to the success of the library additions

• Overallocated all resources on the team

The New 19 conundrum

• Database slowness following the Phase 3 test server migration & upgrade to RHEL7

• Spectre/Meltdown patching

• Symphony 3.5.3 needed to accommodate

“Why’s it so slow?”

• Azure could not accommodate our email sending volume

Volume email service needed

Infrastructure

Migration:Highlights of

Benefits

Along the Way…

SWAN IT gained new insights into Symphony and the

Oracle database configuration

SWAN IT established many new contacts with

SirsiDynix support and engineers

Through SirsiDynix, most significant issues were

resolved early (prior to go-live)

SWAN departments/teams came together to ensure

the success of the project

Improvements were made

to several SWAN services

Volume email service was

implemented

Symphony reports were audited and

consolidated

System cloning process was refined

System backup process was refined

12

Infrastructure Migration: Go-LiveEvening Prior – Sunday, January 13th

• Migration started at 6pm, final sync and config completed by 8pm

• All testing proved successful

• Reports were resumed, overnight maintenance began as usual (no delays)

Infrastructure Migration: Go-Live

Morning of Monday January 14th

• High-volume logins began just before 9am

• System became unstable

• CPU load spiked beyond recovery

• System went offline

13

14

Infrastructure Migration: Go-LiveWhat We Determined

• At full bore, SIP2 services consumed all memory resources.

• Approximately 150MB per active SIP session

• SWAN utilizes over 200 simultaneous SIP sessions on a busy day

• Could be noted by checking SWAP utilization

• More hardware memory was allocated.

• 32GB memory to 64GB memory

• Early versions of the “safety net” scripts were developed to prevent overconsumption of resources from SIP, Web Services, z39.50.

Infrastructure

Migration: Issue

Troubleshooting

Symphony server outages / SIP2 outages on January 14th, 15th

• Due to overconsumption of resources from SIP2 services

• RESOLVED Jan 15th by increasing system resources, creating “safety net” scripts, and

limiting max simultaneous SIP2 connections

Unexpected brief network timeouts/VPN outages, network slowness

• Due to overzealous VPN renegotiation in-hours, packet rewriting configuration

• RESOLVED after-hours in Jan by reconfiguring renegotiation of tunnels and packet MSS-

clamping setting

Envisionware PC Reservation reconfigurations

• Due to lack of hostname support in legacy versions of PC Reservation

• RESOLVED Jan 16th through manual follow-up with impacted libraries

EDI Issues

• *Unrelated to Azure migration, due to fiscal rollover (which occurred around the same

time)

MobileCirc logins

• *Unrelated to Azure migration, due to high volume offline user delinquency file

• RESOLVED Feb 25th through temporary reduction of offline user delinquency file

Infrastructure Migration: Downtime Tracking, Dec

1st through Mar 5th

Enterprise OPAC Server

Uptime 99.4

All Downtime

(P/U) < 1%

WorkFlows / Symphony Server

Uptime 99.6

All Downtime

(P/U) < 1%

*Specific outage data noted in Quarterly packet

Infrastructure

Migration:

Notable

Benefits

• Symphony server data is cached to Solid State disks, allowing the fastest-possible data searching and record reading operations

99% read-caching of data

• Reports are running significantly faster

• Overnight reports completing within allotted window

• Sunday morning deletions completing prior to 6am

• Morning report schedules are largely running on-time

On-schedule Symphony reports

• Reduced impact from Web Services issuesGreater overall

stability

Various automation and financial

benefits unnoted

Infrastructure Migration: Volume Email

Service

30-day stats as of March 6th

Emails Routed: 442,132

Delivered Successfully: 438,632 (99.21%)

Emails Bounced: 529 (.12%) – these are forwarded to your aliased email addresses

Emails reported as spam: 62 (0.01%)

Remainder are currently active/pending delivery

Troubleshooting unreceived emails

Check spam/junk folders

Check email rules or forwarding

Outlook.com/Live/Hotmail users – check “Clutter” and “Other” folder options in Focused

Inbox

Delivery Transit

Label Project

• Web Services-based transit labels

• Generated through SWAN support site

• Currently in limited pilot at several SWAN libraries

• Green Hills (GHS)

• St. Charles (SCD)

• Melrose Park (MPS)

• All libraries welcome to participate

• https://support.swanlibraries.net/help/transit-label-generator

BLUEcloud Mobile

20

Configuration

Questionnaire

Configuration Questionnaire

Graphics Package

App Setup

Quality Control

Testing Go Live

• Completed:• Config Questionnaire, January – February

• Graphics Package, February – March

• Current Stage:• App Setup, March - ?

• Outsourced to Solus, app developer

• Encountered issues with eRC integration

• Upcoming:• Quality Control

• Requires SirsiDynix review

• Testing• SWAN library review and pilot

• Go Live• Anticipated May 2019

SWANx2019: The SWAN

EXPO

• Save the Date: August 16th, 2019

• Location: Moraine Valley Community College

• How does your library enhance the community?

• Now accepting session proposals

• Showcase your library’s innovative methods of serving the community

• Could feature technology or implementation approaches unique to your library

• Also seeking keynote speaker suggestions

Open Forum

Discussion:

Agenda

Topics

• Are they working properly?

Experience with mobile app platforms

• Preferred PM tools?

How do you manage IT projects?

SNMP monitoring tools

Open Forum

Discussion:

Library

Projects

Future RFID Implementations?

New Installations of Self-Check or AMH?

Upgrade to Windows 10?

Upcoming

SWAN Technology

Events

Thursday, April 25th

Location TBD

https://www.librarylearning.info/events/?eventID=27858

SWAN RFID Committee

Thursday September 19th

Location TBD

https://www.librarylearning.info/events/?eventID=27843

SWAN Technology

Users Group

How to

Contact Us

• SWAN Support

• Email: [email protected]

• Phone: 844-SWANLIB (ext. 0)

• Steven Schlewitt, IT & Systems Support Manager


• Phone: 630.326.5993

• Ian Nosek, Systems Administrator


• Phone: 630.326.5887

• Rudy Host, Systems Engineer


• Phone: 630.326.5775

• Michael Szarmach, Systems Administrator


• Phone: 630.326.8929

• Vickie Totton, Systems & Support Consultant


• Phone: 630.326.8714

• Dave Pacin, Systems Administrator


• Phone: 630.326.8360

swan technology users group · • microsoft teams • for better organization and tracking of our...

Documents