presentation available here - ccd charter
TRANSCRIPT
Systems and OperationsSystems and Operations
CCDs PresentationDecember 12, 2007
Jam es BohnsackMike GarciaJason Lowe
Mark BodensteinProject Leader
Mainframe System s Program m ing
Randy Sm ithPeggy Roberts
R ick PolcaroManage r
Facili ties & Laser Printing Services
Moe ArifMary C ronkDoug FlanaganSolom on W elchTom W aldenDavid ShirkKent R ossG len Hoffm an
Mariann C arpenterManage r
System s Adm inistration
Scott SorrentinoMike HeislerJim YangJohn W obusAndrew HeathJavier StrebVacant
Laurie Collinsw orthManage r
Systems Engineering
Bob TaldaDavid Beardsley
Paul ZarnowskiManage r
S torage Services
Michael HojnowskiManage r
Specia l Projects
Brian MessengerAssistant Director
Systems
Jim HowellSenior Technical Lead
Jennifer MooreG ail ShaffG eorge MedlarTodd O lsonVacant
Messaging Services
Rick CochranMichelle MogilLee BrinkDan Bartholom ewJoanne Button
Client Services
Don Mac LeodAssistant DirectorSystem s Services
Jim ConleyJohn RyanBrian W itchey
Juan Salom onTeam Leader
NOC First Shift
Mark A llenJay HowellLillian Isacks
Bryan BenningTeam Leader
NO C Second Shift
Ruth BurroughsCarl MoravecTheresa N orm anMaureen Q uillinan
Chuck Thom asManage r
Production O perations(Evening Shift)
Dan MillerR ich FraboniG reg Marvin
VacantManage r
Production O perations Services
Jam es ReedJohn BeckerBarbara Van E tten
Ken FrostTeam Leader
NO C Third Shift
Jenny SignorTechnical Lead
V icky D eanAssistant Director
O perat ions
Rick MacDonaldDirector
System s and O perations
Services:Services:Server FarmServer FarmSystems Administration Systems Administration Storage FarmStorage FarmEZ-BackupEZ-BackupVMware (In Development)VMware (In Development)
Server FarmServer Farm
Manager: Rick PolcaroStaff: Peggy Roberts Randy Smith
Server FarmServer Farm
The CIT Server Farm provides a secure environment for housing departmental servers. There are more than 500 servers in the facility today. The service includes 24 hour system monitoring and reboot, network connectivity, and uninterruptible power with generator backup. Servers are mounted in CIT provided racks. It is the customer’s responsibility to provide hardware and software maintenance.
Server Farm HistoryServer Farm History The server farm began in CCC in the mid-1980s with a
few Unix systems for data warehousing and research. Our first departmental user was Dining. CISER soon
followed. Major CIT servers into the 1990s
Instruct MachinesEZ-Remote serversPostoffice machinesP2K ServersLibrary Servers
Server farm moved from CCC to Rhodes Hall in 1997.
Server Farm FacilitiesServer Farm Facilities Raised Floor in Rhodes and CCC. Emergency power and generator backups. Standard electrical power consists of 120/208 vac, with 30
amp three prong receptacle. Special requests are directly billed.
Physical security to authorized personnel only, with security cameras.
Basic support, via the NOC, for 24/7 monitoring and basic troubleshooting, rebooting, and system administrator notification.
Server Farm FacilitiesServer Farm Facilities Costs:
– Space is charged by the “U”, an industry standard measurement in a 19” rack – 1.75 vertical inches.
– The monthly server charge = (server “U’s” * $7.60)– There will be a one time installation charge of $118 per
server
The cost for a network connection(s) and NUBB will be charged directly to the user’s University account by CIT NCS.
Server Farm GrowthServer Farm GrowthServer Farm Growth
135153
154155
168179
189
221234
255280
294299
316331
351 338
374378
385
539543
557586
538543
542
456415
482479
0
50100
150200
250
300
350400
450
500550
600
4QFY00
2QFY01
4QFY01
2QFY02
4QFY02
2QFY03
4QFY03
2QFY04
4QFY04
2QFY05
4QFY05
2QFY06
4QFY06
2QFY07
4QFY07
12/3/2007
Serv
er C
ount
0
50100
150200
250
300
350400
450
500550
600
CIT Servers Customer Servers Total
456
130
Server Farm CustomersServer Farm Customers Distribution (number of servers):
CIT Owned: 456Lab of O: 17Library: 29Comp Sci: 14CCE: 10Others: 60 (representing 29 depts)================================Total: 586 (12/07)
QuestionsQuestions
Information / Contact: Server Farm:
http://www.cit.cornell.edu/services/serverfarm [email protected] ???
Systems Administration SupportSystems Administration Support
Manager: Mariann CarpenterStaff: Muhammad (Moe) Arif
Mary Cronk Doug Flanagan Glen Hoffman Kent Ross David Shirk Thomas Walden
Solomon Welch
Systems SupportSystems Support Systems Support provides systems administration and
systems programming support for projects that span the 400+ servers and 75 terabytes of storage located in our machine rooms.
The Systems Administration team provides the hardware and operating system support for CIT’s servers and storage assets.
Systems Engineering provides and enhances tools for our production servers and services, such as NetVigil, as well as specific support for the DNS/DHCP service and its ancillary systems.
Client Systems ServicesClient Systems Services Systems Administration Services
Quote, OS Install, OS Maintenance, Backup Configuration, Userid Maintenance, Application setup assistance, Retirement
The following operating systems are currently supported: SUN Solaris IBM AIX (no new installations)Windows 2000 (no new installations) Windows 2003 (standard and enterprise editions) RedHat Linux Enterprise AS 4.0
Installation into the Server farm facility. Monitoring with 24/7 Support as required.
Client Systems ActivitiesClient Systems Activities Routine Administration. Coordination of hardware maintenance contracts. Lifecycle management of servers, working with our
customers. Automation projects to leverage our staff investment.
Systems Administration DemographicsSystems Administration Demographics We support the OS on approximately 400+ servers. Distribution by Operating System
74% Solaris 1% AIX25% Wintel, Linux
Storage Farm Service
CIT’s Storage Farm service provides storage capacity, connectivity, and management services to servers housed in CIT’s Server Farm and managed by CIT’s Systems Administration Support group.
Storage Farm Staffing
• Manager of Storage Services– Paul Zarnowski
• Manager of Systems Administration– Mariann Carpenter
• Storage Engineers– David Shirk– Kent Ross
Storage Farm Historical Growth
Storage Network Ports
0
100
200
300
400
500
Jul-05 Jul-06 Jul-07
Storage (TB)
0
20
40
60
80
100
Jul-05 Jul-06 Jul-07
Storage Farm: Efficiencies
• Centralized Management– Small set of devices to support & maintain– Common management tools– Fewer resource baskets to manage
• Centralized Procurement Process– Fewer purchases (hardware, maintenance, etc)– Streamlined process
• Higher Storage Utilization
Storage Farm: Advantages
• Easy to Use:– Ease of storage allocation– Faster storage allocation– Maintenance handled by Storage Farm staff– Technology upgrades
• Financial Advantages:– No up-front capital expense or depreciation– Grow storage as needed (pay as you go)– Risk reduction– Lower overall costs (considering all components)
Storage Farm: Components
• Storage Connectivity
• Storage Tiers
• Storage Management
Storage Farm:Connectivity Options
Fibre Channel Storage Area Network (FC-SAN)– Single-attached, to one fabric (red or green)
• Redundancy for Storage Array connectivity only– Dual-attached, for higher availability (red AND green)
• Redundancy for both Array and Server connectivity
SANFabric 1
SANFabric 2
HBA1HBA2 Storage
Array
Ctlr 1Ctlr 2
Storage Farm Services: Storage Tiers
Tier 1 • Highest performance & availability• Only tier providing mainframe connectivity• Examples: High-usage operational databases
Tier 2 • Performance, cost & availability all have some level of importance• Examples: average usage databases & applications
Tier 3 • Lowest cost of online storage• Performance & high availability less important• Examples: Development & Test systems; moderate-activity file servers.
Storage Farm Services:Management Services
• Storage Network connection setup• LUN* allocations & expansions• LUN tier migration• Performance & Capacity planning• Storage device firmware upgrades• Maintenance procedures• 24x7 Health monitoring of storage subsystem
(out-board from server’s FC HBA)• Troubleshooting of storage hardware & software• Procurement• Life-cycle upgrades
*LUN = Logical Unit Number (aka logical disk)
Recent Storage RFP: Goals
• Improved Storage Management– Storage hot spots– LUN resizing effort– Reporting
• Cheaper Storage Connectivity– iSCSI* support
• Lower Storage Costs– Maintenance– “Right-placing” data– Decrease “head room” costs
*iSCSI = SCSI over TCP/IP
(Spring, 2007)
Storage RFP: Process
• Storage RFP Review Committee:– Paul Zarnowski, David Shirk, Kent Ross, Don MacLeod,
Tony Damiani, Ken Friedman, Mike Hojnowski, Brian Messenger
• 15 Vendor Proposals reviewed• 8 On-site Vendor Presentations• 4 Finalists• Compellent vetted & selected
Storage Center
• Stable performance– All I/O spread & balanced across many disk spindles– No more hot spots
• Reduce capital expenditures– Thin provisioning– Automated Tiered Storage
• Supports any open-systems server without agents– Connects via FC, iSCSI or both– Scales from 500 GB to over 300 TB
• Cut operating expenses– Disk virtualization simplifies administration– Eliminate the need for 3rd party software
• Enhanced data availability– Unlimited snapshots without full copies– Replication flexibility for low-cost DR
• Reduce server costs– Reliable boot from SAN
Scalable Enterprise-class SAN solutionwith block-level intelligence
Storage Farm Roadmap
• SAN Fabric simplification– Elimination of older Brocade switches
• iSCSI connectivity– Initially deploy within Server Farm environment
• Dual path FC / iSCSI connectivity– Windows initially
• Improved performance– Higher spindle count
• Snapshot capabilities
Potential Future EnhancementsRecovery Options
Distance
Local(Rhodes)
Campus(CCC)
Wide Area(Weill)
Snapshot Synchronous Asynchronous
Technology
Data Instant Replay Remote Instant Replay – Synchronous
Remote Instant Replay – Asynchronous
Storage FarmPotential Future Enhancements
• Tier 4 storage service– Very large capacity– Low use– Low management– Long-term commitment– Low cost
• Network-Attached Storage (NAS) services– File-level storage (e.g., CIFS, NFS)
• iSCSI storage for non-managed servers
Storage Farm: Rates
Effective:July, 2007
Storage Management Fee (per system) $64 / month
SAN Connection Fee(per FC connection) $52 / month
Tier 1 storage (mainframe) $3.43 / GB-mo
Tier 2 storage $0.49 / GB-mo
Tier 3 storage $0.29 / GB-mo
Questions ???– Contact:
Paul [email protected]
Flash demo of Automated Tiered Storage and Data Progression:http://www.compellent.com/products/demo/demo_ats.html
EZ-Backup is . . .
A Centralized Backup solution that offers:• Network-based • Automated backups• Automated management of backup data• User-driven restore• Off-site backups (but still on-Campus)
• Same solution used to back up CIT’s server farm
• Available to all Cornell departments
EZ-Backup Staff
• EZ-Backup Support Team– Robert Talda– David Beardsley– Randy Barron– Joanne Button– Michelle Mogil– Ron Seccia– and many others throughout CIT
– Paul Zarnowski (Service Manager)
EZ-Backup Supported Platforms
•Operating Systems:– MacOS
• Leopard support imminent– Windows
• Including Vista– Unix (most flavors)
• Linux (RH, SuSE)• Sun Solaris• AIX• Others
– Netware
•Apps / Databases:– Oracle– Microsoft SQL Server– Microsoft Exchange– Microsoft SharePoint– Others
Available for most platforms in use at Cornell.
EZ-Backup Growth(14 years)
EZ-Backup Recent Changes
• New Client Support– Windows Vista– Macintosh Leopard (imminent)
• New TSM Client Software– New Version 5.4– Older versions upgraded to latest patch levels
(for older OSes)– Recommend upgrade to latest software levels
• Merging of CTC TSM service into EZ-Backup– 2nd tape library will give EZ-Backup two-site capability
EZ-Backup Less-recent Changes
• Dynamic Sub-file backup*– Very useful for backing up over slower network
connections, such as Dial-up, DSL, RoadRunner• Journal-Based Backup*
– Speeds backups for large fileservers w/ low change rate• Ability to delete individual backup files• Include/Exclude Preview Capabilities• Open File Support*• Encryption Enhancements• Server Split• CIT On-Site Solutions support available
*Available on Windows clients only
EZ-Backup Roadmap
• Upgrade EZ-Backup/TSM Storage Subsystem– RFP Goals:– Technology refresh– Lower storage costs– Provide faster restores for servers with high object count– Consider use of low-cost disk storage
and data de-duplication• Training Classes• Off-site Capability (Weill Medical Center)
– Pending funding decision• Enhanced Reporting & Management Tools
Current Rates (per Month; effective July, 2007)
Base Rate: $6.50 per system
Base Rate (>50th system): $4.50 per system
Base Storage included: 6.0 GB
Extra Storage rate < 15GB: $0.60/GB
Extra Storage rate > 15GB: $0.40/GB
Static data rate (>100GB): $0.20/GB
Next Rate Change Due: July, 2008
EZ-Backup Pricing History
Questions ???– Contact:
Paul [email protected]
More information is available at:http://ezbackup.cornell.edu
EZ-Backup
Virtual OS Hosting ProjectVirtual OS Hosting Project
Mike HojnowskiManager, Special Projects
Project DeliverablesProject Deliverables Provide Windows hosts under VMware supported
by S&O Sysadmins. Note that Linux is formally not in scope at this time.
Develop a financial model to support to support the service for CIT internal customers.
Develop process improvements to minimize the effort involved in creating virtual instances.
Implement the system in a way that facilitates Emergency Preparedness.
Document this service.
Current EnvironmentCurrent Environment
Canada
Mexico
Jamaica
Panama Peru
VirtualMachine
Containers
VirtualMachine
Containers
Rhodes Hall CCC
PiratesVirtual
MachineContainers
Prod
Test
Dev
Current EnvironmentCurrent Environment Storage: Compellent via SAN
Thin ProvisioningFast full copy, and Snap copy capabilitiesAutomated Tiered Storage
Production Servers (3)IBM 36502-Socket, 4-Core, 2.66 Ghz24G Ram4 Gbit NICs2 SAN HBAs
Expected capacity: 50-60 Virtual Machines
Project StatusProject Status
Project Plan approved 10/05/07. Project implementation has been repeatedly
delayed, due to the Exchange project. Staff resources are freeing up, and we’re
ramping up our efforts.
TimelineTimeline
Development5/1/07 – 12/31/07 (Lengthened due to resource
constraints, cutting the early adopters phase shorter than originally planned).
Early Adopters1/1/08 – 2/28/08
Full Production3/1/08
IssuesIssues SAN failover
Presently we don’t replicate storage from Rhodes to CCC.
We won’t have full “building failure” protection on go-live.
We will address this with either storage technology, or new features in a coming release of VMware after go-live.
FuturesFutures
Linux (First half 2008) Solaris (First half 2008) Virtual “Co-lo” as a Designated Service?
(FY09?)