20th October 2003 Hepix Vancouver - Oxford Site Report

1

Oxford University Particle Physics

Site Report

Pete Gronbech

Systems Manager

20th October 2003 Hepix Vancouver - Oxford Site Report

2

20th October 2003 Hepix Vancouver - Oxford Site Report

3

Central Physics Computing Services

E-Mail hubs In last year 2.7M messages were relayed (78GB), 0.8M from Physics systems. A

further, 2.8M rejected as spam. Last month 345K rejected, 208K delivered. Anti-virus and anti-spam measures increasingly important in email hubs. Some spam

inevitably leaks through and clients need to deal with this in a more intelligent way.

Windows Terminal Servers Use is still increasing. Retired NT4 based service, Now Win2k and 2003. Introduced an 8 CPU server (TermservMP) . Much more powerful system but still

awaiting updated versions of some applications which will run properly on OS.

Web / Database New web server (Windows 2003) in service. Some initial problems with migrating the

permissions from the old service. New web applications for lecture lists, Computer inventory Other databases for admissions and finals

Exchange Servers Running two new servers using Exchange 2003 running on Windows server 2003.

Default for new accounts. Much better Web interface, support for mobile devices and for tunnelling through firewalls. Existing mailboxes will be migrated soon.

20th October 2003 Hepix Vancouver - Oxford Site Report

4

20th October 2003 Hepix Vancouver - Oxford Site Report

5

Particle Physics Strategy The Server / Desktop Divide

Win 2K PC

Linux System

Des

ktop

sS

erve

rs

General Purpose Unix

Server

Group DAQ

Systems

Mail Server

Web Server

Windows File

Server

Win 2K PC

Win 2K PC

Win XP PC

Approx 200 Windows 2000 Desktop PC’s with Exceed used to access central Linux systems

20th October 2003 Hepix Vancouver - Oxford Site Report

6

Windows Status

Migration to Windows 2000 domain nearly complete for PP users and their computers.

Windows XP pro is default OS for new desktops and laptops. We now have to expect routine reboots of desktops to apply

security patches. Give notice whenever possible.

Grant year

Windows Desktops

Installed

Minimum

Spec

Maximum

Spec

98/99 25 P2/350 P3/450

99/00 34 P3/450 P3/650

00/01 22 P3/733 P3/866

01/02 78 P3/1000 P4/1800

02/03 49 P4/2.0GHz P4/2.6GHz

20th October 2003 Hepix Vancouver - Oxford Site Report

7

Migration to Linux

Central Unix systems are Linux based Red Hat Linux 7.3 is becoming the standard Treat Linux as just another Unix and hence

a server OS to be managed centrally. Wish to avoid badly managed desktop PC’s

running Linux. Linux based file server (April 2002) Digital Unix and VMS services were closed in

August 2002 General purpose Linux server installed August

2002 Small batch farm installed Feb 2003

20th October 2003 Hepix Vancouver - Oxford Site Report

9

pplx1 morpheus pplxfs1 pplxgen pplx21Gb/s

ppcresst1 ppcresst2

ppatlas1 atlassbc

ppminos1 ppminos2

grid tbwn01 pptb01 pptb02

Grid Development

pplx3(SNO)

ppnt117(HARP)

CDF

minos DAQ

Atlas DAQ

cresst DAQ

General Purpose Systems

tblcfg se ce

RH7.3

Fermi7.3.1

RH7.3

RH7.3

RH7.3

RH7.3

RH7.3

RH7.3

RH7.3

RH7.3

RH6.2

RH7.1

RH7.3

RH6.2

RH6.2

RH7.3

RH7.3

RH7.3

RH7.3

Fermi7.3.1

PBS Batch Farm

4*Dual 2.4GHz systems

RH7.3

RH7.3

RH7.3

RH7.3

Autumn 2002

4*Dual 2.4GHz systems

RH7.3

RH7.3

RH7.3

RH7.3

Autumn 2003

matrix

7.3.17.3.1

7.3.1

7.3.17.3.17.3.1

General Purpose Linux

pplx2Dual 450MHz

Pentium II1024MB RAM

(1999)

Early Linux Systems

pplx1P4 Xeon 2.4GHz

2GB RAM(1998)

pplx3Dual 800MHz

Pentium II512MB RAM

(2000)

CDF group systemruns Fermi 7.3.1

SNO group systemruns Red Hat 6.2

20th October 2003 Hepix Vancouver - Oxford Site Report

12

The new (April 2002) Linux File Server: pplxfs18*146GB SCSI disks

Dual 1GHz PIII, 1GB RAM

20th October 2003 Hepix Vancouver - Oxford Site Report

13

General Purpose Linux Server : pplxgen

pplxgen is a Dual 2.2GHz Pentium 4 Xeon based system with 2GB ram. It is running Red Hat 7.3It was brought on line at the end of August 2002 to share the load with pplx2 as users migrated off al1 (the Digital Unix Server)

20th October 2003 Hepix Vancouver - Oxford Site Report

14

PP batch farm running Red Hat 7.3 with Open PBS can be seen below pplxgen

This service became fully operational in Feb 2003. Additional 4 worker nodes to be installed this month. (October 2003)

20th October 2003 Hepix Vancouver - Oxford Site Report

15

Power Cut

http://www-pnp.physics.ox.ac.uk/ganglia-webfrontend-2.5.4/

20th October 2003 Hepix Vancouver - Oxford Site Report

16

CDF Linux Systems

Morpheus is an IBM x3708 way SMP 700MHz Xeonwith 8GB RAM and1TB Fibre Channel disksInstalled August 2001

Purchased as part of a JIF grantfor the CDF group

Runs Fermi Red Hat 7.3.1

Will use CDF software developed atFermilab and here to process data from the CDF experiment.

Second round of CDF JIF tender: Dell Cluster - MATRIX10 Dual 2.4GHz P4 Xeon servers running Fermi Linux 7.3.1 and SCALI cluster software. Installed December 2002

20th October 2003 Hepix Vancouver - Oxford Site Report

18

Approx 7.5 TB for SCSI RAID 5 disks

are attached to the master node.

Each shelf holds 14 * 146GB disks.

These are shared via NFS with the worker nodes.

OpenPBS batch queuing software is used.

CDF Linux Systems - MATRIX

20th October 2003 Hepix Vancouver - Oxford Site Report

19

Plenty of space in the second rack for expansion of the cluster.

Additional Disk Shelf with 14*146GB plus two extra nodes will shortly be ordered. (Autumn 2003)

20th October 2003 Hepix Vancouver - Oxford Site Report

20

Grid development systems. EDG Test bed setup, currently 2.0.3

20th October 2003 Hepix Vancouver - Oxford Site Report

21

Tape Backup is provided bya Qualstar TLS4480tape robot with 80 slots and Dual Sony AIT3 drives.Each tape can hold 100GB of data.Installed Jan 2002.

Netvault Software from BakBoneis used, running on morpheus, forbackup of both cdf and particle physics systems. Main userdisks backed up everyweekday night data disks not generallybacked up BUT weekly backups to OUCS HFS service provide some security.

Network Access

CampusBackboneRouter

Super Janet 4 2.4Gb/s with Super Janet 4

OUCSFirewall

depts

depts

PhysicsFirewall

PhysicsBackboneRouter

100Mb/s

1Gb/s

100Mb/s

1Gb/s

BackboneEdgeRouter

depts

100Mb/s

100Mb/s

100Mb/s

depts

100Mb/s

BackboneEdgeRouter

1Gb/s

Physics Backbone Upgrade to Gigabit Autumn 2002

desktop

Serverswitch

PhysicsFirewall

PhysicsBackboneRouter

1Gb/s

1Gb/s

100Mb/s

100Mb/s

ParticlePhysics

desktop

100Mb/s

100Mb/s

1Gb/s

100Mb/s

Clarendon Lab

1Gb/s

LinuxServer

Win 2kServer

Astro

1Gb/s

1Gb/s

Theory

1Gb/s

Atmos

1Gb/s

20th October 2003 Hepix Vancouver - Oxford Site Report

25

Network Security Constantly under threat from worms and viruses. Boundary Firewall’s don’t solve the

problem entirely as people bring infections in on laptops.

New firewall based on stateful inspection. Policy is now `default closed`. Some teething problems as we learnt what protocols were required but there has been a very significant improvement in security.

Main firewall passes average 5.8GB/hour (link saturates at peak). Rejects 26,000 connection per hour (7 per second). Mischievous connects rejected 1500/hour, one every 2.5 secs. During blaster worm this reached 80/sec.

Additional firewalls installed to protect the Atlas construction area and to protect us from attacks via dialup or VPN.

Need better control over how laptops access our network. Migrating to a new Network Address Translation system so all portables connect through a managed `gateway`.

Have made it easier to keep Anti-Virus software uptodate via simply connecting to a web page. Important that everyone managing their own machines takes advantage of this. Very useful for both laptops and home systems

(see http://www.physics.ox.ac.uk/sophos)

Keeping OS’s patched is a major challenge. Easier when machines are all inside one management domain but is still very time consuming. Must compare to perhaps 1-few man months of IT support staff effort to clean out a successful worm from the network.

20th October 2003 Hepix Vancouver - Oxford Site Report

26

Goals for 2003/4 (Computing)

Continue to improve Network security Need better tools for OS patch management Need users to help with their private laptops

– Use automatic updates (e.g. Windows Update)– Update Antivirus software regularly

Segment the network by levels of trust All the above without adding an enormous management overhead !

Reduce number of OS’s Remove last NT4 machines and exchange 5.5 Digital Unix and VMS very nearly gone. Getting closer to standardising on RH 7.3 especially as the EDG software is

now heading that way. Still finding it very hard to support laptops but now have a

standard clone and recommend IBM laptops. What version of Linux to run ? Currently almost all 7.3 but Red

Hat’s proposal to have limited support & hiving off free releases to fedora project will become a problem.

Looking into Single Sign On for PP systems