1

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• History

– Mathematical Sciences at the University of the Witwatersrand,Johannesburg - Undergraduate Lab used Novell, DOS, Win 3.11

– Win 95 too big for network booting

– Wanted to keep network booting

– Citrix too expensive

– Already using Linux elsewhere

– Needed special bootroms and drivers (RedHat 6)

– Time expenditure burning bootroms was high

– Importance of a variety of platforms

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

2

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• History

– A few years later had to upgrade again– Novell/DOS/Win system burnt out– Forced to migrate to Win 2000 machines to provide equivalent

software, Samba on server, avoiding customisation problems– PXE instead of bootroms– RedHat 8 loaded kernel with TFTP– NFS-root

3

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Why not Ltsp?

– Ltsp runs all client applications in server RAM– Avoided ltsp because we didn’t believe it would cope with RAM

requirement of approx 6-20 GB as much Linux code isn’t re-entrant– Our solution runs client applications in client RAM– Our solution only stores data and boot OS on server– Still room for experimentation with ltsp

4

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Issues relevant to a PC site installation:

– Costing– Sustainability– Maintainability– Scalability

5

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Costing - TCO

– Software• Linux is free, saving at least R 1-2000 per-seat, plus no server OS cost• 200 PCs in our setup, so we save about R 2-400 000

– Hardware• No hard disks, save on R 500-1000 per PC• But PXE cards expensive so no substantial saving

– Person-hours• Because the software is centralised, saving on cleaning up clients all

the time• No server or client OS or software costs so money can be spent on

person-hours• Actually only cost 1-2 months’ technicians’ salaries anyway

6

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Sustainability - Keeping it going into the future

– Software incompatibilities have led us to change ethernet cardstwice

– Hard disk labs have lasted about six years– Netboot labs have lasted only about three years each– Software is complex to maintain on netboot– Hard disk solution is more sustainable– No total dependence on a server with hard disks

7

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Maintainability - Day to day running

– Hard disk is hard to maintain: users customise it, data corruptions, harddisk errors, viruses, etc.

– Installing software on hard disk is an effort - you need a CDROM driveor you mirror from a network server

– Even mirroring hard disk from a server takes an hour or so– LCFG– Netboot clients can be created from scratch or restored in a minute or

two– Netboot client OS is on the server, to break it they have to hack the

server– Netboot client OS is about 50 MB– Linux hard disk client - RedHat 8 - is about 2 GB

8

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Two issues:

• How easy or time-consuming to add another PC to the pool• What impact adding another PC to the pool has

– Is our solution easy to add a PC to?

– What is the impact of adding PCs?

9

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Ease of addition

• Hard disk involves multiple CDROMs and lots of questions from the installerpackage

• Or you can use LCFG or mirror from a server• But it takes a long time either way, and you have to check the config or customise

it or correct it afterwards• Netboot client very quick to add - about 2 minutes for full config• Adding new software - you don’t have to go to the machine to add the software;

install on the server installs on the clients

10

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Performance on enlargement

• Netboot much faster than hard disk for one or two clients• More than that, booting simultaneously is slower than hard disk• Our solution does not scale well in terms of performance• Factors impacting performance include:

– OS used– Number of client machines– Network protocols– Network topology– Thrashing on the server– Number of servers– Division of labout between the servers

11

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– OS used• Despite upgrading to 100 Mbit on clients and 1 Gbit on server:• RedHat 8 slower than 6, perhaps due to runlevel (loading graphical OS

as well)• Assuming the full OS is 100 MB, each boot of 200 clients involves 20

GB transferred from server• Gigabit ethernet card in server seems poorly supported by RH 8

– Number of client machines• One or two machines booting takes 30 sec• Hard disk slow; spin-up• But all clients effectively competing for one hard disk: the server’s

12

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Number of client machines

• All clients competing for one network card: the server’s• Even if it is Gigabit, it drops effectively to 100 Mbit at merely 10 clients• Actually performs like 100 Mbit• So in our case it performs at theoretical best of 0.5 Mbit per client

13

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

14

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger– Takes about 10 minutes for a whole lab of 50 PCs to boot

15

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Network protocols

• NFS/UDP vs NFS/TCP• Error correction and header size make NFS/TCP slower on a smaller

LAN and increases latency• 95% UDP traffic in 150b sizes is probably resends or polls• 8 K block size for data packets is optimal size• TFTP of kernel is fast so the performance problem is NFS-related• Many nfsd processes doesn’t seem to help• Setting the MTU to 8 K sped things up but had a strange side-effect• Actimeo entry in /etc/fstab is theoretically necessary but doesn’t seem

to make a difference to resends or polls

16

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Network topology

• Switch stack with 3 Gbps cascade cables throughout• Switches are new so there shouldn’t be collisions• Isolating the server on a single switch joined only by one crossover 100

Mbps cable doesn’t seem to help• So the problem isn’t traffic collisions• Server isolated on Gbit module doesn’t seem to make it better either• So the problem is not topology

17

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Network topology• Maybe server ethernet card can only deliver 100 Mbps• Maybe server hard disk can only deliver 100 Mbps• Maybe kernel is thrashing

- Server thrashing- We allocated an nfsd for each lab client, total 50- Server load at 50 (according to “top”) but server idle at 95%- Implies server is not servicing nfsd processes- TFTP boot is fast so it’s not a server load problem, it’s a scheduler

problem or bottleneck problem

18

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Scalability - Difficulties in making it bigger

– Number of servers and division of labour

• The only thing that made a substantial difference was separatingbootup, OS software, user-level software, onto each of their own disks

• User home directories, files, data and email on a separate server whichNFS-exports to clients

• Four servers: two boot servers for four labs, one application server, oneboot server, one user-data server

19

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Summary– Netboot is hard to set up– Netboot of one or two clients, simultaneously, is very fast– Netboot prevents users breaking their PCs, prevents viruses, customisation

of OS, etc– Easy to install software - install once on the server and instantly all clients

have it– Easy to repair or add a client in minimal time– Hard disks are easy to set up– Hard disks are faster if there are many client machines– Hard disks are easily corrupted or virused– Slow and tedious to install software on hard disk, often have to physically sit

at the machine itself– Hard to repair or redo a hard disk install

20

Network Booting versus Hard Disks -Network Booting versus Hard Disks -Costs and ImplicationsCosts and Implications

• Conclusion

– Netboot is cheaper (using Linux), important in Africa– Netboot easier to maintain on a day-to-day basis– Netboot generally slower– The choice is whether performance is more important than ease of

maintenance and lower cost– Windows netboot options are expensive or difficult– Smaller sites not too hard to maintain if they have hard disks– Larger sites hard to maintain with hard disks (consider LCFG)– But larger sites will perform slower under netboot– Recommend larger sites try netboot, smaller sites not