NCSU’s Virtual Computing Lab: A Cloud Computing Solution

U niversity IT organi-zat ions face ma ny challenging questions: Can we meet the costs

and diverse requirements of the student computer lab? Do we need on-demand desktops? Are support costs escalating due to conflicts among the dozens of applications installed on the lab computers? Do off-campus students have sufficient access to computing resources? Can we afford all the software students require? Are there compatibility issues with student-owned comput-ers? When should we install a newly released version of an important application demanded by faculty? Do we need reconfigurable high-perfor-mance computing (HPC) resources? Are computing lab resources idle?

At North Carolina State University, we struggled with these and many other related issues. Getting off to a good start looked difficult, and doing it with minimal resources didn’t make the situation any easier.

Virtual Computing labIn 2003, NCSU faculty and staff

considered the concept of deliver-ing computing lab applications to remote users but couldn’t identify the right set of tools to implement it. We installed and evaluated several com-mercial packages, but none of them combined the required performance with the necessary affordability.

Coincidentally, at the same time NCSU began a centralized HPC service (http://hpc.ncsu.edu). A primary HPC resource was a Linux cluster using IBM BladeCenter hardware managed with IBM’s freely available Extreme Cluster Administration Toolkit (xCAT). The experience with this platform made it clear that it could also be used to deliver arbitrary images loaded on a blade server on demand.

This realization led to the birth of NCSU’s Virtual Computing Laboratory (http://vcl.ncsu.edu). A central VCL requirement was affordability. Our goal was to use standard industrial components, along with a combina-

tion of open source and proprietary software (including VCL code devel-oped in Perl and PHP at NCSU), to flexibly and economically meet the lab’s needs. Economy of scale and operation, simplicity, ease of access and usability, ease of maintenance, and automation were the keys to pro-viding the needed computing services efficiently and effectively. An impor-tant side benefit was substantial conservation of electrical energy.

VCl operationNCSU’s central HPC group part-

nered with the university’s College of Engineering IT group to develop a Web interface that students could use to request access to specific images. These are software stacks that range from hypervisor, to operat-ing system, to application, to service access methods. We used xCAT to load the requested “bare-metal” image onto a blade server. Neither group had specific funding for the project, just a problem that needed to

Henry E. Schaffer, Samuel F. Averitt, Marc I. Hoit, Aaron Peeler, Eric D. Sills, and Mladen A. Vouk, North Carolina State University

North Carolina State University’s Virtual Computing Lab cost-effectively and efficiently delivers diverse computing resources over the Internet to more than 30,000 students and faculty.

computer 94

IT SYSTEMS PERSPECTIVES

Published by the IEEE Computer Society 0018-9162/09/$25.00 © 2009 IEEE

95JuLY 2009

image library and therefore subse-quent users.

VCl growthWe piloted the initial VCL system

with a few engineering course sec-tions during the fall 2004 semester. The system delivered Windows applications to users via the Remote Desktop Protocol (RDP) and Linux applications to users via Secure Shell (SSH) and the X Window System Pro-tocol. Other access methods were also used, including Web services.

Most NCSU engineering students have laptop computers. With RDP and Secure Shell/X Window, VCL can deliver both Windows and Linux applications to Windows PC, Mac, and Linux laptops. A broadband Internet connection is a practical requirement.

Feedback from users—including many outside the country—was positive, and at the end of the pilot semester a software vendor provided the impetus for a second college to become involved with VCL. The vendor altered the license terms from allowing distribution of its applica-tion to students to requiring that it be run only on university-owned com-puters. This was a relatively heavily

used application, and students were accustomed to using it on their own computers—an inexpensive student version of the software had limitations that often rendered it unsuitable.

Requiring that students go to a campus computer lab to use the application would have provoked an uproar. Fortunately, providing access to the application via VCL satisfied the license terms while allowing students to continue to access the application from their own computers.

Since then, the number of users has increased each semester. We now serve more than 30,000 students and faculty from a dozen or so campuses. There are also numerous pilots in several states, as well as abroad. We have recorded more than 400,000 individual sessions since the project began. While there were barely 700 reservations during the pilot semes-ter, there are now more than 80,000 per semester.

VCL use continues to increase, as Figure 1 shows, confirming that investment in its resources was worthwhile. With the VCL/HPC cloud now serving more than 7 million HPC CPU hours per year, our approach supports large-scale cloud comput-ing at cents to tens of cents per CPU

be solved: application delivery to on- and off-campus students, and onto student-owned computers.

The initial design had to be simple. We used a MySQL database to track each server’s state, maintain informa-tion about each image, and implement a privilege tree. A Web interface, driven by PHP, authenticated stu-dents, displayed a list of applications they were authorized to use, and let them reserve use of an application either immediately or sometime in the future for a specified length of time. The range of future times and length of reservation were customi-zable and could differ based on the user.

Today, when an authenticated user makes a reservation request on the Web interface, the Web portal sched-ules the request.

While originally VCL was bare-metal-oriented, today it loads either a VMware-based image or a bare-metal image. The VCL system processes the request. If it doesn’t find an available real or virtual server with the desired image already loaded, it selects any available server meeting the specifi-cations required for that image, and xCAT, or the appropriate VM loader, dynamically loads the desired image. If all servers are busy, the Web inter-face presents the student with a grid showing future times when they will be available. Each image is equipped with clients that can access the NCSU enterprise storage data network, any network-accessible storage, and pri-vate storage on the end user’s access device.

We incorporated security into many aspects of the VCL design from the beginning (http://vcl.ncsu.edu/papers-publications). One impor-tant feature is that the VCL system re-images the blade servers for each user session, thereby providing users with administrator/root privileges, if desired. Users can easily modify images, but that doesn’t affect the

0

100

200

300

400

500

600

700

800

9/1/

2004

12/1

/200

4

3/1/

2005

6/1/

2005

9/1/

2005

12/1

/200

5

3/1/

2006

6/1/

2006

9/1/

2006

12/1

/200

6

3/1/

2007

6/1/

2007

9/1/

2007

12/1

/200

7

3/1/

2008

6/1/

2008

9/1/

2008

12/1

/200

8

3/1/

2009

Date

Conc

urre

nt re

serva

tions

Figure 1. Virtual Computing Lab reservations from September 2004, when the VCL project began, through February 2009. While there were barely 700 reservations during the pilot semester, there are now more than 80,000 per semester.

computer 96

IT SYSTEMS PERSPECTIVES

control software adjusts virtual local area network (VLAN) settings on the chassis Ethernet switches to connect the servers to private HPC networks instead of the networks used for indi-vidual application delivery to users connecting through the Internet.

The master HPC login node image includes components of the HPC scheduler—currently Platform Com-puting’s Load Sharing Facility (LSF). Each HPC client image has access to a large amount of scratch storage (in terabytes), as well as end-user home directories and HPC backup storage. When an HPC image is loaded, the master scheduler recognizes it and begins to assign it work. To transition out of the Linux HPC cluster mode, our staff closes blades to new HPC jobs and reassigns them to be loaded with individual VCL images once any running jobs complete. In this way, the blade servers are in use many more hours per year than are tradi-tional student lab computers, and the added availability allows much more HPC research computing than would otherwise be possible.

This amortization of VCL cost over many additional uses and users sub-stantially decreases operation costs. The number of uses of a particular server can be increased in two ways relevant to VCL: time and virtualiza-tion. First, we keep the server busy as many hours per year as possible. Dif-ferent applications and users will have different usage patterns and length of use. Having a mix of usage types and patterns reduces time gaps to be filled and keeps servers busy. Second, we use one physical server as multiple virtual servers to serve multiple uses concurrently. Appropriately assigning applications to servers—physical or virtual—is important for both quality of service and cost-efficiency.

Today the image creator chooses between virtual and dedicated server use in the image metadata: how much memory, how much processing power, and so on. Ideally these would be dynamic decisions based not only

applications, we extended VCL to use virtual (hypervised) servers in addi-tion to physical servers. The system creates virtual servers by loading a host virtual server image onto a blade server. VCL manages guest virtual machines by issuing commands to the hypervisor—which is primarily VMware at this time.

eConomiC beneFitsWhile we have had to expand hard-

ware to accommodate the increased use of VCL, it hasn’t been necessary to increase staff to support the system. In NCSU’s experience, a small staff

can provide a diverse set of applica-tions and services using VCL—many of which wouldn’t have been feasible without the system.

While VCL currently supports more than 600 production images, it cen-trally maintains only 10 to 20. These are used as baseline templates that can then be expanded into end-user production images, thereby distrib-uting the cost of image development and maintenance.

The implication of this experience is that scale and utilization level are important for providing cost-efficient VCL/HPC services. At small scale, staff cost dominates—even with a small staff. As scale increases, staff cost is relatively constant and is amortized over more uses, and cost primarily depends on the hardware resource cost and utilization level. This points to maximizing use of the hardware infrastructure.

When students leave for the summer or breaks and individual use diminishes, we use the VCL hardware resources to expand HPC services. This transition is accomplished with-out any physical changes. Under VCL control, xCAT loads an HPC compute image onto idle nodes, and the VCL

hour depending on the type of ser-vices provided.

system upgradesA few years ago, a computer sci-

ence researcher wanted to modify the kernel on the HPC Linux cluster to try to improve performance. The central HPC Linux cluster is a production resource used by researchers from across the university. Running pro-duction work on an experimentally tweaked kernel isn’t the level of ser-vice central IT wants to deliver, but satisfying CS researchers’ needs is also important.

VCL allowed researchers to pro-duce an image of the custom kernel, and our developers extended the VCL system so that it could load an image onto multiple servers with a single reservation. This ability has proven popular not only for research-ers wanting a cluster on demand, or “subcloud,” but also for users seeking complex environments—for example, a database server, application server, and Web server running together.

Another popular extension to the VCL system has been the addi-tion of block reservations designed to support use by classes. A course instructor makes a reservation—say, for 30 seats 10:00 to 10:50 a.m. every Monday and Friday for the semester—for a particular application. Students visit the VCL portal and have access to the already loaded instances of the application image based on their inclusion in the class roll.

As the range of applications deliv-ered by VCL expanded, we found that many didn’t need the full resources of a physical server. With the increasing availability and use of multicore pro-cessors, more applications are falling into this category. To achieve effi-cient hardware utilization with these

The amortization of VCL cost over many additional uses and users substantially decreases operation costs.

97JuLY 2009

serves only as an access device, with minimal requirements. At the same time, VCL’s virtual aspect provides anytime, anywhere delivery that now enables the delivery of services such as compute-intensive courses via dis-tance education, which opens course work to many.

The delivery of many diverse computing services over the Internet, with flexible

provisioning, has led to much greater efficiency, substantial cost savings, and many ways to enable and empower end users. NCSU’s own experience with cloud computing, through its Virtual Computing Lab, indicates that this approach would be beneficial to a much wider audience.

Henry E. Schaffer is Coordinator of Special IT Projects and Faculty Col-laboration at North Carolina State University (NCSU). Contact him at hes@ncsu.edu.

on the image, but also on the particu-lar user requesting the image and his or her usage profile—for example, power user versus novice user.

NCSU realized that there are numerous desirable extensions of VCL capability, but that they greatly exceed the capacity of its available develop-ment resources. Consequently, the university has donated the VCL code to the Apache Software Foundation, where VCL has been accepted as an incubator project (http://cwiki.apache.org/VCL). Through Apache, a commu-nity of developers, including some from NCSU, can continue to extend and enhance VCL while keeping the code freely available.

Seldom considered in the over-all cost calculations are the initial cost and lifetime of computers used directly. Often these are overspecified or replaced while still in good work-ing order to be able to run occasional high-end applications. With the VCL, the user’s computer—running RDP, X Window, or Web service clients—

editor: richard G. mathieu, Dept. of computer Information Systems and management Science, college of Business, James madison univ., Harrisonburg, VA; mathierg@jmu.edu

Samuel F. Averitt is Vice Provost for Information Technology at NCSU. Con-tact him at sam_averitt@ncsu.edu.

Marc I. Hoit is Vice Chancellor for Information Technology and CIO at NCSU. Contact him at marc_hoit@ncsu.edu.

Aaron Peeler is Program Manag-er—IT Advanced Academic Computing Initiatives at NCSU. Contact him at aaron_peeler@ncsu.edu.

Eric D. Sills is Assistant Vice Provost for Research Computing at NCSU. Con-tact him at eric_sills@ncsu.edu.

Mladen A. Vouk is department head and professor of computer science and Assistant Vice Provost for Informa-tion Technology at NCSU. Contact him at vouk@ncsu.edu.

Advertising Sales Representatives

Recruitment:

Mid Atlantic Lisa RinaldoPhone: +1 732 772 0160Fax: +1 732 772 0164Email: lr.ieeemedia@ ieee.org

New EnglandJohn RestchackPhone: +1 212 419 7578Fax: +1 212 419 7589Email: j.restchack@ ieee.org

SoutheastThomas M. FlynnPhone: +1 770 645 2944Fax: +1 770 993 4423Email: flynntom@ mindspring.com

Midwest/SouthwestDarcy GiovingoPhone: +1 847 498 4520Fax: +1 847 498 5911Email: dg.ieeemedia@ ieee.org

Northwest/Southern CA Tim MattesonPhone: +1 310 836 4064Fax: +1 310 836 4067Email: tm.ieeemedia@ ieee.org

JapanTim MattesonPhone: +1 310 836 4064Fax: +1 310 836 4067Email: tm.ieeemedia@ ieee.org

EuropeHilary TurnbullPhone: +44 1875 825700Fax: +44 1875 825701Email: impress@ impressmedia.com

Product:

US East Dawn BeckerPhone: +1 732 772 0160Fax: +1 732 772 0164Email: db.ieeemedia@ ieee.org US CentralDarcy GiovingoPhone: +1 847 498 4520Fax: +1 847 498 5911Email: dg.ieeemedia@ieee.org

US WestLynne StickrodPhone: +1 415 931 9782Fax: +1 415 931 9782Email: ls.ieeemedia@ieee.org

EuropeSven AnackerPhone: +49 202 27169 11Fax: +49 202 27169 20Email: sanacker@ intermediapartners.de

Advertiser PageClassified Advertising 74-76Cloud Computing Conference 2009 25Cray Inc. 6ICSE 2010 72IDA 75IEEE Computer Society Membership 78-80IEEE Insurance 12IPDPS 2010 Cover 3RE 2009 19Seapine Software Cover 4STFA 74Verizon Wireless Cover 2

Advertising PersonnelMarian AndersonSr. Advertising CoordinatorPhone: +1 714 821 8380 Fax: +1 714 821 4010 Email: manderson@computer.org

Sandy BrownSr. Business Development Mgr.Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: sb.ieeemedia@ieee.org

AdvertiSer informAtion JULY 2009 • Computer