Virtualization (& paravirtualization), [x86] Background, Risks, Controls, Audit Steps

Monday April 23, 2007 Track 3 Session 133Michael Hoesing cisa, cissp, ccp, cia, cpa, cmam-hoesing@cox.net (402) 981-7747

Disclaimer: I never said THAT, if you heard THAT, it wasn’t from me. None of the content of this presentation can be attributed to any of my employers, family members, acquaintances, conference sponsors, quail hunting partners, U-tube co-stars past present or future.

Contents 133

• Drivers – why virtualize (including virtualization enhancing security) (3)

• Practical Applications and History (4-5)• Tools – VMWare, XEN, MS, VirtualIron (workstation &

server) & Recent News (6-8)• Definitions, Architecture, Components (9-11)• General Virtualization Risks (12)• VMWare ESX 2.X (how-to, security, defaults) (13-20)• VMWare ESX 3 What’s New, Security (21- 23)• VMWare ESX 3 Vulnerabilities, Logical Access, Security

Settings, (24-28)• Assessment Tools, Resources, Questions (29-31)• Day 2 =Installation & Configuration, Risk, Controls & Audit (Xen) (31-38)• Day 3 = SuSE 10.1 pro Xen “built-in”, Fedora 6 scripts (39-44)

Drivers (why are we talking about this) 133

• Reduced TCO– 1 (or more) CPU can support many servers– 1 Storage Device & KVM can support many servers

(generally no memory savings)– less footprint (rent, utilities,..)

• Security Facilitation– Cheaper redundancy increasing continuity options– Segregate Development from Testing– Run different O/S based assessment tools

simultaneously• Operations

– Support various O/S’s and configurations– Legacy application migration

Practical Applications 133

• Testing – run a version in a sandbox before deployment

• Testing – have multiple OS's and browsers and see how the website looks in different environments

• Academic – build a network the students can take home on a disk, assess an OS

• any other cost saving opportunity

History 133

• one person, one machine life was good

• one person 2 machines (expensive)

• one person, one machine , dual boot (more choice, but only one choice at a time)

• (para)virtualization - many choices all available concurrently

Workstation Versions 133

FilesFiles Files,LVM or Partitions

MS and OS/2, Linux add-onMS and LINUX, hosts and guests

MS * (requires VT or AMD-V)

Each guest unmodifiedEach guest unmodifiedKernel xenU unprivileged

Host kernel unmodified (sw layer)

Host kernel unmodified (sw layer)

Kernel xen0 modified

GuestsGuestsDomainUs

HostHostDomain0

VirtualizationVirtualizationParavirtualization

Virtual PC $0VMWare $189 (workstation)

(or VMWare Server free)XEN $0 3.0.4

Enterprise Versions 133

Hot Move Guests, P2VHot Move Guests & P2VHot Move Guests, P2V

32bit and 64bit guest servers supported

8 guests optimal, 128 max

up to 32 domU's 32 bit only

96 gig maximum16 gig/guest 64 gig maximum

no max (PAE and SMP)

IDE mgmt, SATA , NAS, SAN for host & guests

IDE or SATA for VC NAS or SAN for guests

IDE minimum, lvm, NAS, SAN

VT or AMD-V 64 bit CPU required for host node

64 bit processor, supported

64bit processor supported (since 3.0.0, supports VT

8 sockets and 32 CPUs maxDual processor min, dual

core support,16 physical max Multi, dual core & VT in 3.0.2 , no max CPUs

ParavirtualizationHost special 2.4 kernel

Guest unmodified kernelParavirtualization

VirtualIron3.5 $499/socket +50 support

VMWare $1,000 $3,750 $5,750 (VI=ESX 3.0 VC 2.0)

XenEnterprise 3.0.2 $750 + 150 or $488)

Recent News 133• Mar 06 – Intel VT , AMD-V (ring –1)• May 06 – MS Virtual PC & Server free• July 06 – VMWare Server 1.0.2 build 39867 – FREE, old

GSX, can be totally free if host is Linux • July 06 – VMWare VirtualCenter 2.0.1, ESX 3.0.1 $$• Sept 06 – Xen Enterprise, mgmt console $750• Oct 06 – Xen 3.0.3 unmodified ("hvm") guests • Oct 06 – Fedora Core 6 , Xen 3.0.2 , virt-manager• Nov 06 – Virtual Desktop (VMWare VDI)• Feb 07 – Lab Manager (VMWare)• Mar 07 – Blue Lane Virtual Shield vuls & patches• Apr 07 – Gartner warning, RHEL ES 5 Xen built in

Definitions 133

• Paravirtualization– Faster?– Altered kernel fulfilling requests rather than an app

sitting on top of the kernel– User space applications need no modification– http://www.cl.cam.ac.uk/Research/SRG/netos/

papers/2003-xensosp.pdf

• Virtualization– Safer?– A software component sits between the guest OS and the host

OS interpreting resource requests

• HVM– Bare metal, fastest?, O/S not altered

Xen 2.0 Architecture 133

Event Channel Virtual MMUVirtual CPU Control IF

Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE)

NativeDeviceDriver

GuestOS(XenLinux)

Device Manager & Control s/w

VM0

NativeDeviceDriver

GuestOS(XenLinux)

UnmodifiedUser

Software

VM1

Front-EndDevice Drivers

GuestOS(XenLinux)

UnmodifiedUser

Software

VM2

Front-EndDevice Drivers

GuestOS(XenBSD)

UnmodifiedUser

Software

VM3

Safe HW IF

Xen Virtual Machine Monitor

Back-End Back-End

Source: Ian Pratt of Cambridge & XenSource

Components 133

• VMWare (VI runs on MS, ESX is it’s own “OS”)• XenEnterprise (mgmt console runs on MS, XE hosts it’s own OS)

• Virtual Iron (mgmt console runs on MS & or Linux, VI node has its’s own OS)

• XEN (runs on Linux & netBSD only) [all can be free]

– xen-x.x.x (paravirtualization tool, xend vmm)

– twisted-x.x.x (networking framework [whatever that means])

– linux -2.6.x.x (the kernel I virtualized)

– bridge-utils (layer 2 protocol free bridging)

– sysfs-utils (file system virtualization)

– Zope-interface, iproute2, libcurl, zlib

Virtualization Risks - General 133

• Availability - Host (Dom0) single point of failure• Confidentiality – Memory Sharing

http://www.cs.nps.navy.mil/people/faculty/irvine/publications/2000/VMM-usenix00-0611.pdf

• Integrity- Complexity:– Connectivity– Hardware provisioning– Application Compatibility

• Access Control, correct assignment of user rights• Talent gap, Linux, VMWare, Networking, Storage

VM ESX 2.5 133

• VMWare Security White Paper http://www.vmware.com/pdf/esx2_security.pdf – No public interfaces – Minimal host installation (apache in default install)– Guest isolation (using files)– AV & Firewall recommended (but not supplied)– Su to root– Default non-promiscuous NIC– Code was audited (scope & methodology not stated)– Use VLANs and place management console on separate vlan– Recommends disabling logging of VM messages in guest (?!)– Host OS is 100% VM ??, only drivers are open source– Management Console is from Red Hat 7.2

VM ESX 2.5 (cont) 133

• VMWare ESX Other– Logical Access Control Provided at the OS level in addition

to MUI users– Can overprovision memory , but throttle with weights called

“shares” – (min host mem 192mg for 8 guests)– Watch routing, eth0 DHCP default install– /etc/vmware the goodies like hwconfig and vm-list– VMotion requires a SAN– Provide for swap or core dump on a separate partition– “IBM blade:

• USB CDROM won’t work on RDM installed guests • Bonded NIC failure of both, fix with Net.Zerospeedlinkdown 1

VM ESX 2.5 (cont 2) 133

• VMWARE ESX More– Console OS – host operating system– Service Console – administers host & guests, do not run X

• VMWare Management Interface – http browser based controls the host and guests, 509 certificated, SSL, 90 second refresh window possible multi-user conflict, DOS possible with:

– /usr/lib/vmware-mui/apache/conf/access.conf vmware_SESSION_LENGTH 0• API – HP Insight, Veritas, • SNMP – feed other tools• Remote Console – control the guest • Check /proc/vmware for allowed methods

– .vmx the guest configuration file /root/vmware/ , text editor can alter– .vmdk the guest image file VM MUI has a file manager– Admin manual suggests “flagship” user that is never on vacation– Install manual requires at least one non-root user

VM ESX 2.5 (cont 3) 133

• VMWARE ESX Still More– PXE Install – from a stored image, test then lock the image– Cannot downgrade from dual processor to single processor– LSI Logic SCSI adapter – see 30 pages of howto– VMware-console-2.x.x-xxxx.exe check authorized use– Reinstall VMware Tools overwrites the power level scripts– Move a vm, check the backup software– Dual CPU requires VMWare Virtual SMP– Backup from Service Console requires guest shutdown

VM ESX 2.5 (cont 4) 133

• More more– No USB on Guest (2 factor impact?)– NT can only run on a single processor machine– Guest event log , user is not identified– /etc/pam.d/vmware-authd – /etc/vmware-mui/ssl/mui.crt and mui.key– Security Config:

• Medium – mgmt and remote encrypted, telnet & FTP are not encrypted

• Low – no connections to host are encrypted• Custom -

VM ESX 2.5 (cont 5) 133

• More again– VMFS 2.11 file system, public shared– Physical extent aka partition– SPAN joins across partitions creating a volume, first

“span” formats thus wiping out existing data– Logs /var/log/vmkernel and vmkwarning– /etc/snmp/snmpd.conf trapcommunity public

(rename this) – vmkload_mod –l to list loaded modules– /etc/vmware/hwconfig and vmkmodule.conf

VM ESX 2.5 (cont 6) 133

• More stuff– LUN masking, only allow guests to see what they

need– vmkmultipath -q where the data goes– Set “security” at HIGH

VM ESX 2.5 Default Installation 133

• LILO without a password

• MOTD empty, no login banner

• gopher, news, mail, finger, ftp, samba 2.2.7, telnet 0.17

• login as root , su not required

• 2.4.6 kernel 3/17/05 last update

• cracklib present, but no pword strength enforcement

• /proc/sys/net/ipv4/conf/all/accept_redirects 1

• ports 902 8222 8333

ESX 3 - New Features from 2.5

• Pricing – lower entry level, higher enterprise version

• VirtualCenter 2.0.1, create, move, • License Server – centrally manage, & redeploy

licenses

• HA (high availability)• DRS (distributed resource scheduler)• VMWare Healthcheck $9,000 (security not

mentioned)

ESX 3 - Security133

• the kernel is 2.4, does your policy require anything more current?

• the distribution is based on RHEL 3, is this an approved distro?

• the MOTD file is empty, is this where we want to place a warning banner?

• the default build sets the time at PDT and NTP is not enabled by default

• LILO is gone, grub is the boot loader, but there is still no boot loader password

• the default install allocates 272 meg of memory to the host (add more with a grub edit)

• CIM (Common Information Model) and WBEM (Web Based Enterprise Management) are running on ports5988, 5989 

ESX 3 – Security (cont) 133

• ports 2050, 8042, 27000, 27010, ? seem to be allowed in via the iptables rules? VMotion clear, P2V clear?

• /proc.sys/net/ipv4/source_redirects is set at "1", should this be enabled?

• the first iptables rule is "ACCEPT all -- anywhere anywhere", how do we get past this to the other rule chains?

• /etc/logindefs has the password life set at 90 days • ftp-0.17-17 is installed (but not listening), is it needed? • openssl--0.9.7a-33.17, is this an approved version? • openssh-3.6.1p2, is this an approved version? • umask is 022, is this in line with your standard?

ESX 3 – Security (cont) 133

ESX 3 Security Hardening Whitepaper2007http://www.vmware.com/pdf/vi3_security_hardening_wp.pdf

• “..attacking and individual virtual machine will result in the compromise of only the virtual machine..“ (1 hack OK?) (page 4 clarifies

• Watch patching of dormant (turned off) virtual guests• Rotate logs to prevent DoS• Separate VLANS for management traffic• Configure the firewall (iptables provided)• Use Directory Services (NIS)for admin authentication• Protect Root • SNMP is read only

ESX Vulnerabilities 133 • CVE-2006-2481    • Summary: VMware ESX Server 2.0.x before 2.0.2 and 2.x before 2.5.2 patch 4 stores

authentication credentials in base 64 encoded format in the vmware.mui.kid and vmware.mui.sid cookies, which allows attackers to gain privileges by obtaining the cookies using attacks such as cross-site scripting (CVE-2005-3619).

• Published: 7/31/2006 • CVSS Severity: 2.3 (Low) • CVE-2005-3620      VU#822476 • Summary: The management interface for VMware ESX Server 2.0.x before 2.0.2 patch 1,

2.1.x before 2.1.3 patch 1, and 2.x before 2.5.3 patch 2 records passwords in cleartext in URLs that are stored in world-readable web server log files, which allows local users to gain privileges.

• Published: 12/31/2005 • CVSS Severity: 1.6 (Low)

CVE-2005-3619    • Summary: Cross-site scripting (XSS) vulnerability in the management interface for

VMware ESX 2.5.x before 2.5.2 upgrade patch 2, 2.1.x before 2.1.2 upgrade patch 6, and 2.0.x before 2.0.1 upgrade patch 6 allows remote attackers to inject arbitrary web script or HTML via messages that are not sanitized when viewing syslog log files.

• Published: 12/31/2005 • CVSS Severity: 10.0 (High)

ESX Vulnerabilities (cont) 133

• CVE-2005-3618    • Summary: Cross-site request forgery (CSRF) vulnerability in the management

interface for VMware ESX Server 2.0.x before 2.0.2 patch 1, 2.1.x before 2.1.3 patch 1, and 2.x before 2.5.3 patch 2 allows allows remote attackers to perform unauthorized actions as the administrator via URLs, as demonstrated using the setUsr operation to change a password. NOTE: this issue can be leveraged with CVE-2005-3619 to automatically perform the attacks.

• Published: 12/31/2005 • CVSS Severity: 8.0 (High)

• Per VMWare– SSL keys, change the default ownership to root (assuming root is protected)– 2003-0386 IP restrict and enable & verify reverse mapping off, not applicable to ESX– 2003-0693 SSH 3.6 buffer overflow, not applicable to ESX– 2003-0987 Apache mod_digest replay, not applicable to ESX– 2005-2798 SSH GSSAPIDelegateCredentials, not applicable to ESX– 2006-2444 snmp trap, not applicable to ESX – 2006-3747 cross site scripting w http trace, use separate vlans

ESX 3 – Logical Access 133

• Protect root on the host (and Administrator on the Virtual Center server)

• Users

• Roles

• No History

ESX 3 – Security Setting 133

ESX 3 Assessment Tools 133

• Ecora Auditor Pro 4.1 tool Http://www.ecora.com/ecora/pr/06-11-2006-b.asp

• “regular” Linux assessment of ESX Host– Nessus– CIS/Bastille --assess– LSAT– MTH script

http://www.certconf.org/presentations/2006/

OTHER - Resources 133

• trust a seminar speaker, but verify

• The Source http://www.vmware.com – Technology network http://www.vmware.com/community/index.jspa

– Security topics http://www.vmware.com/vmtn/technology/security/ – Security Response http://

www.vmware.com/support/policies/security_response.html

• Book by Al Muller http://www.amazon.com/Virtualization-VMware-ESX-Server-Muller/dp/1597490199/ref=pd_bxgy_b_text_b/104-0393259-8012733

• Arrasjid & Mills http://download3.vmware.com/vmworld/2005/sln138.pdf

• Virtual Desktops – another multiday topic

• VM cloning of credentials http://www.thoughtpolice.co.uk/vmware/howto/vmware-security-tips.html

• Blogs http://www.virtualization.info/2003/09/virtualization-sites-blogs.html

OTHER 133

• Questions ??• How many Texans does it take to………….

• New fud • http://www.networkworld.com/supp/2007/ndc2/031907-ciso-insight-side-virtualization.html • http://www.gartner.com/it/page.jsp?id=503192

• New Non-fud– Management tools http://www.nworks.com/vmware/– http://members.cox.net/m-d-hoesing/CACS_Virtualization_V2.ppt

• MTH_Linux_Audit_V8.4.txt • chkrootkit.tar.gz • FC6_Xen_Installation_11_2006.doc

• Big 3 –– Current Patches– High setting on connections– Appropriate user rights