virtuozzo linux advanced q&a: deployment and optimization

Virtuozzo Linux Advanced Q&A: deployment and optimization

Andrey Moruga

Product Manager

May 9, 2007

Virtuozzo Linux Advanced Q&A: deployment and optimization2

Why Should I Care?• Want to get most out of your servers?

Better density per physical server More computing power to the applications, not to the internal system tasks Better performance and response time Better uptime

• Review and optimize Hardware configuration Software tuning VE parameters

• And get the result More Competitive pricing Faster ROI

customers per physical servershappy

3

But my Servers Perform Already Well!Maybe they do

But…

They can always do better!


4

Sometimes the Problems are Obvious

# time lsbin dev home lib media mnt proc sbin tmp var boot etc initrd lost+found opt root usr vz

real 0m1.205suser 0m0.001ssys 0m0.004s


• Simple web pages are loaded several seconds

• The file system is very slow

# free -m

total used free shared buffers cached

Mem: 752 740 12 0 145 230

Swap: 1537 1470 67

• Swap space always in use for more than 50-80%

• “uptime” showing load average tens (30 and more)# uptime

15:45:47 up 29 days, 2:04, 1 user, load average: 71.40, 68.27, 70.79

5

Lack of Resources: Symptoms• CPU shortage

CPU(s) have almost 0 idle cycles – always are busy Applications are slow, bad response time in interactive mode A lot of processes are in “R” – running state High load average (reported by “uptime” command): 30 and more

• Disk IO shortage Disk access it very slow. Simple command like “ls” takes forever Applications are slow, bad response time in interactive mode A lot of processes are in “D” – blocked (“D”isk) state High load average (reported by “uptime” command): 30 and more

• Memory Swap space is heavy used; intensive swap-in/out activity OOM (Out-Of-Memory) kills happens often


6

CPU Shortage: Problem Confirmation


• Check CPU idle and number of the processes running# vmstat 2

procs ----------memory--------- --swap-- ----io--- -system- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

38 0 0 218444 148948 235916 0 0 0 5 15 16 0 0 100 0

49 0 0 218444 148948 235916 0 0 0 0 1006 23 100 0 0 0

37 1 0 218444 148948 235916 0 0 0 20 1008 28 100 1 0 0

34 0 0 218444 148948 235916 0 0 0 0 1004 26 100 1 0 0

35 0 0 218444 148948 235916 0 0 0 0 1005 23 100 0 0 0

38 0 0 218444 148948 235916 0 0 0 0 1005 25 100 0 0 0

33 0 0 218444 148948 235916 0 0 0 0 1004 23 100 0 0 0

32 0 0 218444 148948 235916 0 0 0 0 1006 26 100 0 0 0

7

CPU Shortage: Problem Resolution• Find the processes that are always running, and

prevent them from running again ps ax| grep “ R ” or top to find running processes vzpid {PID} to locate a VE where they are running vzstat to see CPU usage on per-VE basis

• Check for possible misconfiguration Compare CPUUNITS assigned to different servers using vzcpucheck -v.

Avoid a significant imbalance of CPUUNITS (by orders of magnitude) between VEs on the same server

Check CPULIMITS applied to VEs. Avoid setting CPULIMUT to less than 50% of single CPU

Examine /proc/user_beancounters file. Failure in the resources allocation may provoke CPU consuming

• Buy more powerful server Just kidding


8

Disk IO Shortage: Problem Confirmation


• Check the number of the processes blocked (in “D” state), the disk IO block counts and CPU wait/idle time

# vmstat 2



0 23 592 19040 2932 601296 0 0 4730 11530 1138 100 2 15 0 84

0 22 592 2768 2876 636860 0 0 5672 17506 1144 119 2 29 0 70

0 21 592 1080 2836 640420 0 0 3820 10368 1135 95 1 10 0 89

0 20 592 2856 2820 634004 0 0 3242 17344 1158 118 3 31 0 67

0 20 592 2756 2812 634092 0 0 5612 10866 1134 109 2 19 0 80

0 20 592 2796 2776 633836 0 0 2112 17392 1176 126 1 15 0 83

0 21 592 2692 2756 639380 0 0 4200 18534 1157 124 3 28 0 69

0 22 592 928 2796 645092 0 0 1110 11908 1143 111 3 18 0 80

• Check if extra disk activity increases the load average dramatically

E.g. backup process increases load average from 30 to 60

9

Disk IO Shortage: Problem Resolution


• Make sure the “noatime” option for /vz partition set# cat /etc/fstab|grep /vz

LABEL=/vz /vz ext3 noatime 1 2

• Make sure slocate doesn’t provoke extra load In the /etc/updatedb.conf file, the PRUNEFS variable should include “vzfs”,

PRUNEPATHS variable should include “/vz” Check that VEs are not running updatedb by default

• In case you run 2.6.8 kernel, CFQ disk scheduler may provide better performance

Add “elevator=cfq” line to your kernel boot parameters and restart the server

• Review the /vz partition block size Though 4096 bytes block size may waste some additional disk space, it

gains slightly better performance than 1024 bytes

10

Disk IO Shortage: Problem Resolution (Contd.)


• Check the file system journal size# debugfs -c /dev/sda3

debugfs 1.35 (28-Feb-2004)

/dev/sda3: catastrophic mode - not reading inode or group bitmaps

debugfs: stat <8>

Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0

User: 0 Group: 0 Size: 8388608

File ACL: 0 Directory ACL: 0

Links: 1 Blockcount: 16452

Fragment: Address: 0 Number: 0 Size: 0

ctime: 0x4421c312 -- Wed Mar 22 16:35:14 2006

atime: 0x00000000 -- Wed Dec 31 19:00:00 1969

mtime: 0x4421c312 -- Wed Mar 22 16:35:14 2006

BLOCKS:

(0-11):1426-1437, (IND):1438, (12-267):1439-1694, (DIND):1695, (IND):1696, (268-523):1697-1952, (IND):1953, (524-779):1954-2209, (IND):2210, (780-1035):2211-246

[...output skipped...]

11



• If needed, increase the file system journal size to the recommended size 64-128MB

# tune2fs -l /dev/sda3|egrep "Filesystem state|Filesystem features"

Filesystem features: has_journal ext_attr resize_inode filetype sparse_super

Filesystem state: clean

# tune2fs -O ^has_journal /dev/hda3

tune2fs 1.35 (28-Feb-2004)

# tune2fs -J size=64 /dev/hda3

tune2fs 1.35 (28-Feb-2004)

Creating journal inode: done

# debugfs -c /dev/hda3

debugfs 1.35 (28-Feb-2004)

/dev/hda3: catastrophic mode - not reading inode or group bitmaps

debugfs: stat <8>

Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0

User: 0 Group: 0 Size: 67108864

[...skip...]

needs_recovery

12 Virtuozzo Linux Advanced Q&A: deployment and optimization

• Check that your VEs have an option to omit log files syncing

# cat /etc/syslog.conf

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none -/var/log/messages

# Log all the mail messages in one place.

mail.* -/var/log/maillog

• Review your hardware configuration SCSI controllers usually provide the better performance due to their

hardware ability to rearrange disk request in the most efficient order Stripe volumes allow you to significantly increase disk IO bandwidth due to

their ability to distribute single operation to multiple volumes simultaneously RAID10 disk arrays may be used in order to have both striping and mirroring

for high performance and data redundancy


13

Memory Shortage: Problem Confirmation


• Check the memory, swap space usage and swap activity

# vmstat 2



0 6 110084 1112 508 10600 100 14422 316 14434 1200 249 1 39 0 60

4 3 52464 1184 1060 10952 108 512 1369 531 1144 229 5 95 0 0

0 21 77692 944 440 5172 60 13244 864 13244 1145 304 0 11 0 88

0 26 98264 968 520 6308 180 10286 778 10286 1181 271 0 5 0 95

0 25 110884 976 580 7560 480 6310 1132 6319 1133 278 0 5 0 96

0 20 90944 972 664 9008 384 13264 1155 13273 1151 318 1 31 0 68

• Search for OOM (Out-Of-Memory) kills OOM kills are reported as kernel messages and may be found in

/var/log/messages file, indicated by string like “Apr 26 15:45:31 testbox kernel: OOM killed process httpd (pid=19245, ve=10000) (mm=db1a9100) exited, free=5619 gen=3429.”

14

Memory Shortage: Problem Resolution• Check your system configuration

Hosting production servers with Virtuozzo usually should have at least 4GB of physical memory installed

Make sure the swap size is not less than the memory size Make sure the proper kernel is used (run “uname -a” to see the kernel

version used)▪ Small UniProcessor systems should use “up” version of kernel

▪ SMP system with up to 4GB of memory should use “smpnosplit” kernel

▪ If the memory size exceed 4GB, “entnosplit” kernel should be used

▪ If the average number of processes on your system exceeds 2,000 – the “split” version of kernel (“smp” and “enterprise” accordingly) is recommended

▪ 64bit kernels have no memory limitations, so “smp” kernel fits systems with any amount of memory installed and number of processes running


15

Memory Shortage: Problem Resolution (Contd.)

• Tune your default applications configuration Switch off unused application features and modules Place rarely used network services under xinet.d based startup

(vzsetxinetd tool allows you to switch a service startup mode) Fine tune applications parameters. Typically, applications tuned for a

dedicated server are more “pre-allocate” oriented than their VE counterparts, so the default application configuration may not be optimal for the VE.

For example, number of apache startup processes may be modified using StartServers, MinSpareServers and MaxSpareServers configuration variables

• Use benefits of Virtuozzo templates Template based applications may provide great memory sharing and saving.

Avoid installing applications in your VEs directly from binary packages


16

Summary• CPU shortage

Symptoms▪ Poor performance, high load average, high number of running processes

Solution▪ Kill abusing processes, review VE CPU and system resources settings

• Disk IO shortage Symptoms

▪ Poor performance, high load average, high number of blocked processes, minimal extra disk activity provokes load spikes

Solution▪ Review server hardware and software configuration, minimize heavy disk IO operations

• Memory shortage Symptoms

▪ High swap usage and swap out activity, often OOM kills

Solution▪ Add more RAM, review VE applications settings, use benefits of Virtuozzo templates


17

Q&A: Deployment and Troubleshooting-1• Q: What the recommended server partitioning is?

A: Use the following general guidelines: Three partitions: / (root), /vz, swap /vz has highest performance requirements, / (root) has minimal ones Assign 4 to 10 GB of disk space for / (root), swap space is equal to the

RAM size, the rest of the disk space given to /vz

• Q: Can I have multiple disk partitions mounted under /vz?

A: Yes, though keep in mind that: Templates (/vz/template) should reside on a single disk partition Single VE private area (/vz/private/{VEID}) should reside on a single disk

partition If you need to use multiple partition for a VE, mount it into the VE’s root

(/vz/root/{VEID}/mnt) instead of the private area


18

Q&A: Deployment and Troubleshooting-2• Q: How can I leverage SAN on Virtuozzo?

A: You can build automatic Virtuozzo failover cluster using Red Hat Cluster Suite software


Active ActivePassiveActiveHardware Crash!!!

VE VE

RH Cluster

Node 1 Node 2 Node 3

VE VE

19

Q&A: Deployment and Troubleshooting-3• Q: What is the easiest way to locate a VE by IP?

A: Assign the VEID based on two last octets of the VE’s IP. For example : 83.241.134.71 → VEID 134071

• Q: What is the major differences between UBC and SLM in the memory accounting?

A: UBC accounts address space, while SLM accounts physical memory usage


1 MB allocated

256 KB used

UBC accounts 1 MBSLM accounts 256 KB

Address space

20

Q&A: Deployment and Troubleshooting-4• Q: How do I stop all the VEs real fast?

A1: # for i in `vzlist -1`; do vzctl stop $i --fast; done

A2: # killall -9 init

• Q: It looks very brutal… can it corrupt the VE file system?

A: No, the file system will be cleanly dismounted by the host operating system. Though, applications data may be potentially corrupted because of unclean application shutdown.


21

Q&A: Deployment and Troubleshooting-5• Q: Do I need to apply updates to my EZ templates?

A: Generally you do not – EZ template is just a set of configuration files. Included applications are updated automatically when you update your VEs

• Q: Do you recommend to maintain our own EZ templates repository, or we can use one on the Internet?

A: Maintain your own. There is a number of reasons: Saving on network bandwidth and better availability Replicated repository inconsistency may be fixed locally Internet repositories may remove older versions of applications, that has

negative impact on VEs using them (preventing a VE migration or restoring from a backup)


22

Q&A: Deployment and Troubleshooting-6• Q: What is the advantage of EZ templates over

Standard templates?

A: They are a way better. A VE update is just a single command instead of hours of manual work

• Q: Since centos.org removed CentOS 4.2 from their repository, I cannot restore CentOS EZ template based VE from a backup. What should I do?

A: Add http://vault.centos.org/4.2/os/i386/ to the list of repositories on your Virtuozzo server. The most reliable way is still using your own (mirrored) repository

• Q: Can I apply Virtuozzo updates by a script?

A: Yes. Use “vzup2date -m batch install”


23

Q&A: Deployment and Troubleshooting-7• Q: Can I assign an IP to a VE by DHCP?

A: Yes, but only in switched networking mode. Routed networking does not support DHCP based IP assignment.

• Q: How do I mount /tmp with noexec, nosuid, nodev options in a VE?

A: # vzctl set VEID --bindmount_add /tmp,nosuid,noexec,nodev --save

• Q: How can I manage a VE CPU power?

A: There are 3 parameters controlling it: A relative CPU weight - CPUUNITS (mandatory) An upper CPU limit - CPULIMIT (optional) A number of CPUs the VE can use - CPUS (optional)


24

Q&A: Deployment and Troubleshooting-8• Q: How do I block particular TCP port for all the VEs

on a server?

A: Use the “FORWARD” chain on the host in your iptables rules:# iptables -I FORWARD -p tcp --dport 6667 -j DROP

• Q: I have a VE with huge static content. The content does not need to be backed up. How do I exclude that content from regular VE backup?

A: Put the static content on external mount point via mount scripts. Note: directories mounted via “--bindmount” option of vzctl are included into the VE backup


25

Q&A: Deployment and Troubleshooting-9• Q: How do I create a partition greater than 2TB? fdisk

tool doesn't seem to be able to do that

A: Use GNU “parted” tool to partition your drive and create GPT labeled partition. Note: currently Linux cannot boot from such partitions.

• Q: I have NAS device which I can mount to the Virtuozzo box over NFS or CIFS. Can I place VE's private area on NAS share?

A: No, currently it is not supported. Though, NFS or CIFS partition may be bind-mounted into a VE.


26

Q&A: Deployment and Troubleshooting-10• Q: When I create iptables rule with 'LOG' target inside a

VE, packets are still not logged. How do I fix it?

A: By default kernel logger is disabled in a VE. To enable it, modify the /etc/rc.d/init.d/syslog file in your VE. Find the lines which read


passed klogd skipped #daemon klogd $KLOGD_OPTIONS

passed klogd skipped #killproc klogd

daemon klogd $KLOGD_OPTIONS

killproc klogd

and change these lines to be

virtuozzo linux advanced q&a: deployment and optimization

Documents

svirtuozzo linux advanced

uptime command

cpu usage

load average tens

running processesvzpid

bad response time

optimizationcheck cpu

s user 0m0