PERFORMANCE TUNING: THE OS, NETWORK, AND COUCHBASE SERVERDean Proctor, Couchbase

©2015 Couchbase Inc. 2

Why do we tune?

Operating System Tuning

©2015 Couchbase Inc. 4

Virtual Memory Architecture

Virtual Memory

Swapper

OOM Killer

Physical Memory

Disk Swap Space

©2015 Couchbase Inc. 5

Memory Allocation

Swap vm.swapiness=[0,1]

Transparent Huge Pages (THP) grub.conf: transparent_hugepage=never echo never >

/sys/kernel/mm/transparent_hugepage/enabled echo never >

/sys/kernel/mm/transparent_hugepage/defrag

©2015 Couchbase Inc. 6

NUMA Architecture

CPU Socket 1

DIMM 1

DIMM 3

CPU Socket 2

DIMM 2

DIMM 4

QPI

©2015 Couchbase Inc. 7

NUMA Architecture

CPU Socket 1

DIMM 1

DIMM 3

CPU Socket 2

DIMM 2

DIMM 4

QPI

Node0 Node1

©2015 Couchbase Inc. 8

Disable NUMA

Disable in BIOS Disable for Couchbase

Init script: numactl –interleave all $DAEMON vm.zone_reclaim_mode=0

©2015 Couchbase Inc. 9

Linux Storage Architecture

Physical

Block

Cache

VFS

©2015 Couchbase Inc. 10

Linux Storage Architecture

PhysicalType Allocation RAID

BlockScheduler IO Request Queue

CacheAllocation Flush Frequency

VFSFS Type Mount options

©2015 Couchbase Inc. 11

VFS Tuning

Filesystem Type EXT4 XFS

Mount options EXT4: noatime,barrier=0,data=writeback XFS:

noatime,nobarrier,logbufs=8,logbsize=256k,allocsize=2M

©2015 Couchbase Inc. 12

Page Cache

Set the limit for dirty bytes in the page cache vm.dirty_bytes vm.dirty_background_bytes

Set the max time for dirty pages vm.dirty_expire_centisecs vm.dirty_writeback_centisecs

Tune VFS cache reclaim vfs_cache_pressure

©2015 Couchbase Inc. 13

Scheduler

Scheduler algorithm echo deadline > /sys/block/<dev>/queue/scheduler

IO request queue echo 1024 > /sys/block/<dev>/queue/nr_requests

©2015 Couchbase Inc. 14

Physical Devices

Use local storage, 10k+ SATA or SSD Provision separate disks for data and indexes Battery-backed RAID Align your IO!

Network Tuning

©2015 Couchbase Inc. 16

Network Architecture

NIC

TX Ring

TX Queue

TX Socket Buffer

RX Socket Buffer

NIC

RX Ring

RX Queue

©2015 Couchbase Inc. 17

Resilience

Buffers net.core.rmem_max net.core.wmem_max net.ipv4.tcp_mem net.ipv4.tcp_rmem net.ipv4.tcp_wmem

Backlogs net.core.netdev_max_back

log net.core.somaxconn net.ipv4.tcp_max_syn_bac

klog

Balance IRQs echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

©2015 Couchbase Inc. 18

Efficiency

Reduce TCP overhead net.ipv4.tcp_sack net.ipv4.tcp_fack

Reduce connection overhead net.ipv4.tcp_fin_timeout net.ipv4.tcp_tw_reuse

Enable auto-tuning net.ipv4.tcp_moderate_rcvbuf net.ipv4.tcp_window_scaling

©2015 Couchbase Inc. 19

Jumbo Frames

8KB Object

1500B MTU

9000B MTUifconfig <dev> mtu 9000

Couchbase Tuning

©2015 Couchbase Inc. 21

Bucket Settings

©2015 Couchbase Inc. 22

Working Set Management

Memory Quota

mem_low_wat

mem_high_wat

©2015 Couchbase Inc. 23

Views

Design documents Fewer reduces threads More increases threads

Automated index updates updateInterval updateMinChanges replicaUpdateMinChanges

POST http://nodename:8091/settings/viewUpdateDaemonupdateInterval=10000&updateMinChanges=1000

©2015 Couchbase Inc. 24

XDCR

©2015 Couchbase Inc. 25

Compaction

©2015 Couchbase Inc. 26

Internal Settings

http://nodename:8091/index.html?enableInternalSettings=1

Thank you.

Get Started with Couchbase Server 4.0: www.couchbase.com/beta

Get Trained on Couchbase: training.couchbase.com