[lvs-users] Kernel Tuning for LVS

Jay Faulkner jay.faulkner at mailtrust.com
Mon Aug 9 15:18:55 BST 2010

> -----Original Message-----
> From: lvs-users-bounces at linuxvirtualserver.org [mailto:lvs-users-
> bounces at linuxvirtualserver.org] On Behalf Of Neal B
> Sent: Friday, August 06, 2010 2:16 PM
> To: lvs-users at linuxvirtualserver.org
> Subject: [lvs-users] Kernel Tuning for LVS
> Over the past day I have been searching the net for the definitive answer for
> the recommended tweaks to the kernel for ipvs.  If you are doing this then
> change this, but that does not exist.  :)


I've done a few things to tune the network for large-traffic here, and I'm going to share with you the values I set, what I set them to, and why. Please remember YMMV and don't modify something unless you completely understand what it does. In a nutshell, if you break stuff using these values, it's not my fault :).

These values are on -- I'm unsure as to what they do or translate to in newer or older kernels.


These three values make the ARP cache larger. This is not needed in all environments, but generally only costs ram so it doesn't hurt to increase it:
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh1 = 1024

The next options are for TCP window scaling, I won't go into details because there are so many internet howtos on how to tweak your window scaling, but here are the places and what I usually set them to:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65535 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

Next, I always disable proxy arp (RHEL, by default, sets the kernel to arp any IP on the box on any interface; usually this is undesirable and can break things for edge routers) -- make sure to set these values for net.ipv4.conf.[interface].arp_[ignore|announce] as well.
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

A couple of other important sysctls imo:
net.core.netdev_max_backlog = 2500 # Increases the receive queue in the kernel
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 86400 # lowers default timeout (default: 5d - I think it's a little silly)

Non-SYSCTL tweaks:
/sbin/ifconfig eth[#] txqueuelen 2000 # increases the size of the transmit queue

Check to see what the max ring buffer size is for your NIC card (should be listed under Pre-set Maximums): 
ethtool -g eth[#]

If any of the "Current Hardware Settings" values are less than the maximums, set them to the max:
ethtool -G eth[#] rx [value]
ethtool -G eth[#] tx [value]

The biggest tweak I can advise you to do though is to *run a good network card*. I don't put a load balancer in production, regardless of load, unless it's running an e1000[e] driver NIC. These drivers have proven themselves the most stable of any I've run, and have an additional setting you can set to lower interrupts on a high-load system:
(inside modprobe.conf)
options e1000[e]  InterruptThrottleRate=3000,3000,3000,3000 

This tells the e1000 driver to *never* make more than 3000 interrupts per second. This is good if you want pure volume of traffic at the expense of a little more latency (very little more, really).

I hope this has been helpful. Please let me know if you have any questions.

Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner at rackspace.com

More information about the lvs-users mailing list