[lvs-users] Kernel Tuning for LVS

Jay Faulkner jay.faulkner at mailtrust.com
Mon Aug 9 15:18:55 BST 2010


> -----Original Message-----
> From: lvs-users-bounces at linuxvirtualserver.org [mailto:lvs-users-
> bounces at linuxvirtualserver.org] On Behalf Of Neal B
> Sent: Friday, August 06, 2010 2:16 PM
> To: lvs-users at linuxvirtualserver.org
> Subject: [lvs-users] Kernel Tuning for LVS
> 
> Over the past day I have been searching the net for the definitive answer for
> the recommended tweaks to the kernel for ipvs.  If you are doing this then
> change this, but that does not exist.  :)
> 

Hey,

I've done a few things to tune the network for large-traffic here, and I'm going to share with you the values I set, what I set them to, and why. Please remember YMMV and don't modify something unless you completely understand what it does. In a nutshell, if you break stuff using these values, it's not my fault :).

These values are on 2.6.27.46 -- I'm unsure as to what they do or translate to in newer or older kernels.

SYSCTLS:

These three values make the ARP cache larger. This is not needed in all environments, but generally only costs ram so it doesn't hurt to increase it:
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh1 = 1024

The next options are for TCP window scaling, I won't go into details because there are so many internet howtos on how to tweak your window scaling, but here are the places and what I usually set them to:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65535 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

Next, I always disable proxy arp (RHEL, by default, sets the kernel to arp any IP on the box on any interface; usually this is undesirable and can break things for edge routers) -- make sure to set these values for net.ipv4.conf.[interface].arp_[ignore|announce] as well.
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

A couple of other important sysctls imo:
net.core.netdev_max_backlog = 2500 # Increases the receive queue in the kernel
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 86400 # lowers default timeout (default: 5d - I think it's a little silly)


Non-SYSCTL tweaks:
/sbin/ifconfig eth[#] txqueuelen 2000 # increases the size of the transmit queue

Check to see what the max ring buffer size is for your NIC card (should be listed under Pre-set Maximums): 
ethtool -g eth[#]

If any of the "Current Hardware Settings" values are less than the maximums, set them to the max:
ethtool -G eth[#] rx [value]
ethtool -G eth[#] tx [value]


The biggest tweak I can advise you to do though is to *run a good network card*. I don't put a load balancer in production, regardless of load, unless it's running an e1000[e] driver NIC. These drivers have proven themselves the most stable of any I've run, and have an additional setting you can set to lower interrupts on a high-load system:
(inside modprobe.conf)
options e1000[e]  InterruptThrottleRate=3000,3000,3000,3000 

This tells the e1000 driver to *never* make more than 3000 interrupts per second. This is good if you want pure volume of traffic at the expense of a little more latency (very little more, really).



I hope this has been helpful. Please let me know if you have any questions.

--
Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner at rackspace.com




More information about the lvs-users mailing list