[lvs-users] NFCT and PMTU

lvs at elwe.co.uk lvs at elwe.co.uk
Tue Sep 11 22:22:59 BST 2012

On Tue, 11 Sep 2012, Julian Anastasov wrote:

>> Under CentOS 3 (traditional interrupts) with SMP affinity set to all cores
>> (or rather half the cores for the external NIC and half for internal NIC)
>> load scaled linearly until it fell off a cliff and load hit 100% and more
>> generated traffic resulted in no more throughput (lots of Xoffs). I also
>> have some old data showing NFCT improving performance on CentOS 3.
> 	So, keeping netfilter conntracks (conntrack=1) uses
> less CPU cycles than creating conntracks with every
> packet (conntrack=0). I hope you have large nf_conntrack_max
> value for the conntrack=1 case.

I should give you some more information about my directors. As well as 
being LVS directors they are doing firewalling with netfilter. I use 
netfilter marks to tell IPVS which connections to route to which pool. 
Thus netfilter will be tracking the state of every packet whether 
conntrack=0 or conntrack=1.

I suspect that having IPVS keep netfilter up to date with what it is doing 
is helping it to find the state of the connection in its state tracking 
table quicker, thus there is less CPU load.

I have nf_conntrack_max set to just over 16 million, though I rarely go 
over 2 million tracked connections (a good percentage are UDP). I also 
have all the netfilter and IPVS timeouts set to much lower values than the 
defaults, but still safe values. When I changed these values I reduced my 
tracked connections by 90%. I also have my hash table set to the maximum 
size, to avoid hash hits as much as possible.

At one point I experimented with spliting the director and netfilter 
firewall onto seperate servers (with NFCT off on the director and no 
netfilter modules loaded). The softirq load split exactly half onto each 
server. I believe this is because IPVS and netfilter are very simular in 
their state tracking and this is the most expensive part of both netfilter 
and IPVS. Certainly getting rid of connection attempts from the most 
prolific spammers in netfilter's raw table, before it does state tracking, 
provides a huge reduction in the number of conntracks and the softirq 

I always put the fastest Xeon's I can get my hands on in my directors. At 
one point I had a sub optimal memory configuration, so the server was 
striping across 2 memory buses instead of 3. When I fixed the memory 
configuration I saw softirq drop by 33%, suggesting the director was 
waiting for reads from main memory most of the time.


>> Looking at my monitoring graphs for one director when I flipped conntrack
>> from 1 to 0 overall traffic in the peak hour stayed at 1.4Gb while softirq
>> load on the busiest core rose from around 43% to around 62%. Average
>> sotirq load across all cores rose from 27% to 40%. I realise these figures
>> don't tie up with those higher up, but this is a different director with a
>> different mix of services. I have another with no email doing 1.1Gb of
>> traffic and only 15% softirq on the busiest core. Email is expensive to
>> process!
> Regards
> --
> Julian Anastasov <ja at ssi.bg>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

More information about the lvs-users mailing list