[lvs-users] strange tcp issue

L.S. Keijser leon at linux.nl
Tue Jul 16 11:00:59 BST 2013


Hi,

First of all, this might have nothing to do with LVS, but I'm exploring
all options. Hopefully someone here can point me in the right direction.

The setup:

- 2 directors in a pacemaker cluster with floating ip's etc.
- some realservers behind it

Half of the connections are handled by LVS, the other half is done by
Varnish (running locally on the director). 

What we observer when there's a large number of connections (OpenNMS
reports over 400 requests p/sec), a client sending a SYN sometimes waits
a long time for a SYN/ACK to get send by the server. I've experienced
waiting for more than a minute for the SYN/ACK to arrive. 

I see on the directory that my SYN packets do arrive. The host just
doesn't do anything with them for quite some time. Here's a small
snippet from the director:

21:25:44.557421 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135062813 ecr
0,nop,wscale 7], length 0
21:25:45.546065 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135063816 ecr
0,nop,wscale 7], length 0
21:25:47.548218 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135065820 ecr
0,nop,wscale 7], length 0
21:25:51.554730 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135069824 ecr
0,nop,wscale 7], length 0
21:25:59.570857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135077840 ecr
0,nop,wscale 7], length 0
21:25:59.570886 IP y.y.y.y.80 > x.x.x.x.43369: Flags [S.], seq
548329830, ack 1941249137, win 5792, options [mss 1460,sackOK,TS val
2126658556 ecr 135077840,nop,wscale 7], length 0
21:25:59.592085 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 135077873 ecr 2126658556], length 0
21:25:59.592097 IP x.x.x.x.43369 > y.y.y.y.80: Flags [P.], seq 1:105,
ack 1, win 115, options [nop,nop,TS val 135077873 ecr 2126658556],
length 104
21:25:59.592124 IP y.y.y.y.80 > x.x.x.x.43369: Flags [.], ack 105, win
46, options [nop,nop,TS val 2126658561 ecr 135077873], length 0
21:25:59.592389 IP y.y.y.y.80 > x.x.x.x.43369: Flags [P.], seq 1:384,
ack 105, win 46, options [nop,nop,TS val 2126658562 ecr 135077873],
length 383
21:25:59.622844 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 384, win
123, options [nop,nop,TS val 135077909 ecr 2126658562], length 0
21:25:59.622857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [F.], seq 105, ack
384, win 123, options [nop,nop,TS val 135077909 ecr 2126658562], length
0
21:25:59.622893 IP y.y.y.y.80 > x.x.x.x.43369: Flags [F.], seq 384, ack
106, win 46, options [nop,nop,TS val 2126658569 ecr 135077909], length 0
21:25:59.639766 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 385, win
123, options [nop,nop,TS val 135077926 ecr 2126658569], length 0

x.x.x.x = my client
y.y.y.y = IP on the director

As you see, the first SYN gets sent at 21:25:44 and only gets a SYN/ACK
reply at 21:25:59. After that, the communication is as expected.


After doing some reading I've made the following adjustments to sysctl :

net.ipv4.ip_local_port_range = 18000    65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0

I don't think the problem is on the director's side, but I'm not sure.
The fact that i see SYN packets coming in as I send them, and the host
not responding to them, makes me doubt myself again ..

Any advice is most welcome.

Thanks,

Léon




More information about the lvs-users mailing list