LVS-NAT: intermittent problem with responses from realservers

Graeme Fowler graeme at graemef.net
Tue Oct 19 21:25:58 BST 2004


On Thu, 14 Oct 2004, Horms wrote:
> You should get a "not hit" for all packets that come through the
> box that are not related to LVS, and a "hit" for all packets that are.
> If you think that packets are getting "not hit" when they should get
> "hit" then try looking at ipvsadm -Lcn, this should give you a dump
> of the connections that LVS is trying to match incoming packets against.

That might be quite difficult, given the rate of traffic through these boxes!

Details:
Fedora Core 2
kernel 2.6.8-1.521 (recompiled/repackaged to switch on IP_VS_DEBUG)
keepalived 1.1.7
ipvsadm 1.24
ip_vs version 1.2.0 (size=4096) from kernel package
LVS-NAT
persistence 120 sec

This should make it look fairly obvious. The following sanitised logfile shows 
a connection from a remote NMS poller failing to get through correctly. I'll 
annotate it in sections.

00:28:16  IPVS: lookup/in TCP $NMS_ADDR:52059->$VIP_ADDR:80 hit
00:28:16  IPVS: Incoming packet: TCP $NMS_ADDR:52059->$VIP_ADDR:80
00:28:16  IPVS: TCP input  [S...] $RS2_ADDR:80->$NMS_ADDR:0 state: 
NONE->SYN_RECV cnt:2
00:28:16  IPVS: After DNAT: TCP $NMS_ADDR:52059->$RS2_ADDR:80

OK so far, connection arrives to a previously stored connection from 60 or so 
seconds ago. Packet gets sent off to the RS, but then it comes back and...

00:28:16  IPVS: lookup/out TCP $RS2_ADDR:80->$NMS_ADDR:52059 not hit
00:28:16  IPVS: packet continues traversal as normal: TCP 
$RS2_ADDR:80->$NMS_ADDR:52059
00:28:16  IPVS: lookup/out TCP $NMS_ADDR:52059->$RS2_ADDR:80 not hit
00:28:16  IPVS: packet continues traversal as normal: TCP 
$NMS_ADDR:52059->$RS2_ADDR:80

...the packets aren't getting rewritten by the ip_vs code, so the client isn't 
seeing them. It then sends another SYN which gets sent in to the RS:

00:28:19  IPVS: lookup/in TCP $NMS_ADDR:52059->$VIP_ADDR:80 hit
00:28:19  IPVS: Incoming packet: TCP $NMS_ADDR:52059->$VIP_ADDR:80
00:28:19  IPVS: After DNAT: TCP $NMS_ADDR:52059->$RS2_ADDR:80

...but again it misses the bucket and gets passed to the netfilter code 
instead of ip_vs:

00:28:19  IPVS: lookup/out TCP $RS2_ADDR:80->$NMS_ADDR:52059 not hit
00:28:19  IPVS: packet continues traversal as normal: TCP 
$RS2_ADDR:80->$NMS_ADDR:52059
00:28:19  IPVS: lookup/out TCP $NMS_ADDR:52059->$RS2_ADDR:80 not hit
00:28:19  IPVS: packet continues traversal as normal: TCP 
$NMS_ADDR:52059->$RS2_ADDR:80

...and so on. I have miles and miles of this happening occasionally.

I hope that I haven't missed anything obvious in my interpretation of the 
logs.

Any hints or pointers greatly received!

Graeme



More information about the lvs-users mailing list