[lvs-users] LVS-TUN BUG? - Intermittent incorrect source IP in TUN header for non-local realsevers (PBR no help)

Michael Vallaly lvs at nolatency.com
Wed Jun 17 23:04:51 BST 2015

Hello fellow LVS/IPVS users..

I have been running IPVS for a few years with great success but seem to
have recently run into a very perplexing problem in TUN mode with what
seems to be stale/leaked routing lookups?

On a IPVS director with multiple interfaces (and policy base routing)
it seems IPVS TUN code will emit IPIP packets from the correct
interface (according to PBR) but use/select the wrong source IP in the
TUN IP header. This is reproducible on our systems, but is _VERY_
infrequent (eg. 354 out of 1.5mil packets had an incorrect/invalid
source IP on our system in the last 24 hours)

Details of the failing IPVS director as follows:

uname -r
3.18.14 (mainline vanilla)

head -1 /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=32768)

ip route show
default via dev vlan200 dev vlan10  proto kernel  scope link  src dev vlan20  proto kernel  scope link  src dev vlan200  proto kernel  scope link  src dev vlan500  proto kernel  scope link  src

ip rule show
0:      from all lookup local
32764:  from all to lookup 3
32765:  from lookup 3
32766:  from all lookup main
32767:  from all lookup default

ip route show table 3
default via dev vlan500  src

ipvsadm -Ln
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  254 wlc
  ->              Tunnel  1      0          1528
FWM  255 wlc
  ->               Tunnel  1      0          35


In the following examples I have a remote client ( attempting
to hit a LVS-TUN service (, the packets are marked in
iptables mangle and are matched by IPVS using FWMARK (254 or 255).

IPVS clusters with realservers on locally connected networks (FWM 255),
the emitted TUN packets seems to work as expected 100% of the time. Eg.
IPVS encapsulates the original packet (src dst in a IP
header (src dst and emits the packet via
vlan10 interface.

tcpdump -n -i vlan10
IP > IP > Flags
[S], seq 2658459997, win 65535, options [mss 1460,nop,wscale
5,nop,nop,TS val 832819348 ecr 0,sackOK,eol], length 0 (ipip-proto-4)

Alternatively if the inbound traffic is handled by a IPVS cluster with
remote L3 realservers, things work most of the time, then the
unexpected happens)

IPVS clusters with realservers on a remote L3 network (FWM 254), IPVS
encapsulates the original packet (src dst in a IP
header (src dst and emits the packet via
the vlan500 interface passing it to the L2 (mac address) of 

This worked swimmingly well until I noticed that very intermittently
the following happens:

IPVS encapsulates the original packet (src dst in a IP
header (src dst and emits the packet via
the vlan500 interface passing it to the mac address of
(NOTE: the use of rather than the expected
source IP in the TUN header)

Example of broken packet via tcpdump
tcpdump -n -i vlan500
IP > IP > Flags
[S], seq 1418130791, win 65535, options [mss 1386,nop,wscale
5,nop,nop,TS val 1213718660 ecr 0,sackOK,eol], length 0 (ipip-proto-4)

So in the last 24 hours out of 1.5M packets emitted by IPVS on vlan500
(FWM 254) I had 349 packets which get emitted with the wrong source IP
address in the Tunnel IP header. The periods where the wrong source IP
is used by IPVS seem to last for ~2-5min at a time, and affects all
traffic in the LVS cluster with remote L3 realservers. 

Another interesting meta-point is, on a LVS director with lots of
interfaces and active IPVS clusters, I only see IPVS select invalid
source IPs which meet the following criteria:

1. The IP exists on on of the directors local interfaces
2. The IP is currently used as a source IP for TUN traffic in other
IPVS clusters that are currently active (passing TUN traffic to local

I am guessing this is related to the "saddr" address being stale/not
reinitialized/leaked in the do_output_route4 function of ip_vs_xmit.c? 

Im not sure what would cause this type of issue in IPVS, or if its
really related to recent routing changes in the kernel?

Anyone have thoughts/suggestions?



Michael Vallaly <lvs at nolatency.com>

More information about the lvs-users mailing list