[lvs-users] IPVS uses loopback address for tunneling outer source IP

[Julian Anastasov]

	Hello,

On Wed, 29 Aug 2018, Yunzhao li wrote:

> We are seeing some very strange thing on IPVS tunnel mode(we are
> running centos 7 with kernel 3.10.0):
> 
> Occasionally, the tcpdump shows that IPVS changes tunnelled outer
> source IP address to a loopback address, like 127.146.255.255, and the
> tunnelled packets had been dropped . However, most of time, the outer
> source IP address is the normal IP address bind to the physical
> ethernet interface. When this happened, we dumped the routing tables
> and iptables, and it looks this issue is not due to either routing
> setting or a iptables' SNAT. Since this issue happened on two hosts,
> we also doubt that this is a hardware problem. We also briefly checked
> the ip_vs_tunnel_xmit() call and it looks it retrieves outer header
> information from kernel's routing part. We are wondering whether this
> is a desired behaviour of IPVS, due to IPVS doesn't want to send out
> packets at that time?

	Looks like a bug already fixed in Linux 4.2, 3.10.91, 3.18.23:

commit 4754957f04f5f368792a0eb7dab0ae89fb93dcfd
Author: Julian Anastasov <ja at ssi.bg>
Date:   Sat Jun 27 14:39:30 2015 +0300

    ipvs: do not use random local source address for tunnels
    
    Michael Vallaly reports about wrong source address used
    in rare cases for tunneled traffic. Looks like
    __ip_vs_get_out_rt in 3.10+ is providing uninitialized
    dest_dst->dst_saddr.ip because ip_vs_dest_dst_alloc uses
    kmalloc. While we retry after seeing EINVAL from routing
    for data that does not look like valid local address, it
    still succeeded when this memory was previously used from
    other dests and with different local addresses. As result,
    we can use valid local address that is not suitable for
    our real server.
    
    Fix it by providing 0.0.0.0 every time our cache is refreshed.
    By this way we will get preferred source address from routing.
    
    Reported-by: Michael Vallaly <lvs at nolatency.com>
    Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
    Signed-off-by: Julian Anastasov <ja at ssi.bg>
    Signed-off-by: Simon Horman <horms at verge.net.au>

diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index b99d80695b1f..ec30d68ccc0b 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -130,7 +130,6 @@ static struct rtable *do_output_route4(struct net *net, __be32 daddr,
 
 	memset(&fl4, 0, sizeof(fl4));
 	fl4.daddr = daddr;
-	fl4.saddr = (rt_mode & IP_VS_RT_MODE_CONNECT) ? *saddr : 0;
 	fl4.flowi4_flags = (rt_mode & IP_VS_RT_MODE_KNOWN_NH) ?
 			   FLOWI_FLAG_KNOWN_NH : 0;

Regards

--
Julian Anastasov <ja at ssi.bg>