[lvs-users] ipvsadm is not forwarding connections

Anders Henke anders.henke at 1und1.de
Mon Jun 23 19:22:36 BST 2014


On 23.06.2014, Dennis Jacobfeuerborn wrote:
> On 23.06.2014 11:57, Anders Henke wrote:
> > On 18.06.2014, Stephen Carville wrote:
> >> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
> >> get it to forward connections. When I try to connect, it doesn't write
> >> anything in /var/log/messages to tell me what is happening. Netstat
> >> doesn't see anything listening on the interface IP (I read elsewhere
> >> that is normal) and tshark sees the incoming SYN but there is either a
> >> timeout or a RST.
> >>
> >> Rules right now:
> >>
> >> $ ipvsadm -L
> >>
> >> IP Virtual Server version 1.2.1 (size=4096)
> >> Prot LocalAddress:Port Scheduler Flags
> >>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
> >> TCP  10.212.160.40:4172 lc persistent 360
> >>   -> 10.212.170.162:4172          Route   1      0          0
> >>   -> 10.212.170.163:4172          Route   1      0          0
> >>
> >> IP forwarding is turned on:
> >>
> >> $ sysctl net.ipv4.ip_forward
> >> net.ipv4.ip_forward = 1
> > 
> > Short answer: switch to kernel 3.6 or newer, turn off rp_filter for the interface receiving the reply packet, and replace rp_filter functionality by more accurate and flexible iptables rules in the FORWARD chain.
> 
> Since he is running CentOS 6.5 he can simply set
> /proc/sys/net/ipv4/conf/<interface>/accept_local to 1 to prevent packets
> from being dropped as martians. This was introduced in 2.6.33 but
> backported to recent RHEL/CentOS kernels so no need to go to 3.6 or newer.
> You still have to set the rp_filter though since this is a different
> isssue than the martian packet one.

It's a little bit more complicated :-)

In this setup, the VIP is configured on eth0, but the realserver is contacted via eth1. The realserver will most likely to reply via eth1 as well.

Upstream network equipment will route incoming traffic for the VIP to eth0, and by seperating the realservers onto a dedicated network, we're removing the need for handling the ARP problem with realserver configuration. As long as no realservers are connected to eth0, the realservers don't need to disable ARP replies at all.

Regarding kernel configuration, accept_local is only revelant if rp_filter is set.

-if eth1 has both rp_filter=1 and accept_local=1, the kernel 
 checks if the IP address is routed and local to eth1, otherwise it is being
 dropped. The VIP is configured on eth0, so the packet is being discarded.
-If eth1 has rp_filter=2 (loose mode) and accept_local=1, the packet 
 is accepted.

 Loose mode doesn't check if the incoming interface matches the routing 
 decision, just if "any" route on the host, including a default route, 
 does know where to send this packet may be sent to.

 So whenever your system does have a default route configured, 
 rp_filter=2 for any interface other than your default interface 
 is almost equivalent to rp_filter=0.
 The only difference is the handling of addresses local to the balancer,
 and by configuring accept_local=1 for that interface, one is also removing
 that single difference.
 In the end, you don't have any kind of reverse path filtering for 
 that specific interface, but you've just added some check to 
 your kernel which will always return true, but merely "looks" like
 some added level of security.

 That's part of the reason why in such a situation, I'm advocating to
 de-configure rp_filter for that specific interface: you don't gain
 security, you're just adding a useless check. If you do want to
 improve security, you need to add the expected or aimed level of 
 security by dedicated iptables forwarding rules.

=== Alternative option

Another option is to configure the VIP to be hosted on eth1,
configure all interfaces with rp_filter=1 (default on CentOS 6.5)
and additionally configure eth1 with accept_local=1:

$ ip address del 10.212.160.40/32 dev eth0
$ ip address add 10.212.160.40/32 dev eth1
$ sysctl -w net.ipv4.conf.all.rp_filter = 1
$ sysctl -w net.ipv4.conf.eth1.accept_local = 1

Due to Linux' weak host model, packets for 10.212.160.40 will be accepted
on any interface and sent to the realserver. The realserver replies are 
received at an interface hosting that IP address, so accept_local will match
and forward replies as intended.

Downside: most network engineers looking at this configuration instinctively
will assume a misconfiguration. It's not that intuitive, as it's relying on
the weak host model. However, it's consistent with most realserver 
configurations and does have the benefits of strict mode reverse path 
filtering.

As there are no forwarding restrictions per default other than the just 
configured reverse path filtering and we're running on ip_forward=1, one 
should still take care of the exact forwarding configuration, e.g. by 
adding iptables rules to the forwarding chain. Otherwise, a malicious host 
on eth0 may access any host running behind eth1 just by adding the 
balancer as a router.

Specific iptables forwarding rules or selectively enabling forwarding only for eth1 does remove that risk. However, the later one is very specific to the exact realserver network configuration (you need to enable forwarding on eth0 to forward traffic from eth0 to eth1, if your balancer acts as the only router for your realservers).

So if one is adding fine-granular forwarding rules for a specific interface anyway, which are tighter than rp_filter for that interface - why care about rp_filter for eth1?



Best,

Anders
-- 
1&1 Internet AG              Expert Systems Architect (IT Operations)
Brauerstrasse 50             v://49.721.91374.0
D-76135 Karlsruhe            f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, 
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, 
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren



More information about the lvs-users mailing list