[lvs-users] Wrong RIP is used for fwmark-based groups pointing to different ports and protocols of the same RIP

Julian Anastasov ja at ssi.bg
Sat Jan 5 21:11:24 GMT 2019


	Hello,

On Tue, 11 Dec 2018, Bogdan Rudas wrote:

> I try to do load-balance the service which is listening both TCP and UDP
> using same port number *1111*. There are set of real servers, each of them
> host few application instances listening on different ports (1111/udp,
> 1116/udp, 1115/tcp, 1117/tcp)
> I used fwmark to group connection by protocol, i.e. 1111/udp have *fwmark 1*,
> 1111/tcp have *fwmark 2.*
> LVS-NAT mode is in use.
> What I see is when the client connection coming to 1111/*TCP* the LVS try
> to pass it to RTI:1111/*TCP *while the RIP have TCP listeners on ports 1115
> and 1117 and LVS was configured accordingly.

	This is a long-standing behaviour to ignore rport for real servers
added to persistent fwmark-based virtual service. What can be changed is 
to depend on rport being 0 or not:

# Map services 1:1 (port is preserved)
-r RIP:0 => translate only daddr=VIP->RIP
# Map all vports to same rport
-r RIP:RPORT => translate both address and port

	But we risk to change behaviour to existing installations.

> Here is test connection info:
> # ipvsadm -L -n -c
> IPVS connection entries
> pro expire state       source             virtual            destination
> *TCP* 00:55  SYN_RECV    113.25.79.34:38204 115.5.6.138*:1111 *10.1.2.13
> *:1111* *<<<< It should be RIP:1115 or RIP:1117 HERE!*
> IP  00:49  NONE        113.25.79.34:0   0.0.0.2:0          10.1.2.13:0
> TCP 01:00  SYN_RECV    113.25.79.34:38206 115.5.6.138:1111 10.1.2.13:1111
> 
> Below are saved firewall and ipvs configurations:
> 
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *mangle
> :PREROUTING ACCEPT [9507:869411]
> :INPUT ACCEPT [7637:700414]
> :FORWARD ACCEPT [70:10945]
> :OUTPUT ACCEPT [8781:1045717]
> :POSTROUTING ACCEPT [8851:1056662]
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [82:11111]
> :OUTPUT ACCEPT [16923:2797363]
> -A INPUT -p icmp -m comment --comment "000 accept all icmp" -j ACCEPT
> -A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j
> ACCEPT
> -A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "002 reject local
> traffic not on loopback interface" -j REJECT --reject-with
> icmp-port-unreachable
> -A INPUT -m comment --comment "003 accept related established rules" -m
> state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "100 accept
> ssh connection" -m state --state NEW -j ACCEPT
> -A INPUT -d 224.0.0.0/8 -p vrrp -m comment --comment "104 Accept VRRP
> proto" -j ACCEPT
> -A INPUT -m comment --comment "999 drop all" -j REJECT --reject-with
> icmp-port-unreachable
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *nat
> :PREROUTING ACCEPT [42:2928]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [4:360]
> :POSTROUTING ACCEPT [4:360]
> -A PREROUTING -p tcp -m multiport --dports 1111 -j MARK --set-xmark
> 0x2/0xffffffff
> -A PREROUTING -p *udp* -m multiport --dports 1111 -m comment --comment "101
> MARK udp 1111 traffic from everywhere" -j MARK --set-xmark *0x1*/0xffffffff
> -A PREROUTING -p *tcp* -m multiport --dports 1111 -m comment --comment "101
> MARK tcp 1111 traffic from everywhere" -j MARK --set-xmark *0x2*/0xffffffff
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *raw
> :PREROUTING ACCEPT [17569:3914462]
> :OUTPUT ACCEPT [16925:2798776]
> COMMIT
> 
> # ipvsadm -S
> -A -f 1 -s wrr -p 50
> -a -f 1 -r 10.1.2.12:1111 -m -w 2
> -a -f 1 -r 10.1.2.12:1116 -m -w 2

	This is not supported: many real services on same real IP
when persistence is used for fwmark. Because: packets to different vport 
can be marked with same fwmark to be handled by same virtual service.
As result, the fwmark template <IPPROTO_IP,caddr,0,fwmark,0,daddr,0> does 
not use dport from the packet and we can not distinguish the real services
that differ just by rport.

	In short, virtual service that can handle traffic to different
vports can not use persistence together with many real services on same 
IP.

> -a -f 1 -r 10.1.2.13:1111 -m -w 2
> -a -f 1 -r 10.1.2.13:1116 -m -w 2
> -a -f 1 -r 10.1.2.14:1111 -m -w 2
> -a -f 1 -r 10.1.2.14:1116 -m -w 2
> -A -f 2 -s wrr -p 50
> -a -f 2 -r 10.1.2.12:1115 -m -w 2
> -a -f 2 -r 10.1.2.12:1117 -m -w 2
> -a -f 2 -r 10.1.2.13:1115 -m -w 2
> -a -f 2 -r 10.1.2.13:1117 -m -w 2
> -a -f 2 -r 10.1.2.14:1115 -m -w 2
> -a -f 2 -r 10.1.2.14:1117 -m -w 2
> 
> 
> # ipvsadm -L -n
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
> FWM  1 wrr persistent 50
>   -> 10.1.2.12:1111           Masq    2      0          0
>   -> 10.1.2.12:1116           Masq    2      0          0
>   -> 10.1.2.13:1111           Masq    2      0          0
>   -> 10.1.2.13:1116           Masq    2      0          0
>   -> 10.1.2.14:1111           Masq    2      0          0
>   -> 10.1.2.14:1116           Masq    2      0          0
> FWM  2 wrr persistent 50
>   -> 10.1.2.12:1115           Masq    2      0          0
>   -> 10.1.2.12:1117           Masq    2      0          0
>   -> 10.1.2.13:1115           Masq    2      0          0
>   -> 10.1.2.13:1117           Masq    2      0          1
>   -> 10.1.2.14:1115           Masq    2      0          0
>   -> 10.1.2.14:1117           Masq    2      0          0
> 
> 
> OS is CentOS 7 with kernel *3.10*
> 
> Thank you.

Regards

--
Julian Anastasov <ja at ssi.bg>



More information about the lvs-users mailing list