[lvs-users] so-called "one arm NAT" LVS-NAT and interaction with netfilter/iptables

Masuda, Bond Bond_Masuda at intuit.com
Thu Jun 26 19:18:28 BST 2014


LVS users,

I have a "one arm NAT" configuration with so-called "local node" that I'm trying to make work with LVS.

I have two servers, syslog servers (call them A and B), and one client (we'll call C). Both A and B also run LVS with keepalived. There is a VIP address managed by keepalived. The idea is that when client C contacts the VIP, it will hit either A or B based on LVS scheduling. I'm doing both UDP and TCP protocols. For purposes of this discussion with respect to packets and the IP addresses, I'll use the format (source IP, destination IP).

When client contacts the VIP, a packet is sent with (C-IP, VIP). It arrives at A that holds the VIP at the moment, and proceeds to LOCAL_IN and LVS-NAT changes the packet to either (C-IP, A-IP) or (C-IP, B-IP). In the former case (C-IP, A-IP) case, it gets processed by the local syslog process. In the later case (C-IP, B-IP), it gets sent back out to B. However, because the packet is (C-IP, B-IP), B's replies go to (B-IP, C-IP) and doesn't get processed by the LVS-NAT anymore. Connection hangs... as the C is expecting reply from (VIP, C-IP).

I know in reading the various FAQ and postings about "one arm NAT" LVS-NAT situations is to have the "real servers" be the default GW. This won't work for me since the real servers are also the LVS servers. However, I was thinking that if I can use netfilter/iptables to source NAT the load balanced packets, I can change the (C-IP,B-IP) show up as (A-IP, B-IP) instead, and hence B's replies would go back to A (the current LVS-NAT) and sent back to C as (VIP, C-IP). However, the packet doesn't seem to traverse the "netfliter stack" as expected:

(i setup a -j LOG rule within every chain [PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING] of every table [mangle, nat, filter], and watch the packet traverse the "netfilter stack")


Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-nat-PRER: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-POSR: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

This is a packet from (C-IP, VIP), goes through NAT/PREROUTING->ROUTING->MANGLE/INPUT->FILTER/INPUT->(LVS-NAT?)->ROUTING->MANGLE/OUTPUT->FILTER/OUTPUT->MANGLE/POSTROUTING.

What I see missing are MANGLE/PREROUTING and particularly interested in the missing NAT/POSTROUTING, where I was hoping to SNAT the packet.

According to this post:

http://archive.linuxvirtualserver.org/html/lvs-users/2011-03/msg00019.html

It would appear that at some point, LVS-NAT packets did traverse NAT/POSTROUTING and one could SNAT the LVS outbound packets?

I'm using:

RHEL 6.5 with RH kernel 2.6.32-431.17.1.el6.x86_64
ipvsadm-1.26-2.el6.x86_64
keepalived-1.2.13-1.el6.x86_64

here are the -j LOG iptables rules I'm using:

[root at sprdrsyas400 ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-PRER: '

Chain INPUT (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-FORW: '

Chain OUTPUT (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-OUTP: '

Chain POSTROUTING (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-POSR: '
[root at sprdrsyas400 ~]# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 8739 packets, 500K bytes)
pkts bytes target     prot opt in     out     source               destination
    7   252 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-PRER: '

Chain POSTROUTING (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-POSR: '

Chain OUTPUT (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-OUTP: '
[root at sprdrsyas400 ~]# iptables -vnL
Chain INPUT (policy ACCEPT 292K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-FORW: '

Chain OUTPUT (policy ACCEPT 361K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-OUTP: '


And this is LVS on A:

[root at sprdrsyas400 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  10.180.88.35:514 rr ops
  -> 10.180.88.29:514            Local   1      0          0
  -> 10.180.88.30:514            Masq    1      0          0

I would appreciate any help...

Thanks in advance,
-Bond



More information about the lvs-users mailing list