[lvs-users] LVS-SNAT only works locally (ubuntu 11.04 amd64)

Tom van Leeuwen tom.van.leeuwen at saasplaza.com
Fri May 27 13:20:27 BST 2011


Hi list, this is my first post, so please be gentle.
I'm trying to get a lvs (lab)setup to work with snat.
I've been reading several posts and the mailinglist, and they told that 
the snat should work with kernel 2.6.36 and higher.

Therefor I have installed ubuntu 11.04 amd64 which has the following 
packages:
loadbalancer-ng ~ # uname -a
Linux loadbalancer-ng 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 
UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
loadbalancer-ng ~ # iptables -V
iptables v1.4.10
loadbalancer-ng ~ # ipvsadm -v
ipvsadm v1.25 2008/5/15 (compiled with popt and IPVS v1.2.1)

The setup is the following:

+--------------------------+
|      linux router        |
+------------+-------------+  ip 172.16.31.1/24
              |
              |
+------------+-------------+  ip 172.16.31.10/24
|     lvs loadbalancer     |  VIP: 172.16.31.10:80
+------------+-------------+  ip 172.16.29.10/24
              |
              |
+------------+-------------+  ip 172.16.29.5/24
|    linux realserver      |  RIP: 172.16.29.5:80
+--------------------------+

loadbalancer-ng ~ # ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.31.10:80 rr
   -> 172.16.29.5:80               Masq    1      0          0


When I do the following:
router ~ # wget -O /dev/null http://172.16.31.10:80 -q; echo $?
0

I can see the following traffic on the real:
real www # tcpdump -i eth1 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
14:10:22.070213 IP 172.16.31.1.60572 > 172.16.29.5.80: Flags [S], seq 
531016714, win 5840, options [mss 1460,sackOK,TS val 2116681 ecr 
0,nop,wscale 5], length 0
14:10:22.070241 IP 172.16.29.5.80 > 172.16.31.1.60572: Flags [S.], seq 
511492588, ack 531016715, win 5792, options [mss 1460,sackOK,TS val 
2097489 ecr 2116681,nop,wscale 5], length 0
14:10:22.073096 IP 172.16.31.1.60572 > 172.16.29.5.80: Flags [.], ack 1, 
win 183, options [nop,nop,TS val 2116681 ecr 2097489], length 0
14:10:22.073385 IP 172.16.31.1.60572 > 172.16.29.5.80: Flags [P.], seq 
1:111, ack 1, win 183, options [nop,nop,TS val 2116681 ecr 2097489], 
length 110
14:10:22.073400 IP 172.16.29.5.80 > 172.16.31.1.60572: Flags [.], ack 
111, win 181, options [nop,nop,TS val 2097489 ecr 2116681], length 0
14:10:22.073507 IP 172.16.29.5.80 > 172.16.31.1.60572: Flags [P.], seq 
1:245, ack 111, win 181, options [nop,nop,TS val 2097489 ecr 2116681], 
length 244
14:10:22.073606 IP 172.16.29.5.80 > 172.16.31.1.60572: Flags [F.], seq 
245, ack 111, win 181, options [nop,nop,TS val 2097489 ecr 2116681], 
length 0
14:10:22.076351 IP 172.16.31.1.60572 > 172.16.29.5.80: Flags [.], ack 
245, win 216, options [nop,nop,TS val 2116681 ecr 2097489], length 0
14:10:22.076793 IP 172.16.31.1.60572 > 172.16.29.5.80: Flags [F.], seq 
111, ack 246, win 216, options [nop,nop,TS val 2116681 ecr 2097489], 
length 0
14:10:22.076830 IP 172.16.29.5.80 > 172.16.31.1.60572: Flags [.], ack 
112, win 181, options [nop,nop,TS val 2097489 ecr 2116681], length 0

It is NOT SOURCE NATTED.

When I do the following:
loadbalancer-ng ~ # wget -O /dev/null http://172.16.31.10:80 -q; echo $?
0

real www # tcpdump -i eth1 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
14:11:16.511561 IP 172.16.29.10.40215 > 172.16.29.5.80: Flags [S], seq 
1416440597, win 32792, options [mss 16396,sackOK,TS val 2165715 ecr 
0,nop,wscale 5], length 0
14:11:16.511594 IP 172.16.29.5.80 > 172.16.29.10.40215: Flags [S.], seq 
1375362407, ack 1416440598, win 5792, options [mss 1460,sackOK,TS val 
2102933 ecr 2165715,nop,wscale 5], length 0
14:11:16.512355 IP 172.16.29.10.40215 > 172.16.29.5.80: Flags [.], ack 
1, win 1025, options [nop,nop,TS val 2165715 ecr 2102933], length 0
14:11:16.513378 IP 172.16.29.10.40215 > 172.16.29.5.80: Flags [P.], seq 
1:111, ack 1, win 1025, options [nop,nop,TS val 2165715 ecr 2102933], 
length 110
14:11:16.513402 IP 172.16.29.5.80 > 172.16.29.10.40215: Flags [.], ack 
111, win 181, options [nop,nop,TS val 2102933 ecr 2165715], length 0
14:11:16.513538 IP 172.16.29.5.80 > 172.16.29.10.40215: Flags [P.], seq 
1:245, ack 111, win 181, options [nop,nop,TS val 2102933 ecr 2165715], 
length 244
14:11:16.513590 IP 172.16.29.5.80 > 172.16.29.10.40215: Flags [F.], seq 
245, ack 111, win 181, options [nop,nop,TS val 2102933 ecr 2165715], 
length 0
14:11:16.514541 IP 172.16.29.10.40215 > 172.16.29.5.80: Flags [.], ack 
245, win 1059, options [nop,nop,TS val 2165715 ecr 2102933], length 0
14:11:16.515023 IP 172.16.29.10.40215 > 172.16.29.5.80: Flags [F.], seq 
111, ack 246, win 1059, options [nop,nop,TS val 2165715 ecr 2102933], 
length 0
14:11:16.515040 IP 172.16.29.5.80 > 172.16.29.10.40215: Flags [.], ack 
112, win 181, options [nop,nop,TS val 2102933 ecr 2165715], length 0

It IS SOURCE NATTED!!! What I've noticed is that the POSTROUTING rules 
are not hit in the first tcpdump.

Other relevant configuration needed for this setup (correct me if I'm 
wrong):

loadbalancer-ng ~ # iptables -vnL POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               
destination
     1    60            all  --  *      *       0.0.0.0/0            
0.0.0.0/0
     6   360 SNAT       all  --  *      *       0.0.0.0/0            
0.0.0.0/0           vaddr 172.16.31.10 vport 80 to:172.16.29.10

loadbalancer-ng ~ # sysctl net.ipv4.vs.conntrack
net.ipv4.vs.conntrack = 1

loadbalancer-ng ~ # sysctl -a 2> /dev/null | grep \.forwarding | grep -v 
mc_forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.eth2.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.lo.forwarding = 1
net.ipv6.conf.eth0.forwarding = 1
net.ipv6.conf.eth1.forwarding = 1
net.ipv6.conf.eth2.forwarding = 1


loadbalancer-ng ~ # lsmod
Module                  Size  Used by
xt_string              12586  0
xt_mark                12563  0
xt_DSCP                12629  0
xt_dscp                12597  0
xt_multiport           12597  0
xt_hashlimit           17698  0
xt_owner               12498  0
xt_iprange             12541  0
xt_NFQUEUE             12699  0
ipt_addrtype           12599  0
iptable_filter         12810  0
xt_tcpudp              12603  0
ipt_REJECT             12576  0
iptable_nat            13182  1
iptable_mangle         12734  0
ip_tables              27456  3 iptable_filter,iptable_nat,iptable_mangle
xt_CHECKSUM            12549  0
ebtables               30915  0
ip6table_filter        12815  0
ip6_tables             27845  1 ip6table_filter
ipt_LOG                17016  0
xt_conntrack           12728  0
xt_connmark            12755  0
xt_state               12578  0
ipt_MASQUERADE         12759  0
nf_nat                 25736  2 iptable_nat,ipt_MASQUERADE
nf_conntrack_ipv4      19640  3 iptable_nat,nf_nat
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
xt_ipvs                12536  1
x_tables               29545  26 
xt_string,xt_mark,xt_DSCP,xt_dscp,xt_multiport,xt_hashlimit,xt_owner,xt_iprange,xt_NFQUEUE,ipt_addrtype,iptable_filter,xt_tcpudp,ipt_REJECT,iptable_nat,iptable_mangle,ip_tables,xt_CHECKSUM,ebtables,ip6table_filter,ip6_tables,ipt_LOG,xt_conntrack,xt_connmark,xt_state,ipt_MASQUERADE,xt_ipvs
ip_vs_rr               12602  1
ip_vs                 137211  4 xt_ipvs,ip_vs_rr
nf_conntrack           81956  8 
iptable_nat,xt_conntrack,xt_connmark,xt_state,ipt_MASQUERADE,nf_nat,nf_conntrack_ipv4,ip_vs
libcrc32c              12644  1 ip_vs
snd_intel8x0           38272  0
snd_ac97_codec        134270  1 snd_intel8x0
ac97_bus               12730  1 snd_ac97_codec
snd_pcm                96531  2 snd_intel8x0,snd_ac97_codec
snd_timer              29602  1 snd_pcm
psmouse                73535  0
serio_raw              13166  0
snd                    67346  4 
snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
virtio_balloon         13153  0
soundcore              12680  1 snd
snd_page_alloc         18529  2 snd_intel8x0,snd_pcm
i2c_piix4              13303  0
lp                     17789  0
parport                46458  1 lp
floppy                 74120  0

There's obviously something wrong, but I cannot find it.
This is the loadbalancer configuration:
iptables -t nat -A POSTROUTING -m ipvs --vaddr 172.16.31.10 --vport 80 
-j SNAT --to-source 172.16.29.10
ipvsadm -A -t 172.16.31.10:80 -s rr
ipvsadm -a -t 172.16.31.10:80 -r 172.16.29.5:80 -m

I hope someone can tell me why this does not work as expected.

Another question I have: I would like to have a subnet for my VIPS. Is 
it necessary to configure all VIPS as alias IP's on the loadbalancer to 
get it to work?




More information about the lvs-users mailing list