Problems getting LVS to work
Mark Wadham
mark.wadham at areti.net
Thu Mar 29 14:51:49 BST 2007
Hi Roberto,
Thanks for the quick response!
Roberto Nibali wrote:
> Hi Mark,
>
> Excellent problem report!
>
*takes a bow*
>> We have a load balancer (with the lvs kernel stuff) at 100.1.1.1,
>> with a second IP address 100.1.1.2.
>>
>> We have two mail servers at 120.1.1.1 and 120.1.1.2. The load
>> balancer is supposed to balance connections between the two
>> mailservers. We have another load balancer at 130.1.1.1 which works
>> fine, but the new load balancer is set up seemingly the same and yet
>> it just does not work.
>>
>> Load balancer configuration (100.1.1.2)
>> ===========================
>> net.ipv4.conf.all.forwarding = 1
>>
>> eth0 has 100.1.1.1, eth0:0 has 100.1.1.2
>
> And their netmasks are 24, resp. 32?
Yes
>
>> # ipvsadm --list -n
>> IP Virtual Server version 1.2.1 (size=4096)
>> Prot LocalAddress:Port Scheduler Flags
>> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>> TCP 100.1.1.2:25 wlc
>> -> 120.1.1.1:25 Tunnel 1 0 0
>> -> 120.1.1.2:25 Tunnel 1 0 0
>>
>> iptables has no rules and is default-to-accept. There is no firewall
>> in front of the box.
>>
>> Mail server 1 (120.1.1.1)
>> =================
>>
>> relevant iptables rules:
>>
>> $IPTABLES -A INPUT -i eth0 -s 100.1.1.2 -p ipencap -j ACCEPT
>> $IPTABLES -A INPUT -i tunl0 -p tcp --dport smtp -j ACCEPT
>
> Why do you need those rules if you're not having any netfilter rules
> and a ACCEPT policy?
>
The mailservers _do_ have firewall rules, its just the new load balancer
that does not. However, I don't think this is a firewall issue as
dropped packets still show up in tcpdump, and also I am able to telnet
directly to port 25 on both mailservers from the new (broken) load balancer.
>> Mail daemon listening on all IPs:
>>
>> # netstat -natp |grep TEN |grep 25
>> tcp 0 0 0.0.0.0:25 0.0.0.0:*
>> LISTEN 14505/exim4
>
> Excellent.
>
>> tunl0:0 is the tunnel interface for the existing load balancer (that
>> works)
>>
>> # ifconfig tunl0:0
>> tunl0:0 Link encap:IPIP Tunnel HWaddr
>> inet addr:130.1.1.2 Mask:255.255.255.255
>> UP RUNNING NOARP MTU:1480 Metric:1
>>
>> tunl0:3 is the tunnel interface for the new load balancer that
>> doesn't work
>>
>> # ifconfig tunl0:3
>> tunl0:3 Link encap:IPIP Tunnel HWaddr
>> inet addr:100.1.1.2 Mask:255.255.255.255
>> UP RUNNING NOARP MTU:1480 Metric:1
>>
>> Mail server 2 (120.1.1.2)
>> =================
>> Same as mailserver 1
>>
>>
>> The current load balancer at 130.1.1.1 uses 130.1.1.2 for load
>> balancing inbound smtp/25 connections to the two mailservers. If i
>> telnet to 130.1.1.2 from my work machine at 140.1.1.1, this is the
>> tcpdump sequence:
>
> I'm a bit confused by your obfuscation technique :), what's the
> designation for the servers regarding the obfuscated IP ranges in
> 100.x.x.x, the 120.x.x.x, the 130.x.x.x and the 140.x.x.x?
>
> 140: your test machine
> 130: working LVS tunnel
> 120: RS (mail server)
> 100: new (non-functional) LVS tunnel
>
> Is my observation correct?
>
Yes, sorry for the obfuscation - I was all for just pasting the real IPs
but my manager refused to let me ;)
>> load 1
>>
>> 10:28:34.904382 IP 140.1.1.1.3948 > 130.1.1.2.25: S
>> 3712043866:3712043866(0) win 65535 <mss 1260,nop,nop,sackOK>
>> 10:28:34.909107 IP 140.1.1.1.3948 > 130.1.1.2.25: . ack 151923593 win
>> 65535
>> 10:28:55.134362 IP 140.1.1.1.3948 > 130.1.1.2.25: . ack 52 win 65484
>>
>> mail 2 eth0 (this could be mail 1 here, in the example the
>> connection was passed to mail 2)
>>
>> 10:28:34.583491 IP 130.1.1.2.25 > 140.1.1.1.3948: S
>> 151923592:151923592(0) ack 3712043867 win 5840 <mss 1460,nop,nop,sackOK>
>> 10:28:54.608731 IP 130.1.1.2.25 > 140.1.1.1.3948: P 1:52(51) ack 1
>> win 5840
>>
>> mail 2 tunl0
>>
>> 10:28:34.583459 IP 140.1.1.1.3948 > 130.1.1.2.25: S
>> 3712043866:3712043866(0) win 65535 <mss 1260,nop,nop,sackOK>
>> 10:28:34.588206 IP 140.1.1.1.3948 > 130.1.1.2.25: . ack 151923593 win
>> 65535
>> 10:28:54.813191 IP 140.1.1.1.3948 > 130.1.1.2.25: . ack 52 win 65484
>>
>>
>> This tcpdump shows a full tcp connection via the working load
>> balancer to one of the mail servers, in this case mail 2. The SMTP
>> servers are configured to pause for 20 seconds before showing their
>> banner, which accounts for the delay between the packets.
>
> So this works perfectly, as shown above, which actually indicates that
> you have at one point got LVS to work. Sidenote: Your LVS seems to be
> a bit out of sync regarding time; otherwise your trace looks odd.
>
Yes, it was actually someone else who got it working before, and he is
far too busy to assist me with the new one :)
>> Now, if I try the same thing but telnet to 100.1.1.2:25 (the new load
>> balancer), the connection times out. tcpdumps show:
>
> Care to show the whole ipvsadm -L -n output? Or is the one above
> representative enough to display the problem?
>
Didn't I paste this above? --list is the same as -L I believe, at least
the output is no different..
>> load balancer
>> =========
>> 11:01:48.231327 IP 140.1.1.1.4042 > 100.1.1.2.25: S
>> 1821058469:1821058469(0) win 65535 <mss 1260,nop,nop,sackOK>
>> 11:01:51.195252 IP 140.1.1.1.4042 > 100.1.1.2.25: S
>> 1821058469:1821058469(0) win 65535 <mss 1260,nop,nop,sackOK>
>> 11:01:57.230423 IP 140.1.1.1.4042 > 100.1.1.2.25: S
>> 1821058469:1821058469(0) win 65535 <mss 1260,nop,nop,sackOK>
>
> Indicates a routing or network configuration issue.
>
>> Both of the mail servers show no traffic whatsoever on eth0 or the
>> tunnel interface.
>
> Looks like the scheduler is not invoked or the packet does not match
> the configuration.
>
>> On the load balancer, /proc/sys/net/ipv4/vs/debug_level is set to 9
>> and the follow messages were observed in syslog:
>
> Excellent:
>
>> Mar 29 11:01:48 dev1 kernel: IPVS: lookup/in TCP
>> 140.1.1.1:4042->100.1.1.2:25 not hit
>> Mar 29 11:01:48 dev1 kernel: IPVS: lookup service: fwm 0 TCP
>> 100.1.1.2:25 hit
>
> Now this is very very weird. The normal TCP service lookup did not
> succeed, although it should have, but the FWM TCP service lookup did.
> Are you sure that:
>
> a) You have cleanly shutdown (rmmod ip_vs if necessary) IPVS between
> the functional and the non-functional test conduct?
ipvs is compiled statically into the kernel, so how would I shut it
down? I had no idea it was necessary to shut it down and bring it back
up, although I have rebooted the server a couple of times which I am
sure would accomplish the same effect.
> b) You have no iptables or iproute2 rules indicating firewall marks?
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iproute2
bash: iproute2: command not found
I built this server myself and never did anything with iproute2.. so
I'm guessing the answer is no. Although I do believe Debian is evil and
so I guess it could have possibly done this itself behind my back.
> c) You have no port 0 service set up?
Definitely not
>
>> Mar 29 11:01:48 dev1 kernel: IPVS: ip_vs_wlc_schedule(): Scheduling...
>> Mar 29 11:01:48 dev1 kernel: IPVS: WLC: server 120.1.1.1:25
>> activeconns 0 refcnt 1 weight 1 overhead 0
>> Mar 29 11:01:48 dev1 kernel: IPVS: Bind-dest TCP c:140.1.1.1:4042
>> v:100.1.1.2:25 d:120.1.1.1:25 fwd:T s:0 conn->flags:182
>> conn->refcnt:1 dest->refcnt:2
>> Mar 29 11:01:48 dev1 kernel: IPVS: Schedule fwd:T c:140.1.1.1:4042
>> v:100.1.1.2:25 d:120.1.1.1:25 conn->flags:1C2 conn->refcnt:2
>
> This looks like it would happily send it.
>
>> Mar 29 11:01:48 dev1 kernel: IPVS: TCP input [S...]
>> 120.1.1.1:25->140.1.1.1:4042 state: NONE->SYN_RECV conn->refcnt:2
>
> Ok, we do the state transition indicating that we've allocated the
> connection structure for the hash table entry.
>
>> Mar 29 11:01:51 dev1 kernel: IPVS: lookup/in TCP
>> 140.1.1.1:4042->100.1.1.2:25 hit
>
> Second SYN as seen in your non-functional tcpdump trace.
>
>> Mar 29 11:01:57 dev1 kernel: IPVS: lookup/in TCP
>> 140.1.1.1:4042->100.1.1.2:25 hit
>
> Third SYN as seen in your non-functional tcpdump trace.
>
>> Mar 29 11:02:04 dev1 kernel: IPVS: Unbind-dest TCP c:140.1.1.1:4039
>> v:100.1.1.2:25 d:120.1.1.2:25 fwd:T s:3 conn->flags:182
>> conn->refcnt:1 dest->refcnt:2
>
> This is not belonging to the trace above since it's port 4039 which
> must have been a test performed before you took the trace. Most likely
> this one ran into the normal 60 sec timeout.
>
>> I really am at a loss as to why this doesn't work, the debug log
>> seems to show IPVS passing traffic to mail 1 (120.1.1.1) however the
>> tcpdump for that server shows absolutely nothing. If anyone can
>> point me in the right direction here I would be very grateful.
>
> Can you show your routing information on your LVS? As well as the tun*
> device configuration in the proc-fs?
>
Sure, by LVS i'm going to assume you mean the broken load balancer.
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
100.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 100.1.1.254 0.0.0.0 UG 0 0 0 eth0
# find /proc |grep tun
#
This is odd, tunl0 does exist:
# ifconfig tunl0
tunl0 Link encap:IPIP Tunnel HWaddr
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Don't know why its absent from /proc.
Thanks again for your assistance,
Mark
--
Mark Wadham
e: mark.wadham at areti.net t: +44 (0)20 8315 5800 f: +44 (0)20 8315 5801
Areti Internet Ltd., http://www.areti.net/
===================================================================
Areti Internet Ltd: BS EN ISO 9001:2000
Providing corporate Internet solutions for more than 10 years.
===================================================================
Search lvs-users Archives
More information about the lvs-users
mailing list