DR Load balancing active/inactive connections

[RU Admin]

I've been using IPVS for almost two years now, I started out with 6 
machines (1 director, 5 real servers) and was using LVS-NAT.  During the 
first year that I was running that email server everything worked 
perfectly with LVS-NAT.  About a year ago, I decided to setup another 
email server, this time with 5 machines (1 director, 4 real servers) and 
decided it was time to get LVS-DR working, which I successfully did.  I 
then decided to switch over my first email server (the one with 6 
machines) to LVS-DR, since the other LVS-DR server was working great. 
Both of my email servers have been working great with LVS-DR for the past 
year, with one major exception (which has just recently started getting 
worse, because of the large volumes of connections coming into the 
servers).  The problem I am having is that my active/inactive connections 
are not being listed properly.  What I mean, is that the counter for my 
active/inactive connections just keep going up and up, and are constantly 
being skewed.  I read through a good number of archived messages on this 
mailing list, and I keep seeing everyone saying "Those numbers ipvsadm 
are showing, are just for reference, they don't really mean anything, 
don't worry about them."   Well, I can tell you first hand, when you use 
wlc (weighted least connections), those number obviously DO mean 
something.  My machines are no longer being equally balanced between 
because my connection counts are off, and this is really effecting the 
performance of my email servers.  When running "ipvsadm -lcn", I can see 
connections with the CLOSE state going from 00:59 to 00:01, and then 
magically going back to 00:59 again for no reason.  The same holds true 
for ESTABLISHED connections, I see them go from 29:59 to 00:01 and then 
back to 29:59, and I know for a fact that the connection from the client 
has ended.

I'm currently using "IP Virtual Server version 1.2.0", and I know that 
there is a 1.2.1 version available, but my problem is that my email 
servers are in a production environment, and I really don't want to 
recompile a new kernel with the latest IPVS if that isn't going to solve 
the problem.  I'd hate to cause other problems with my system because of a 
major kernel upgrade.

I can only hope that someone has some suggestions, I am a firm supporter 
of IPVS, and as I said I've been using it for 2 years now and one of my 
email servers handles over 30,000,000 emails in one month (or almost 1 
million emails a day).  So we heavily relying on IPVS.  There is another 
department in our organization that spent thousands of dollars on 
FoundryNet load balancing productions, and I've been able to accomplish 
the same tasks (and handle a higher load) by using IPVS, so clearly IPVS 
is a solid product.  Unfortunately, I just really need to figure out what 
is going on with the connection count problems.

I not sure what information you guys need, but here's some info about my 
setup.  If you need any more details, feel free to ask.

6 Dell PowerEdge SC1425
Dual Xeon 3.06Ghz processors
2GB DDR
160GB SATA
Running Debian Sarge

1 machine is the director, the other 5 are the real servers.  All 6 
machines are on the same subnet (with public IPs), and the director is 
using LVS-DR for load balancing.  Just to give you an idea as to the types 
of connection numbers I'm getting:
   Prot LocalAddress:Port Scheduler Flags
     -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
   TCP  vip.address.here:smtp wlc
     -> realserver1.ip.here:smtp     Route   50     648        2357
     -> realserver2.ip.here:smtp     Route   50     650        2231
     -> realserver3.ip.here:smtp     Route   50     648        2209
Whereas when using LVS-NAT (which was 100% perfect), my numbers would be 
something like:
     -> realserver1.ip.here:smtp     Route   50     16        56
     -> realserver2.ip.here:smtp     Route   50     14        50
     -> realserver3.ip.here:smtp     Route   50     15        48

I use keepalived to manage the director and to monitor the real servers. 
The only "tweaking" that I've done to IPVS, is I have to run this:
   /sbin/ipvsadm --set 1800 0 0
before starting up keepalived, just so that the active connections will 
stay active for 30 minutes.  In other words, we allow our users to idle 
their connection for 30 minutes, and after that, then the connection 
should be terminated.  And I put "0 0" there, because from what I've 
read, that tells ipvsadm to not change those other two values (in other 
words, leave the defaults as is).

That's about all I can think of, the only other wierd thing that I had 
to do was to tweak some networking settings on the real servers to fix 
the pain-in-the-@$$ ARP issues that come with DR.  But I doubt those 
changes would have anything to do with the director's load balancing 
problems. Those tweaks were only done on the real servers, and they were 
to just silence the broadcasting of the MAC address for the VIP (dummy0) 
interfaces on the real servers.

And for those interested, I switched from LVS-NAT to LVS-DR, because I 
really feel that you can get much better network throughput by using DR 
instead of NAT.  I know I've read a bunch of messages on the mailing list 
saying that NAT is just as good, but I think the one major advantage of 
IPVS is that it supports DR, whereas almost every other load balancing 
product I've seen uses some type of NATing (in other words, all network 
traffic goes in and out of the director).  To have a setup, like I do now 
where only incoming traffic has to go through the director, is absolutely 
fantasic, because the cluster (for lack of a better word) can be easily 
expanded.  With LVS-NAT, when you add more real servers, all you get is 
more CPU power, you don't get any more network throughput, with LVS-DR 
when you add a new real server, you completely expand your cluster, not 
just one part of it.

Sorry for the long email.  But I really would appreciate any help that can 
be provided.

Thanks!

Craig