kernel 2.6.17.6 IPVS kernel: IPVS: ip_vs_wrr_schedule(): no available servers

[Bryan Heitman]

I have used LVS on the 2.4 kernel tree for quite some time with keepalived, 
the product has worked great.  Thank you to the LVS team!

I recently switched to new 64bit xeons with kernel 2.6.17.6 and I am now 
having some strange issues with LVS.

Currently I am getting flooded in /var/log/messages with kernel: IPVS: 
ip_vs_wrr_schedule(): no available servers.

However, running ipvsadm reports I have all of my real servers loaded into 
IPVS.

I have 48 entries in the ipvsadm table, I am pasting one of them here:

TCP  imap.mydomain.net:imap wrr persistent 300
  -> localhost.localdomain:imap   Local   0      0          0
  -> 10.0.5.56:imap               Route   0      0          0

IPVS is also not functioning as you can imagine since it seems to think it 
has no real servers.

I have seem some minor changes between 2.6.17.6 and 2.6.19.2 but nothing I 
can see that relates to any bugfixes, so by upgrading I do not think it 
would solve this problem.  Have I overlooked something?

I have tried shutting off the real servers and let keepalived fail out of 
every service which results in the real servers being removed from ipvsadm. 
...I have then re-activated the real servers which adds the real servers 
back in correctly to ipvsadm.  This did not fix the problem but keepalived 
appeared to be operating correctly. I have fixed the problem by killing 
keepalived and starting it back up.  Perhaps this unloads and loads the ipvs 
kernel module?

I am also still working on the steps to reproducing this problem.

Has anyone seen anything like this?

Best

Bryan Heitman