[lvs-users] Cluster using only one of two available nodes

Ross Michael (LEEDS PCT) michael.ross at nhs.net
Wed Jul 27 12:47:17 BST 2011


Problem sorted.  It was an ARP issue on the apache servers.

For anyone else who has a similar problem, check your /etc/sysctl.conf on your apache servers and check you have something like the following (replacting eth0 with your own nic name).

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.eth0.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.eth0.arp_announce=2

Cheers

Mike.


________________________________________
From: Ross Michael (LEEDS PCT) [michael.ross at nhs.net]
Sent: 26 July 2011 09:13
To: lvs-users at linuxvirtualserver.org
Subject: [lvs-users] Cluster using only one of two available nodes

Hi all,

I have something of a bit of an odd issue that I have so far been unable to find a solution for, so I am hoping that you can help.

I have a 4 server rig:
2 x directors (heartbeat HA)        192.168.8.111, 192.168.8.112
2 x apache servers                           192.168.8.121, 192.168.8.122
Virtual IP                                              192.168.8.120

The HA directors are set up and appear to be working perfectly (and failing over correctly).  The Apache servers are also working as expected and will both happily return content when connected to directly by IP.  However, when connecting through the virtual IP, the currently active director will only send requests to one of the servers and seems to completely ignore the other one.

What makes this rather odd is that the director recognises both apache servers, checks them regularly for response and reports correctly if either one goes down.

ipvsadm -Ln
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.8.120:80 rr persistent 600
  -> 192.168.8.121:80             Route   1      0          0
  -> 192.168.8.122:80             Route   1      0          0

You can see both servers.  Both are responding to the director alive requests:
Both apache logs report: 192.168.8.111 - - [26/Jul/2011:09:02:30 +0100] "GET /alive.html HTTP/1.1" 200 12 "-" "libwww-perl/5.805"

The director is set up like this:-
ldirectord.cf:
checktimeout=10
checkinterval=2
autoreload=yes
quiescent=no

virtual=192.168.8.120:80
        real=192.168.8.121:80 gate
        real=192.168.8.122:80 gate
        fallback=127.0.0.1:80 gate
        service=http
        request="alive.html"
        receive="Still alive"
        scheduler=rr
        protocol=tcp
        checktype=negotiate
        persistent=600

If I take out 8.121 (the apache server it insists on using), it will not direct traffic to 8.122.  Something else I have noticed is that if no apache server is available, it will not use the fallback either.

Something else I have noticed, but i am not sure it is relevant is that if I do a ipvsadm -Ln -stats, I get:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.8.120:80                  358     7618        0   527015        0
  -> 192.168.8.121:80                    0        0        0        0        0
  -> 192.168.8.122:80                    0        0        0        0        0

Notice that all the connections are on the virtual IP and the real IP's are saying zero across the board even though the system appears (on the surface) to be working properly.  I thought that for every connection via the VIP, it would increment 'Conns, InPkts etc' for the real server, though I could be wrong.

Any thoughts would be most appreciated.

Cheers

Mike


********************************************************************************************************************

This message may contain confidential information. If you are not the intended recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents:
to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland
NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients
NHSmail provides an email address for your career in the NHS and can be accessed anywhere
For more information and to find out how you can switch, visit www.connectingforhealth.nhs.uk/nhsmail

********************************************************************************************************************

********************************************************************************************************************

This message may contain confidential information. If you are not the intended recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents:
to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland
NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients
NHSmail provides an email address for your career in the NHS and can be accessed anywhere
For more information and to find out how you can switch, visit www.connectingforhealth.nhs.uk/nhsmail

********************************************************************************************************************




More information about the lvs-users mailing list