[lvs-users] ldirectord does not transfer connections when a real server dies

Malcolm Turnbull malcolm at loadbalancer.org
Tue Apr 30 11:50:20 BST 2013


The quiescent=no will only give the desired behaviour on health check
failures (it will rip the server out).
Manually setting the weight to 0 will just drain the server...

So best bet is force it to fail the negotiate check
OR
manually pull it out....



On 30 April 2013 11:38, Konstantin Boyanov <kkboyanov at gmail.com> wrote:
> Hello,
>
> First off thanks for your reply! I tried setting quiescent to NO, but the
> result was not changed - when i set the weight of one of the real servers
> to 0 and then reload the pages on the client, the connections to the dead
> server stay on the dead server. The only thing affecting the behaviour of
> the connections which I found was decreasing the KeepAliveTimeout on the
> client. But this is no solution...
>
> Do you mean that I have to _manually_ remove the server and then add it
> back in with weight of zero?
>
> About the health check - I am using the negotiate method so we can easily
> pull of real servers for maitenance. Does the quiescent=no setting means
> that I would have to manually readd each real server after maintenance?
>
> Best Regards,
> Konstantin Boyanov
>
> On Tue, Apr 30, 2013 at 12:01 PM, Malcolm Turnbull <malcolm at loadbalancer.org
>> wrote:
>
>> Konstantin,
>>
>> Easier said than done but...
>> You would need to completely remove the server from the LVS table,
>> then you can put it back in with a weight of zero.
>> This is similar to the health check behaviour when you set:
>>
>> quiescent = no
>>
>>
>>
>>
>>
>> On 30 April 2013 10:30, Konstantin Boyanov <kkboyanov at gmail.com> wrote:
>> > Hello LVS users,
>> >
>> > I am using ldirectord to load balance two IIS servers. The
>> > ldirectord.cglooks like this:
>> >
>> >
>> >     autoreload = yes
>> >     quiescent = yes
>> >     checkinterval = 1
>> >     negotiatetimeout = 2
>> >     emailalertfreq = 60
>> >     emailalert = Konstantin.Boyanov at mysite.com
>> >     failurecount = 1
>> >
>> >     virtual = 172.22.9.100:80
>> >         checktimeout = 1
>> >         checktype = negotiate
>> >         protocol = tcp
>> >         real = 172.22.1.133:80 masq 2048
>> >         real = 172.22.1.134:80 masq 2048
>> >         request = "alive.htm"
>> >         receive = "I am not a zombie"
>> >         scheduler = wrr
>> >
>> > The load balancing is working fine, the real servers are visible etc.
>> > Nevertheless I am encountering a problem with a simple test:
>> >
>> > 1. I open some connections from a client browser (IE 8) to the sites that
>> > are hosted on the real servers
>> > 2. I cange the weight of the real server which server the above
>> connections
>> > to 0 and leave only the other real server alive
>> > 3. I reload the pages to regenerate the connections
>> >
>> > What I am seeing with ipvsadm -Ln is that the connections are still on
>> the
>> > "dead" server. I have to wait up to one minute (I suppose some TCP
>> timeout
>> > from the browser-side) for them to transfer to the "living" server. And
>> If
>> > in this one minute I continue pressing the reload button the connections
>> > stay at the "dead" server and their TCP timeout counter gets restarted.
>> >
>> > So my question is: Is there a way to tell the load balancer in NAT mode
>> to
>> > terminate / redirect existing connections to a dead server *immediately*
>> > (or close to immediately)?
>> >
>> > It seems to me a blunder that a reload on the client-side can make a
>> > connection become a "zombie", e.g. be bound to a dead real server
>> although
>> > persistance is not used and the other server is ready and available.
>> >
>> > The only thing that I found affecting this timeout is changing the
>> > keepAliveTimeout in the Windows machine running the IE8 which I use for
>> the
>> > tests. When I cahnged it from the dafault value of 60 seconds to 30
>> seconds
>> > the connections could be transferred after 30 seconds. It seems to me
>> very
>> > odd that a client setting can affect the operation of a network component
>> > as the load balancer.
>> >
>> > And another thing - what is the colum named "Inactive Conenctions" in the
>> > output from ipvsadm used for? Which connections are considered inactive?
>> >
>> > And also in the output of ipvsadm i see a couple of connections with the
>> > state TIME_WAIT. What are these for?
>> >
>> > Any insight and suggestions are highly appreciated !
>> >
>> > Cheers,
>> > Konstantin
>> >
>> >
>> >
>> > P.S: Here is some more information about the configuration:
>> >
>> >     # uname -a
>> >     Linux 3.0.58-0.6.2-default #1 SMP Fri Jan 25 08:31:01 UTC 2013 x86_64
>> > x86_64 x86_64 GNU/Linux
>> >
>> >     # ipvsadm -L
>> >     IP Virtual Server version 1.2.1 (size=4096)
>> >     Prot LocalAddress:Port Scheduler Flags
>> >       -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
>> >     TCP  lb-mysite.com wrr
>> >       -> spwfe001.mysite.com:h Masq    10     0          0
>> >       -> spwfe002.mysite.com:h Masq    10     0          0
>> >
>> >     # iptables -t nat -L
>> >     Chain PREROUTING (policy ACCEPT)
>> >     target     prot opt source               destination
>> >
>> >     Chain INPUT (policy ACCEPT)
>> >     target     prot opt source               destination
>> >
>> >     Chain OUTPUT (policy ACCEPT)
>> >     target     prot opt source               destination
>> >
>> >     Chain POSTROUTING (policy ACCEPT)
>> >     target     prot opt source               destination
>> >     SNAT       all  --  anywhere             anywhere
>> > to:172.22.9.100
>> >     SNAT       all  --  anywhere             anywhere
>> > to:172.22.1.130
>> >
>> >
>> >     # ip a
>> >     1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>> >         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> >         inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
>> >         inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
>> >     2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> > state UNKNOWN         qlen 1000
>> >         link/ether 00:50:56:a5:77:ae brd ff:ff:ff:ff:ff:ff
>> >         inet 192.168.8.216/22 brd 192.168.11.255 scope global eth0
>> >     3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> > state UNKNOWN         qlen 1000
>> >     link/ether 00:50:56:a5:77:af brd ff:ff:ff:ff:ff:ff
>> >     inet 172.22.9.100/22 brd 172.22.11.255 scope global eth1:1
>> >     inet 172.22.8.213/22 brd 172.22.11.255 scope global secondary eth1
>> >     4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> > state UNKNOWN qlen 1000
>> >         link/ether 00:50:56:a5:77:b0 brd ff:ff:ff:ff:ff:ff
>> >         inet 172.22.1.130/24 brd 172.22.1.255 scope global eth2
>> >
>> >
>> >     # cat /proc/sys/net/ipv4/ip_forward
>> >     1
>> >     # cat /proc/sys/net/ipv4/vs/conntrack
>> >     1
>> >     # cat /proc/sys/net/ipv4/vs/expire_nodest_conn
>> >     1
>> >     # cat /proc/sys/net/ipv4/vs/expire_quiescent_template
>> >     1
>> > _______________________________________________
>> > Please read the documentation before posting - it's available at:
>> > http://www.linuxvirtualserver.org/
>> >
>> > LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
>> > Send requests to lvs-users-request at LinuxVirtualServer.org
>> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)870 443 8779
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
>> Send requests to lvs-users-request at LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/



More information about the lvs-users mailing list