New system, higher active connections?

[H. Wade Minter]

On Oct 15, 2006, at 9:14 AM, Roberto Nibali wrote:

>> We just moved our primary load balancer from an ancient Dell  
>> PowerApp 120 running Red Hat 7.3 to a newer Dell PowerEdge 750  
>> running Fedora Core 5.  However, we're noticing something weird.
>
> Was 7.3 with 2.4.x kernel?

Yes - 2.4.18-18.7.x

>
>> Where before, we were seeing Active Connections in the 1-4 range  
>> even during normal usage, we're now seeing them in the 12-16 range  
>> on average.  We've got the same weighting on the new server as we  
>> did on the old.
>
> Different Server system and most importantly, different software  
> configuration. IPVS between 2.4 and 2.6 (provided my assumption  
> above holds) has change significantly with regards to the ratio of  
> active/inactive connections. We've seen that in our rrdtool/MRTG  
> graphs as well.

It's not so much that we've seen the numbers change, we've seen an  
actual load impact on our application.  Last night, for example, the  
load-balanced Apache instances behind LVS were slowing to a crawl and  
hitting MaxClients even under a fairly light load.  So I think it's  
more than just us having to readjust our expectations as to the  
number of active connections.

>
>> Does anyone have any ideas why we might be seeing such a jump on  
>> this newer system?
>
> Different kernel, where at least for the (w)LC scheduler the RS  
> calculation is done differently. On top of that, the TCP stack has  
> changed tunables and you hardware also behaves differently. The LVS  
> state transition timeouts are different between 2.4.x and 2.6.x  
> kernels, IIRC and so, for example if you're using LVS-DR, the  
> active connection to passive connection transition takes more time,  
> thus yielding a potentially higher amount of sessions in state  
> active connection.
>
> We'd need more information if you want to dig this phenomenon.

Sure.  This is a very vanilla FC5 system running the  
2.6.17-1.2174_FC5 kernel.  I'm certainly open to suggestions for  
tuning that we should do in order to get decent performance - I'd  
hate to have to drop back to the RH7.3 box.  We pretty much ported  
the LVS configuration over verbatim from the 7.3 to the FC5, so the  
only thing that's changed is the OS and hardware.

One thing we did notice is that by removing the persistent= line in  
our https balancer (we have two stanzas - one for port 80 and one for  
port 443), the active connection numbers for the port 80 stanza  
dropped dramatically.  Of course, that broke one part of our  
application, so we had to reenable it (though at 30 seconds as  
opposed to 300 seconds), but hopefully that's a clue.

I'm attaching our LVS configuration for reference.

Thanks,
Wade


######
# Global Directives
checktimeout=10
checkinterval=5
autoreload=no
logfile="/var/log/ldirectord.log"
quiescent=yes

virtual=128.109.135.22:80
         fallback=127.0.0.1:80
                 real=192.168.0.13:80 masq 2
                 real=192.168.0.15:80 masq 2
                 real=192.168.0.16:80 masq 2
                 real=192.168.0.17:80 masq 2
                 real=192.168.0.18:80 masq 2
                 real=192.168.0.19:80 masq 2
                 real=192.168.0.20:80 masq 2
                 real=192.168.0.21:80 masq 3
                 real=192.168.0.22:80 masq 3
                 real=192.168.0.23:80 masq 3
                 real=192.168.0.24:80 masq 3
                 real=192.168.0.25:80 masq 3
                 real=192.168.0.26:80 masq 3
                 real=192.168.0.27:80 masq 4
                 real=192.168.0.28:80 masq 4
                 real=192.168.0.29:80 masq 4
                 real=192.168.0.30:80 masq 4
                 real=192.168.0.31:80 masq 4
         service=http
         request="lvs/lvs_donotremove"
         receive="lvs up"
         scheduler=wrr
         protocol=tcp
         checktype=negotiate

virtual=128.109.135.22:443
         fallback=127.0.0.1:443
                 real=192.168.0.13:443 masq 2
                 real=192.168.0.15:443 masq 2
                 real=192.168.0.16:443 masq 2
                 real=192.168.0.17:443 masq 2
                 real=192.168.0.18:443 masq 2
                 real=192.168.0.19:443 masq 2
                 real=192.168.0.20:443 masq 2
                 real=192.168.0.21:443 masq 3
                 real=192.168.0.22:443 masq 3
                 real=192.168.0.23:443 masq 3
                 real=192.168.0.24:443 masq 3
                 real=192.168.0.25:443 masq 3
                 real=192.168.0.26:443 masq 3
                 real=192.168.0.27:443 masq 4
                 real=192.168.0.28:443 masq 4
                 real=192.168.0.29:443 masq 4
                 real=192.168.0.30:443 masq 4
                 real=192.168.0.31:443 masq 4
         service=https
         request="lvs/slvs_donotremove"
         receive="slvs up"
         scheduler=wrr
         persistent=30
         protocol=tcp
         checktype=negotiate