[lvs-users] Could LVS/DR cause a bottleneck with media servers?

[Roger Littin]

Hi,

I have a set up using ldirectord in direct routing to 30 media servers.

The director is a dell R210 4 core with 8GB memory and has 2 1Gbps network connections, 1 public & 1 private.

Each real server is a Dell R610 16 core with 12GB memory and has the same network connections as above.

All the servers are running centos 5.

According to the data center, these servers are connected to pairs of 40Gb switches and there is ample capacity.

What happens is when the connections per real server get to around 1000 – 1200 concurrent connections, the bandwidth outgoing per server wont go above about 250Mbps which relates to about 7.5Gbps across all servers.  At that time is when the complaints start coming in about stream problems.

I guess the question is could the director somehow be limiting the throughput on the real servers or is the dc not telling the truth about?
The bandwidth going through the nics on the director is around 50 – 75Mbps in on the public nic and out on the private nic to the real servers.

Before we started using lvs, we had 10 servers running with round robin dns and these would easily handle 900Mbps each at the same time.

lidrectord config file.

# Global Directives
checktimeout=10
checkinterval=5
#fallback=127.0.0.1:80
autoreload=yes
callback="/etc/ha.d/syncsettings.sh"
logfile="/var/log/ldirectord.log"
#logfile="local0"
#emailalert="admin at x.y.z"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no

virtual=147
        real=172.31.214.12 gate 100
        real=172.31.214.13 gate 100
        real=172.31.214.14 gate 100
        real=172.31.214.15 gate 100
        real=172.31.214.16 gate 100
        real=172.31.214.17 gate 100
        real=172.31.214.18 gate 100
        real=172.31.214.19 gate 100
        real=172.31.214.21 gate 100
        real=172.31.214.22 gate 100
        real=172.31.214.23 gate 100
        real=172.31.214.24 gate 100
        real=172.31.214.25 gate 100
        real=172.31.214.26 gate 100
        real=172.31.214.28 gate 100
        real=172.31.214.29 gate 100
        real=172.31.214.30 gate 100
        real=172.31.214.31 gate 100
        real=172.31.214.32 gate 100
        real=172.31.214.33 gate 100
        real=172.31.214.34 gate 100
        real=172.31.214.35 gate 100
        real=172.31.214.36 gate 100
        real=172.31.214.37 gate 100
        real=172.31.214.38 gate 100
        real=172.31.214.39 gate 100
        real=172.31.214.40 gate 100
        real=172.31.214.41 gate 100
        real=172.31.214.42 gate 100
        scheduler=wlc
        protocol=fwm
        persistent=60
        netmask=255.255.255.255
        service=http
        checkport=1935
        request="/"
        receive="Wowza Media Server 2"

iptables

*mangle
:PREROUTING ACCEPT [438:421747]
:INPUT ACCEPT [438:421747]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [95:14749]
:POSTROUTING ACCEPT [122:21354]
-A PREROUTING –d *.*.*.147 -p tcp -m tcp --dport 80 -j MARK --set-mark 0x93
-A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 443 -j MARK --set-mark 0x93
-A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 554 -j MARK --set-mark 0x93
-A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 1935 -j MARK --set-mark 0x93
-A PREROUTING -d *.*.*.147 -p udp -m udp --dport 6970:9999 -j MARK --set-mark 0x93
COMMIT

the main port that is used is 1935.

Thanks in advance.

Regards,

Roger.