[lvs-users] Could LVS/DR cause a bottleneck with media servers?

[Malcolm Turnbull]

On 16 February 2011 02:56, Roger Littin <roger at rlmedia.co.nz> wrote:
> Hi,
>
> I have a set up using ldirectord in direct routing to 30 media servers.
>
> The director is a dell R210 4 core with 8GB memory and has 2 1Gbps network connections, 1 public & 1 private.
>
> Each real server is a Dell R610 16 core with 12GB memory and has the same network connections as above.
>
> All the servers are running centos 5.
>
> According to the data center, these servers are connected to pairs of 40Gb switches and there is ample capacity.
>
> What happens is when the connections per real server get to around 1000 – 1200 concurrent connections, the bandwidth outgoing per server wont go above about 250Mbps which relates to about 7.5Gbps across all servers.  At that time is when the complaints start coming in about stream problems.
>
> I guess the question is could the director somehow be limiting the throughput on the real servers or is the dc not telling the truth about?
> The bandwidth going through the nics on the director is around 50 – 75Mbps in on the public nic and out on the private nic to the real servers.
>
> Before we started using lvs, we had 10 servers running with round robin dns and these would easily handle 900Mbps each at the same time.
>
> lidrectord config file.
>
> # Global Directives
> checktimeout=10
> checkinterval=5
> #fallback=127.0.0.1:80
> autoreload=yes
> callback="/etc/ha.d/syncsettings.sh"
> logfile="/var/log/ldirectord.log"
> #logfile="local0"
> #emailalert="admin at x.y.z"
> #emailalertfreq=3600
> #emailalertstatus=all
> quiescent=no
>
> virtual=147
>        real=172.31.214.12 gate 100
>        real=172.31.214.13 gate 100
>        real=172.31.214.14 gate 100
>        real=172.31.214.15 gate 100
>        real=172.31.214.16 gate 100
>        real=172.31.214.17 gate 100
>        real=172.31.214.18 gate 100
>        real=172.31.214.19 gate 100
>        real=172.31.214.21 gate 100
>        real=172.31.214.22 gate 100
>        real=172.31.214.23 gate 100
>        real=172.31.214.24 gate 100
>        real=172.31.214.25 gate 100
>        real=172.31.214.26 gate 100
>        real=172.31.214.28 gate 100
>        real=172.31.214.29 gate 100
>        real=172.31.214.30 gate 100
>        real=172.31.214.31 gate 100
>        real=172.31.214.32 gate 100
>        real=172.31.214.33 gate 100
>        real=172.31.214.34 gate 100
>        real=172.31.214.35 gate 100
>        real=172.31.214.36 gate 100
>        real=172.31.214.37 gate 100
>        real=172.31.214.38 gate 100
>        real=172.31.214.39 gate 100
>        real=172.31.214.40 gate 100
>        real=172.31.214.41 gate 100
>        real=172.31.214.42 gate 100
>        scheduler=wlc
>        protocol=fwm
>        persistent=60
>        netmask=255.255.255.255
>        service=http
>        checkport=1935
>        request="/"
>        receive="Wowza Media Server 2"
>
> iptables
>
> *mangle
> :PREROUTING ACCEPT [438:421747]
> :INPUT ACCEPT [438:421747]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [95:14749]
> :POSTROUTING ACCEPT [122:21354]
> -A PREROUTING –d *.*.*.147 -p tcp -m tcp --dport 80 -j MARK --set-mark 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 443 -j MARK --set-mark 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 554 -j MARK --set-mark 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 1935 -j MARK --set-mark 0x93
> -A PREROUTING -d *.*.*.147 -p udp -m udp --dport 6970:9999 -j MARK --set-mark 0x93
> COMMIT
>
> the main port that is used is 1935.
>
> Thanks in advance.
>
> Regards,
>
> Roger.
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users


Roger,

My gut would say that your switch is getting saturated (not the director).
I wonder if you could get some of the servers to reply to a different
switch (local subnet) but still through the director i.e.
prove the director can can more load if the outgoing traffic is
through a different switch.
Also CPU load on the director is a pretty good indicator of stress.
We've had customers doing similar kind of load (but every setup is different).


-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/