[lvs-users] extremely slow (10kb/s) LVS NAT on new RHEL 6 load balancer

Alois Treindl alois at astro.ch
Sat Nov 26 21:34:12 GMT 2011



On 26.11.11 21:50, Joseph Mack NA3T wrote:
> On Sat, 26 Nov 2011, Alois Treindl wrote:
>
>> The httpd real server introduces 200 millisecond delays
>> between packets when sending.
>                          ^^^^^^^
> I assume you mean "replying"

yes. replying to the client's http request, trasnmitting data to the client

>
> Whenever I get this it's something like
>
> o the routing is wrong (are you getting a shower of icmp
> packets?)

how would I see them with tcpdump?


> o the httpd is phoning home (asking for a service on the
> internet eg DNS) that has to time out

turned this off, I think. httpd is logging without using DNS, nothing is 
running on real servers except the rare requests from my LVS test access.

As I said, I can remove the LVS router and can connect a cable directly 
and then I get high performance.

I try to look into TCP_NODELAY because the 200 msec point in that direction.

Your help is really appreciated.

I have tried to compile ipvsadm 1.26 but it has too many problems 
compiling on my RHEL6. I have to stick with the CentOs 6 ipvsadm 1.25.9

I begin to despair because I have a fixed setup date for the new cluster 
to keep early next week, and I cannot test it when the LVS balancer is 
not working.

---------
Here is a fraction of tcpdump, as seen on the client, started off
on the client by wget  http://LVS/large_file_name
as80.stro.ch is the client, lv2 the LVS
19:51:25.610063 IP as80.astro.ch.39594 > lv2.astro.com.http: Flags [S], 
seq 3584270897, win 5840, options [mss 1460,sackOK,TS val 432865933 ecr 
0,nop,wscale 7], length 0
19:52:45.207361 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [S], 
seq 545625682, win 5840, options [mss 1460,sackOK,TS val 432945531 ecr 
0,nop,wscale 7], length 0
19:52:45.207746 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [S.], 
seq 951261745, ack 545625683, win 5792, options [mss 1460,sackOK,TS val 
12093210 ecr 432945531,nop,wscale 7], length 0
19:52:45.207763 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.], 
ack 1, win 46, options [nop,nop,TS val 432945531 ecr 12093210], length 0
19:52:45.207882 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [P.], 
seq 1:122, ack 1, win 46, options [nop,nop,TS val 432945532 ecr 
12093210], length 121
19:52:45.208202 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
ack 122, win 46, options [nop,nop,TS val 12093211 ecr 432945532], length 0
19:52:45.209374 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 
432945532], length 1448
19:52:45.209384 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 
432945532], length 1448
19:52:45.209389 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 
432945532], length 1448
   DELAY
19:52:45.410088 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 
432945532], length 1448
19:52:45.410103 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.], 
ack 1449, win 69, options [nop,nop,TS val 432945734 ecr 12093413], length 0
19:52:45.410482 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 
432945734], length 1448
19:52:45.410492 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 
432945734], length 1448
   DELAY
19:52:45.611086 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], 
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093614 ecr 
432945734], length
1448

----------------
here is the traffic as seen on the LVS, internal interface to the client w1
(it is not the same file transfer, this is why the timestamps are different)
18:53:44.819241 IP as80.astro.ch.36962 > w1.http: Flags [S], seq 
862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr 
0,nop,wscale 7], length 0
18:53:44.819412 IP w1.http > as80.astro.ch.36962: Flags [S.], seq 
1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val 
8552820 ecr 429405138,nop,wscale 7], length 0
18:53:44.819505 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1, win 
46, options [nop,nop,TS val 429405138 ecr 8552820], length 0
18:53:44.819588 IP as80.astro.ch.36962 > w1.http: Flags [P.], seq 1:122, 
ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 121
18:53:44.819746 IP w1.http > as80.astro.ch.36962: Flags [.], ack 122, 
win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0
18:53:44.820270 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:4345, 
ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 4344
   DELAY
18:53:45.021411 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:1449, 
ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405138], length 1448
18:53:45.021577 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1449, 
win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0
18:53:45.021757 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 
429405341], length 2896
   DELAY
18:53:45.222410 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 
429405341], length 1448
18:53:45.222593 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 2897, 
win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0
18:53:45.222752 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 
429405542], length 2896
   DELAY
18:53:45.423386 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 
429405542], length 1448
18:53:45.423567 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 4345, 
win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0
18:53:45.423728 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 
429405742], length 2896
   DELAY
18:53:45.624412 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 
429405742], length 1448
18:53:45.624572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 5793, 
win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0
18:53:45.624753 IP w1.http > as80.astro.ch.36962: Flags [P.], seq 
13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 
429405943], length 2896
   DELAY
18:53:45.825385 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 
429405943], length 1448
18:53:45.825547 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 7241, 
win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0
18:53:45.825728 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 
429406144], length 2896
   DELAY
18:53:46.026413 IP w1.http > as80.astro.ch.36962: Flags [.], seq 
7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr 
429406144], length 1448
18:53:46.026572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 8689, 
win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0

------------
same transfer, observed on LVS external interface, time stamps agree 
with sample 2
18:53:44.819225 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [S], 
seq 862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr 
0,nop,wscale 7], length 0
18:53:44.819424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [S.], 
seq 1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val 
8552820 ecr 429405138,nop,wscale 7], length 0
18:53:44.819499 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 0
18:53:44.819584 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [P.], 
seq 1:122, ack 1, win 46, options [nop,nop,TS val 429405138 ecr 
8552820], length 121
18:53:44.819758 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0
18:53:44.820284 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 1:4345, ack 122, win 46, options [nop,nop,TS val 8552820 ecr 
429405138], length 4344
   DELAY
18:53:45.021424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 
429405138], length 1448
18:53:45.021567 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 1449, win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0
18:53:45.021771 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 
429405341], length 2896
   DELAY
18:53:45.222424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 
429405341], length 1448
18:53:45.222583 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 2897, win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0
18:53:45.222766 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 
429405542], length 2896
   DELAY
18:53:45.423399 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 
429405542], length 1448
18:53:45.423557 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 4345, win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0
18:53:45.423742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 
429405742], length 2896
   DELAY
18:53:45.624425 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 
429405742], length 1448
18:53:45.624562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 5793, win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0
18:53:45.624915 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [P.], 
seq 13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 
429405943], length 2896
   DELAY
18:53:45.825398 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 
429405943], length 1448
18:53:45.825537 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 7241, win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0
18:53:45.825742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 
429406144], length 2896
18:53:46.026426 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], 
seq 7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr 
429406144], length 1448
18:53:46.026562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], 
ack 8689, win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0

----------
I also tried to look at the tcp socks involved in the transfer, with ss
But I do not really understand what I see.

seen on client:
ss -t -i -o -e dst lv2.astro.com
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      0            192.53.104.2:51478       192.53.104.100:http
uid:101 ino:22082073 sk:ffff88082b018100
ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:50 cwnd:4 send 46.3Mbps 
rcv_rtt:174.75 rcv_space:60816

seen on w1 real server:
ss -t -i -o -e dst 192.53.104.2
ESTAB      0      36200        10.1.1.1:http            192.53.104.2:51506
timer:(on,132ms,0) uid:104 ino:37970 sk:ffff88081164f6c0
ts sack cubic wscale:7,7 rto:201 rtt:1.875/0.75 ato:40 cwnd:10 
ssthresh:2 send 61.8Mbps rcv_space:5792

--------------
when I do a direct http transfer not through the LVS, (which is fast, 
everything is conencted with 1Gb links), then I see these tcp socks

on client:
ss -t -i -o -e dst 192.53.104.2
State       Recv-Q Send-Q      Local Address:Port  Peer Address:Port
ESTAB      0      0            192.53.104.2:36300 
192.53.104.99:http     uid:101 ino:22146399 sk:ffff880451ef4180
ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:40 cwnd:4 send 46.3Mbps 
rcv_rtt:1.875 rcv_space:6199088

on server:
ESTAB       0      2920616  ::ffff:192.53.104.99:http 
::ffff:192.53.104.2:36305    timer:(on,206ms,0) uid:104 ino:47402 
sk:ffff8801261071c0
ts sack cubic wscale:7,7 rto:206 rtt:6.5/0.75 ato:40 cwnd:507 
ssthresh:504 send 903.6Mbps rcv_space:5792

They seem to have the same values.




More information about the lvs-users mailing list