LVS performance bug

[Roberto Nibali]

Hi Mike,

Expect long delays in my answers ...

> Ran it again and here's what I see. We currently have 8 Gigs of memory
> installed. It doesn't appear from the "free" command that we ran

I'm interested in your hardware configuration (e.g. EMT64 or true 
64bit), so could you please send along following output:

dmesg -s 100000
dmidecode
lspci -v
cat /proc/meminfo

> completely out of memory. Heres what "free" and "slabinfo" say before
> the test. This is sitting idle. 
> [root at jackets-a upgrade]# free -m
>              total       used       free     shared    buffers
> cached
> Mem:          8118       1508       6609          0        274
> 974
> -/+ buffers/cache:        259       7858
> Swap:         2000          0       2000
> [root at jackets-a upgrade]# 

Looks like the standard 4GB split, could you please also send your 
.config for the running kernel (probably /proc/config.gz)? Would it be 
possible for you to upload those files somewhere, so I can download and 
properly analyse them?

> ip_vs_conn         39091  39105    256   15    1 : tunables  120   60
> 8 : slabdata   2607   2607      0
> ip_fib_alias          18    113     32  113    1 : tunables  120   60
> 8 : slabdata      1      1      0
> ip_fib_hash           18    113     32  113    1 : tunables  120   60
> 8 : slabdata      1      1      0

Hmm, thought 2.6.18 already had fib_trie ...

> arp_cache             11     15    256   15    1 : tunables  120   60
> 8 : slabdata      1      1      0
> RAW                    5      6    640    6    1 : tunables   54   27
> 8 : slabdata      1      1      0
> UDP                   35     36    640    6    1 : tunables   54   27
> 8 : slabdata      6      6      0
> tw_sock_TCP           81     90    128   30    1 : tunables  120   60
> 8 : slabdata      3      3      0
> request_sock_TCP       8     59     64   59    1 : tunables  120   60
> 8 : slabdata      1      1      0
> TCP                   55     60   1280    3    1 : tunables   24   12
> 8 : slabdata     20     20      0

Ok.

> Then we ran the test again and cranked up the traffic. It took a few
> minutes and then it happened again. 
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.
> IPVS: ip_vs_conn_new: no memory available.

You obviously ran out of kernel memory.

> Heres what "free" and "slabinfo" say after the test. 
> [root at jackets-a upgrade]# free
>              total       used       free     shared    buffers
> cached
> Mem:       8313112    1371072    6942040          0      62200
> 334064
> -/+ buffers/cache:     974808    7338304
> Swap:      2048184          0    2048184

You've warmed up the cache, now let's cook dinner :).

> ip_vs_conn        2774925 2774925    256   15    1 : tunables  120   60
> 8 : slabdata 184995 184995      0

If I'm not mistaken this is the 3G/1G split. Looks like you're using 
EMT64 with PAE (only 4GB DMA addressing, however this is none of your 
concern).

> So the only thing I see shooting up higher in memory used is
> buffers/cache used seems to grow. But in the slabinfo the ip_vs_conn
> active objects grows fast. I watched it grow during the test from 39K
> objects to over 2 million objects. Maybe something isn't being reset or
> returned to the pool. We are running the OPS patch(one packet
> scheduling) because we are using LVS for the udp service DNS. I'm sure
> it treats connections differently than the regularly hashed connections
> thing. 
> 
> If you need anything else let me know. I have a reproducer now that
> makes it happen regularly. 

And that would be? testlvs?

Best regards,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc