Performance degradation due to slab object size increase

[Roberto Nibali]

Hello guys,

I've been helping with debugging a high traffic web site that deploys 
LVS_DR and is hitting IPVS limits :(. We've got to re-think our 
statements about IPVS being as fast or close to routing speed. Also CPU 
L1 cache is getting in our way, with Intel pushing those lousy 
dual/quad-core CPUs. They run with 16KB shared i/d-cache cross CPU (avg 
cache miss close to 1%!) and this is just not enough anymore to handle 
the slab cache lookup for sites with a high number of packets per 
second. I haven't been able to measure the DTLB invalidation rate but I 
suspect it high enough. And it seems that the bucket size of the hash 
table is also not so insignificant than anticipated and calculated 
before (around 2001/02 before switching to Jenkins hash).

I'll share the details hopefully before I leave for my sabbatical.

Just one question: When did IPVS fatten to a 256 byte slab object size 
for struct ip_vs_conn? This is fatal! (Joe will need to rewrite about 
1432 pages of his Howto :)). Seriously: We need to put ip_vs_conn on a 
diet. What about the idea of the comment at the end of the ip_vs_conn 
structure definition in ../include/net/ip_vs.h:

        /* Note: we can group the following members into a structure,
            in order to save more space, and the following members are
            only used in VS/NAT anyway */
         struct ip_vs_app        *app;           /* bound ip_vs_app 
object */
         void                    *app_data;      /* Application private 
data */
         struct ip_vs_seq        in_seq;         /* incoming seq. struct */
         struct ip_vs_seq        out_seq;        /* outgoing seq. struct */
};

How big is struct ip_vs_conn actually? It's only a 256 byte object since 
we must be slightely bigger than 128 bytes and do a HW_CACHE_ALIGN on 
kmem_cache_create().

Please comment if you find time. Thanks and best regards,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc