[lvs-users] RX Packet drops on high traffic LVS in DR setup

Sashi Kant sashi.kant at eng.admob.com
Thu Oct 16 18:24:18 BST 2008

Thanks Graeme,

We have bonding setup in all the servers and it works fine.

I agree with your observation about packet per second problem , rather
than MB/s issue. I was able to push > 800 Mb/s of traffic to similar
setup (bonding-rr + Link aggregation )using a single machine as traffic

We start seeing dropped packet when traffic goes above 40k pps (tx+rx). 

Situation was improved when we start using both NIC in round robin mode,
however we did not see 2X improvement as expected. Could PCI-X bus limit
be playing a role ?

             description: PCI bridge
             product: 631xESB/632xESB/3100 Chipset PCI Express Root Port
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci at 00:1c.0
             version: 09
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport-driver
                description: PCI bridge
                product: EPB PCI-Express to PCI-X Bridge
                vendor: Broadcom
                physical id: 0
                bus info: pci at 02:00.0
                version: c3
                width: 32 bits
                clock: 33MHz
                capabilities: pci normal_decode bus_master cap_list
              *-network DISABLED
                   description: Ethernet interface
                   product: NetXtreme II BCM5708 Gigabit Ethernet
                   vendor: Broadcom Corporation
                   physical id: 0
                   bus info: pci at 03:00.0
                   logical name: eth3
                   version: 12
                   serial: 00:1d:09:6a:db:aa
                   width: 64 bits
                   clock: 66MHz
                   capabilities: bus_master cap_list ethernet physical
                   configuration: broadcast=yes driver=bnx2 latency=32
mingnt=64 multicast=yes
                   resources: iomemory:da000000-dbffffff irq:169

We are using bnx2 driver from dist kernel.



On Thu, 2008-10-16 at 09:39 +0100, Graeme Fowler wrote:
> On Wed, 2008-10-15 at 10:50 -0700, Sashi Kant wrote:
> > We experiencing lots of RX packets being dropped on LVS server. 
> > Moving some high traffic VIPs to other LVS which reduced rate of packet
> > drops however not completely eliminated. 
> > 
> > Anyone seen this issue or have insight into it ?
> Nope.
> I might suggest, however, that the combination of Broadcom NICs, bonded,
> and the Cisco device is the cause of the problem.
> If you switch to a single NIC (for testing) with no bonding, does the
> problem go away or does it get worse?
> It seems that you may need to do some system tuning to get your system
> to handle the high packet rates you have - it's likely the pps, not the
> bps, which is the source of the problem.
> What Broadcom driver is being used?
> Graeme
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

More information about the lvs-users mailing list