[lvs-users] LVS rejecting large packets

Roger Littin roger at rlmedia.co.nz
Thu Nov 24 01:10:43 GMT 2011

Hi Simon,

I didn't want to hijack the other thread so I have started a new one.

The issue I was seeing was with tcp packets to media servers behind LVS.
When the media streams are going out to the clients, everything is fine as the packets going through LVS are tiny.

When I try to send incoming streams through LVS, I start seeing issues sometimes where an icmp is sent back to the client stating the packet is too large to forward to the real server.  This causes the incoming stream to get further and further behind.

Originally, on all the servers, I had mtu set to 1500 and tcpdump would be showing some packets that were larger than this and would be rejected.  I increased the mtu to 9000 but then tcpdump simply reported that the same clients were sending packets larger than 9000 bytes. 

The last tests I did was on a server that I no longer have access to so I cannot be sure of the settings then.

My current servers are running Centos 5.7 2.6.18-274.7.1.el5 and have GRO off.  LRO is not listed.

[root at laxlb01 ~]# ethtool -k eth0
Offload parameters for eth0:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

I see that tcp segmentation offload is set to on.  Will this cause the same effect?

I would really like to get to the bottom of what is causing this problem so that I can run the incoming streams through the LVS.  At the moment, we are balancing these with DNS which is not ideal.



On 24/11/2011, at 1:09 PM, Simon Horman wrote:

> Hi Roger,
> if you are seeing that problem then I suggest turning off
> both GRO and LRO on the interface on the director that receives
> the packets.
> e.g.
> # ethtool -k eth0 | egrep '(receive-offload)'
> generic-receive-offload: on
> large-receive-offload: on
> # ethtool -K eth0 lro off
> # ethtool -K eth0 gro off
> # ethtool -k eth0 | egrep '(receive-offload)'
> generic-receive-offload: off
> large-receive-offload: off
> IPVS may be used in conjunction with GRO since 2.6.39,
> so it should not be necessary to disable GRO if
> a 2.6.39 is used.
> IPVS is not compatible with LRO as of 3.1 (or 3.2, when it is released)
> LRO is specific to TCP.
> GRO affects both TCP and UDP (and other protocols IIRC).

More information about the lvs-users mailing list