[lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

[Eric Robinson]

Malcom -- My wife makes wonderful fudge at Christmas time. Where should I send yours? 

Because...

ethtool -K p2p1 gro off
ethtool -K p2p2 gro off 
ethtool -K p2p3 gro off 
ethtool -K p2p4 gro off

...totally fixed it! The change was instantaneous. I didn't even have to down/up the interfaces.

I'm sitting here stunned. The application isn't freezing any more. I hardly know what to do with myself.

Thanks!

--Eric

-----Original Message-----
From: lvs-users-bounces at linuxvirtualserver.org [mailto:lvs-users-bounces at linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
Sent: Saturday, September 19, 2015 12:05 AM
To: LinuxVirtualServer.org users mailing list. <lvs-users at linuxvirtualserver.org>
Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

Eric,

Ah , OK I thought that might be too simple :-).

Some general advice from our end is:
We have had problems with both LRO & GRO in the past (we disable them both), and we also run irqbalance (to save on CPU overhead from network traffic).
Also we tend to use Intel NICs with the latest drivers.
We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I can't be much help.





On 18 September 2015 at 23:54, Eric Robinson <eric.robinson at psmnv.com> wrote:
> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>
> --Eric
>
>
> -----Original Message-----
> From: lvs-users-bounces at linuxvirtualserver.org 
> [mailto:lvs-users-bounces at linuxvirtualserver.org] On Behalf Of Malcolm 
> Turnbull
> Sent: Friday, September 18, 2015 2:20 PM
> To: LinuxVirtualServer.org users mailing list. 
> <lvs-users at linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went 
> to Sh*t
>
> Eric,
>
> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>
> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson at psmnv.com> wrote:
>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>
>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no 
>> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no 
>> destination available: no destinations present Sep 18 09:06:29 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present 
>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no 
>> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no 
>> destination available: no destinations present Sep 18 09:06:34 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present 
>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no 
>> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no 
>> destination available: no destinations present Sep 18 09:06:38 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present 
>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no 
>> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no 
>> destination available: no destinations present Sep 18 09:06:43 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present 
>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no 
>> destinations present
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - 
>> lvs-users at LinuxVirtualServer.org Send requests to 
>> lvs-users-request at LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org 
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org 
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org Send requests to lvs-users-request at LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users