From derrickltl at gmail.com  Wed Jul  8 12:29:11 2020
From: derrickltl at gmail.com (Derrick Lim)
Date: Wed, 08 Jul 2020 11:29:11 -0000
Subject: [lvs-users] packet occasionally gets forwarded to the wrong server
Message-ID: <CAGB=RwKfaONXyZVKJNc=hWX9iA-YTQA12=YqbPYCRxVMnUoZ9w@mail.gmail.com>

Hi all,

I've a number of IPVS directors using the source hashing scheduler
(sh, sh-fallback) and TUN method to load balance to a number of real
servers with the same weight. All the IPVS directors are configured
with the same settings and BGP is used to advertise the VIP address on
to the network. All IPVS directors run independently and do not sync
connections.

It is my understanding that sh is a consistent hashing algorithm that,
regardless of which IPVS director traffic from the same connection
(5-tuple) ends up on, the packet will be forwarded to the same real
server. Is this understanding correct? Should I be moving to maglev
instead? I'm on ubuntu 18.04, which is why I am using sh instead of
maglev.

However, very occasionally, a small number of packets from the same
connection gets forwarded to a different real server when they end up
on a different IPVS director. This causes the real server to send a
RST since it has no idea about the connection. Why does this happen?

ipvsadm -L -n looks like this:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  a.b.c.d:443 sh (sh-fallback)
  -> x.y.z.1:443              Tunnel  10     0          0
  -> x.y.z.2:443              Tunnel  10     0          0
  -> x.y.z.3:443              Tunnel  10     0          0
  -> x.y.z.4:443              Tunnel  10     0          0

I believe this issue is similar to what Philip Moore asked about in
2019-11, which led me to discover that I needed to have the real
servers in the same sequence on each director. However, that didn't
solve my issue, which is why I am asking for help here.

Thanks in advance for any help.

Regeards,
Derrick


From tangyang at zhihu.com  Thu Jul  9 04:57:09 2020
From: tangyang at zhihu.com (=?UTF-8?B?5ZSQ6ZizLeaKgOacr+S4reWPsA==?=)
Date: Thu, 09 Jul 2020 03:57:09 -0000
Subject: [lvs-users] estimation_timer take too much cpu usage.
Message-ID: <CACBoFd+fNS=m8jsZSvzW9h9UxuhO61TWSZM91t5yrndjufw+Vg@mail.gmail.com>

On my kubernetes node , there are some tcp packets randomly delayed between
two containers on one node.
I notice that estimation_timer softirq execute about 30ms+.(traced by this
tool: https://github.com/bytedance/trace-noschedule)

there are about 3000+ local address shown by 'ipvsadm -l'.

Hyper threading is enabled and there are 72 logical cores on my machine.

Does anyone know how to reduce the execution time of estimation_timer
softirq?

Thank you.