[lvs-users] persistence and destination IP hashing

Malcolm Turnbull malcolm at loadbalancer.org
Fri Oct 24 14:21:36 BST 2014


David,

The dh scheduler only really works if the kernel can see the
destination address, what you need is for traffic passing through the
load balancer to be transparently load balanced to its destination....

So rather than clients requesting the load balancers VIP (virtual IP)...
You need to change the routing so that the clients request
www.microsoft.com or www.google.com directly BUT these requests are
routed through the load balancer....
Then you need to tell the load balancer to transparently intercept
that traffic with something like:

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 443 -j MARK --set-mark 1
ip rule add prio 100 fwmark 1 table 100
ip route add local 0/0 dev lo table 100

The same way you would with a transparent SQUID proxy....

Check out page 15 of the Loadbalancer.org web filter deployment guide
for more information about this kind of set up:
http://pdfs.loadbalancer.org/Web_Proxy_Deployment_Guide.pdf




On 24 October 2014 14:11, David Waddell <David.Waddell at owmobility.com> wrote:
> Hi
>     We are trying to use LVS as a virtual load balancer around a transparent http proxy, and are wondering if it is possible to use persistence and destination hashing together
>     (from our tests and as suggested in how-tos, persistence is CIP based, and therefore may not be so).
>
>     To explain our setup a little
> -           We have a pool of VMs running a transparent http proxy
> -          We have setup an LVS service on either side of this proxy with
>                     -  A 'request' path service that will schedule requests into the proxy, using src based hashing
>                     -  A 'response' path service that will schedule the http responses through the proxy, using dst based hashing.
> -          The http proxy maintains state around the 'session' (set of urls requested by client ip),  so we wish to direct clients to the same proxy instance.
>             Src hashing allows to achieve this, and dst hashing on the response path ensure TCP connections get established correctly.
>
>      An issue arises when we try to add new instances of the proxy to the pool.
>      The hashing changes, which breaks the statefulness (user request  may go to different server).
>      To that extent, we added persistence, which worked for requests.
>
>     However, persistence on the response path is sending , perhaps as expected,  TCP packets to the 'wrong' proxy instance in a lot of cases.
>     This is because the persistence logic is using the web server IP address (the CIP on the response path).
>
> An  example  of the problem for us :
>
> Using 2 client ips 172.29.0.12, 172.29.0.11;    real servers (http proxies) 192.168.90.58, 192.168.90.59; web server 192.168.10.17.
>
> IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn FWM  1 sh persistent 60
>   -> 192.168.90.58:0              Route   1      0          0
>   -> 192.168.90.59:0              Route   1      0          0 FWM  3 dh persistent 60
>   -> 192.168.90.58:0              Route   1      0          0
>   -> 192.168.90.59:0              Route   1      0          0
>
> FWM1 represents the request path; FMW 2 the response path.
>
>        IPVS connection entries
>        pro             expire        state                source                                  virtual                            destination        pe name        pe_data
> A) TCP       00:58        ESTABLISHED  172.29.0.12:45659            192.168.10.17:80        192.168.90.58:80
> B) IP           00:58        NONE                192.168.10.17:0                  0.0.0.3:0                       192.168.90.59:0
> C) TCP       01:55        FIN_WAIT        172.29.0.11:50919            192.168.10.17:80        192.168.90.59:80
> D) IP          00:55        NONE                 172.29.0.11:0                     0.0.0.1:0                         192.168.90.59:0
> E) TCP       00:59        SYN_RECV        192.168.10.17:80             172.29.0.12:14038       192.168.90.59:14038
> F) TCP       01:55        FIN_WAIT         192.168.10.17:80             172.29.0.11:42443       192.168.90.59:42443
> G) IP          00:58        NONE                 172.29.0.12:0                    0.0.0.1:0                         192.168.90.58:0
>
> In the example above,  C and F represent a successful proxied http request, C the request from client to proxy, F the response from web server to proxy.
> The request C/F was made first and establishes a persistent connections  from client  172.29.0.11 ->192.168.90.59 proxy  and for web server 192.168.10.17 to proxy 192.168.90.59.
> All well.
>
> We subsequently make a request A) from client 172.29.0.11, src hashing places this correctly on proxy 192.168.90.59.
> The proxy then requests to web server, and the response is shown in E) - persistence directs the response to proxy 192.168.90.58, meaning a TCP connection 192.168.90.59 <-> 192.168.10.17 cannot be established.
>
>     Obviously re-engineering of the proxy instances to share state would be the ideal solution, as then persistence would not be required.
>      But we are wondering if there is any means to combine destination hashing and persistence successfully ?  As currently persistence is enforcing scheduling based on src IP, even when dh is specified.
>
> Thanks
> David
>
>
>
>
>
>
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/



More information about the lvs-users mailing list