[lvs-users] persistence and destination IP hashing

Malcolm Turnbull malcolm at loadbalancer.org
Fri Oct 24 15:11:46 BST 2014


David,

Ah I see , my mistake I read it too quickly...
As far as I'm aware the only thing you could try is to pre-populate
the hash algorithm i.e.

Setup 10 backend servers at the start...
But only have X of them with a weight >0
As you need more proxies then just modify the weight and the dh algo
should not need to re-hash?

I haven't tested this on LVS but I've seen it done on HAProxy before
we helped implement proper stick tables.







On 24 October 2014 14:59, David Waddell <David.Waddell at owmobility.com> wrote:
> Hi Matt,
>
>     Thanks for the reply.
>
>     We have a setup much like you describe - requests are being made directly to web server, and we route these to the lb and the LB processes using fwms.
>    The DH algo is working fine for us;  but when we add persistence, we have a problem.
>     As soon as the persistence logic establishes affinity between a proxy and a web server, we cannot establish a TCP connection from any other proxy to that web server.
>
>     The CIP based persistence is overriding the DIP hashing on the path back from the web server to the proxy.
>
>     Session state on the proxy is the reason we are attempting this; if it is possible to combine DH and persistence, it would resolve the problem for us.
>
>
> Thanks
> David
>
>
> -----Original Message-----
> From: lvs-users-bounces at linuxvirtualserver.org [mailto:lvs-users-bounces at linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
> Sent: 24 October 2014 14:22
> To: LinuxVirtualServer.org users mailing list.
> Subject: Re: [lvs-users] persistence and destination IP hashing
>
> David,
>
> The dh scheduler only really works if the kernel can see the
> destination address, what you need is for traffic passing through the
> load balancer to be transparently load balanced to its destination....
>
> So rather than clients requesting the load balancers VIP (virtual IP)...
> You need to change the routing so that the clients request
> www.microsoft.com or www.google.com directly BUT these requests are
> routed through the load balancer....
> Then you need to tell the load balancer to transparently intercept
> that traffic with something like:
>
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --dport 443 -j MARK --set-mark 1
> ip rule add prio 100 fwmark 1 table 100
> ip route add local 0/0 dev lo table 100
>
> The same way you would with a transparent SQUID proxy....
>
> Check out page 15 of the Loadbalancer.org web filter deployment guide
> for more information about this kind of set up:
> http://pdfs.loadbalancer.org/Web_Proxy_Deployment_Guide.pdf
>
>
>
>
> On 24 October 2014 14:11, David Waddell <David.Waddell at owmobility.com> wrote:
>> Hi
>>     We are trying to use LVS as a virtual load balancer around a transparent http proxy, and are wondering if it is possible to use persistence and destination hashing together
>>     (from our tests and as suggested in how-tos, persistence is CIP based, and therefore may not be so).
>>
>>     To explain our setup a little
>> -           We have a pool of VMs running a transparent http proxy
>> -          We have setup an LVS service on either side of this proxy with
>>                     -  A 'request' path service that will schedule requests into the proxy, using src based hashing
>>                     -  A 'response' path service that will schedule the http responses through the proxy, using dst based hashing.
>> -          The http proxy maintains state around the 'session' (set of urls requested by client ip),  so we wish to direct clients to the same proxy instance.
>>             Src hashing allows to achieve this, and dst hashing on the response path ensure TCP connections get established correctly.
>>
>>      An issue arises when we try to add new instances of the proxy to the pool.
>>      The hashing changes, which breaks the statefulness (user request  may go to different server).
>>      To that extent, we added persistence, which worked for requests.
>>
>>     However, persistence on the response path is sending , perhaps as expected,  TCP packets to the 'wrong' proxy instance in a lot of cases.
>>     This is because the persistence logic is using the web server IP address (the CIP on the response path).
>>
>> An  example  of the problem for us :
>>
>> Using 2 client ips 172.29.0.12, 172.29.0.11;    real servers (http proxies) 192.168.90.58, 192.168.90.59; web server 192.168.10.17.
>>
>> IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags
>>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn FWM  1 sh persistent 60
>>   -> 192.168.90.58:0              Route   1      0          0
>>   -> 192.168.90.59:0              Route   1      0          0 FWM  3 dh persistent 60
>>   -> 192.168.90.58:0              Route   1      0          0
>>   -> 192.168.90.59:0              Route   1      0          0
>>
>> FWM1 represents the request path; FMW 2 the response path.
>>
>>        IPVS connection entries
>>        pro             expire        state                source                                  virtual                            destination        pe name        pe_data
>> A) TCP       00:58        ESTABLISHED  172.29.0.12:45659            192.168.10.17:80        192.168.90.58:80
>> B) IP           00:58        NONE                192.168.10.17:0                  0.0.0.3:0                       192.168.90.59:0
>> C) TCP       01:55        FIN_WAIT        172.29.0.11:50919            192.168.10.17:80        192.168.90.59:80
>> D) IP          00:55        NONE                 172.29.0.11:0                     0.0.0.1:0                         192.168.90.59:0
>> E) TCP       00:59        SYN_RECV        192.168.10.17:80             172.29.0.12:14038       192.168.90.59:14038
>> F) TCP       01:55        FIN_WAIT         192.168.10.17:80             172.29.0.11:42443       192.168.90.59:42443
>> G) IP          00:58        NONE                 172.29.0.12:0                    0.0.0.1:0                         192.168.90.58:0
>>
>> In the example above,  C and F represent a successful proxied http request, C the request from client to proxy, F the response from web server to proxy.
>> The request C/F was made first and establishes a persistent connections  from client  172.29.0.11 ->192.168.90.59 proxy  and for web server 192.168.10.17 to proxy 192.168.90.59.
>> All well.
>>
>> We subsequently make a request A) from client 172.29.0.11, src hashing places this correctly on proxy 192.168.90.59.
>> The proxy then requests to web server, and the response is shown in E) - persistence directs the response to proxy 192.168.90.58, meaning a TCP connection 192.168.90.59 <-> 192.168.10.17 cannot be established.
>>
>>     Obviously re-engineering of the proxy instances to share state would be the ideal solution, as then persistence would not be required.
>>      But we are wondering if there is any means to combine destination hashing and persistence successfully ?  As currently persistence is enforcing scheduling based on src IP, even when dh is specified.
>>
>> Thanks
>> David
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
>> Send requests to lvs-users-request at LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/



More information about the lvs-users mailing list