[lvs-users] lvs. Nat

[daryl herzmann]

Hello,

Are you using a broadcomm NIC?  Perhaps you are hitting the issue with GRO 
enabled causing brutal performance.  If so, you could try disabling GRO 
and see if things are speedy.

ethtool -K eth0 gro off
ethtool -K eth1 gro off

https://bugzilla.redhat.com/show_bug.cgi?id=854066

daryl

On Thu, 1 May 2014, net.study.sea at gmail.com wrote:

>
> Thanks for reply!
>
> I have configured the reply going through director ,but the reply is very slowly.
>
> 在 2014-4-30，22:36，Anders Henke <anders.henke at 1und1.de> 写道：
>
>> On 30.04.2014, net.study.sea at gmail.com wrote:
>>> 在 2014-4-30，19:56，Anders Henke <anders.henke at 1und1.de> 写道：
>>>
>>>> There are multiple issues with this.
>>>> -IP-address changes
>>>> -possible port changes
>>>>
>>>>
>>>> The client initiates a connection from Client-IP A to Director-VIP B:
>>>>
>>>> A->B, using a random source port number to destination port 80.
>>>>
>>>> The director rewrites the IP packet as if it had been sent
>>>> from Client A to Realserver C and sends it to the realserver:
>>>> A->C, using the client's source port number to destination port 80.
>>>>
>>>> If the realserver would reply to those director-NATed packet,
>>>> the reply would look like this:
>>>>
>>>> C->A, from port 80 to client's source port
>>>
>>> At this time,if C could reply directly to A
>>> ,not through director ,then what is the result?
>>>
>>>
>>> After I configure well Lvs in NAT mode,
>>> I find it deal with client's request so slowly, can not find the reason.
>>
>> If you're balancing only a single tcp port, the director will reply
>> to any other tcp ports and protocols.
>>
>> So "ping" (icmp) will be answered from the director with the
>> correct IP address, and the attempt to access a port without
>> a listener on the director will result in "connection refused".
>>
>> When you're accessing the balanced tcp port, the realserver will reply
>> from an unexpected IP address and the client will drop the packet as
>> being invalid.
>>
>> After some timeout, the client will retry sending the packet,
>> which will also fail. So it's not "slow" for your client to access the balanced service, it's "impossible" to establish a working connection.
>>
>>
>> If you do want to bypass the director, ask yourself why you'd like to do so:
>> - you're expecting more outgoing bandwith than the director will
>>  be able to handle: you need to use either direct routing or tunneling.
>>
>> - you're not comfortable with the idea of making your director also
>>  your host's default gateway.
>>
>>  By choosing tunneling or direct routing, you'll gain that possibility,
>>  but maybe you don't want to care about the ARP problem (a minor
>>  configuration on every realserver - if you've forgotten one,
>>  that realserver may attract all traffic to be balanced) or
>>  about monitoring service X at IP address Y (the balanced IP
>>  is not directly accessible for monitoring, so all checks need
>>  to be performed at a different IP).
>>
>>  If it's just a mindset issue - you may see it that way: if your director
>>  is not available, your service is not available and it doesn't matter
>>  that you can't access your hosts. In order to re-establish service,
>>  you need to make your director available anyway, and after you've done so,
>>  you may need to take care about your hosts.
>>
>>
>>
>>
>> Anders
>>
>>
>>>> The client in turn knows it did create a connection from A to B,
>>>> but doesn't know about a connection from A to C and so can't
>>>> match those reply packets with "the other" connection. Well,
>>>> their destination port is known to be in use with a different connection,
>>>> so why should A assume this to be valid?
>>>>
>>>> The reply packets are being dropped as being invalid.
>>>>
>>>>
>>>>
>>>> Depending on your configuration, your director may also change
>>>> the destination port, e.g. from 80 to 8080:
>>>> A->C, using the client's source port number to destination port 8080
>>>>
>>>> ... and so the realserver's reply would look like this:
>>>>
>>>> C->A, from port 8080 to client's source port
>>>>
>>>> The client in turn knows it did create a connection from A to B.
>>>> Any replies from C won't match that connection and so are being dropped.
>>>> Neither IP address nor port number do even closely match a known connection,
>>>> there's no way for the client to match them.
>>>>
>>>> In both cases, any firewall protecting A will perform similar
>>>> checks and drop the reply packets, as they are not related to a known
>>>> outgoing connection.
>>>>
>>>> By routing and "un-NAT'ing" the reply packet via the director, the
>>>> director will rewrite the packet from 'C->A' to 'B->A' and rewrite
>>>> any port changes as well to meet the expectations of A (or any
>>>> firewall in between director and A).
>>>>
>>>>
>>>> In Direct Routing mode, the IP packet isn't changed at all, just
>>>> the ethernet destination MAC address ("outside" of the IP packet)
>>>> is changed. In Tunneling mode, the incoming IP packet is encapsulated and
>>>> isn't changed as well.
>>>>
>>>> That's why in Direct Routing and Tunneling mode, any replies will be sent
>>>> bypassing your director and your director will only see "incoming" packets.
>>>>
>>>> And that's also the reason why in NAT mode, both incoming and outgoing
>>>> bandwith are limited by the director, while in DR/TUN mode, the director
>>>> only limits the incoming bandwith. Beside this, the extra complexity of
>>>> performing NAT for incoming and outgoing packets does also affect the
>>>> overall performance of the director.
>>>>
>>>>
>>>> Best,
>>>>
>>>> Anders
>>>>
>>>>
>>>>
>>>> On 30.04.2014, net.study.sea at gmail.com wrote:
>>>>> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>>>>>
>>>>> 在 2014-4-29，22:43，Anders Henke <anders.henke at 1und1.de> 写道：
>>>>>
>>>>>> On 29.04.2014, net.study.sea at gmail.com wrote:
>>>>>>> In nat mode,does director do dnat and snat packets for all request packets?
>>>>>>
>>>>>> In NAT mode, the director does perform destination nat on request packets,
>>>>>> so your realserver does still see the correct "source" ip address.
>>>>>>
>>>>>> However, replies then need to be sent via the director as well, so the
>>>>>> director can reverse the changed destination IP address and the client
>>>>>> does see the reply packet from the "expected" IP address/port.
>>>> --
>>>> 1&1 Internet AG              Expert Systems Architect (IT Operations)
>>>> Brauerstrasse 50             v://49.721.91374.0
>>>> D-76135 Karlsruhe            f://49.721.91374.225
>>>>
>>>> Amtsgericht Montabaur HRB 6484
>>>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>>>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>>>> Jan Oetjen, Christian Würst
>>>> Aufsichtsratsvorsitzender: Michael Scheeren
>> --
>> 1&1 Internet AG              Expert Systems Architect (IT Operations)
>> Brauerstrasse 50             v://49.721.91374.0
>> D-76135 Karlsruhe            f://49.721.91374.225
>>
>> Amtsgericht Montabaur HRB 6484
>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>> Jan Oetjen, Christian Würst
>> Aufsichtsratsvorsitzender: Michael Scheeren
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

-- 
/**
  * Daryl Herzmann
  * Assistant Scientist -- Iowa Environmental Mesonet
  * http://mesonet.agron.iastate.edu
  */