[lvs-users] Port mapping with LVS-DR using fwmark

Jacoby Hickerson hickersonjl at gmail.com
Fri Jan 24 02:19:11 GMT 2014


Just to clarify the packets are going to the loopback of node 1, when they
should be going to node 2.  This is shown in the tcpdump output:
Here is the output from the lo device of the first node:
02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length
0
02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq
0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
length 173
02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack 2,
win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq
173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length
0

Thanks,
Jacoby


On Wed, Jan 22, 2014 at 7:04 PM, Jacoby Hickerson <hickersonjl at gmail.com>wrote:

> I spoke to soon about this configuration working, the output of ipvsadm
> lead me to believe connections and packets were being load balanced,
> however they are now all coming from the real server which also is the
> director.  I'll call this the 'first node'.
>
> Here is some more info in addition to the details in my previous emails:
> First node routing:
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> 0.0.0.0         172.17.0.16    0.0.0.0         UG    0      0        0
> bond0
> 172.17.0.0      0.0.0.0         255.255.0.0    U     0      0        0
> bond0
>
> Second node routing:
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> 0.0.0.0         172.17.0.16     0.0.0.0         UG    0      0        0
> bond0
> 172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0
> bond0
>
> The second node also has the VIP on the loopback device
>
> How I setup the director
> ipvsadm -C
> ipvsadm -A -f 100 -s rr
> ipvsadm -a -f 100 -r 172.17.0.16 -w 100
> ipvsadm -a -f 100 -r 172.17.0.17 -w 100
>
> After conducting a few wget's here are the expected results from ipvsadm
> while not port mapping:
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes
> OutBytes
>   -> RemoteAddress:Port
> FWM  100                                12       61        0     5416
>    0
>   -> 172.17.0.16:0                       6       25        0     2422
>    0
>   -> 172.17.0.17:0                       6       36        0     2994
>    0
>
> Actual results from ipvsadm when using port mapping:
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes
> OutBytes
>   -> RemoteAddress:Port
> FWM  100                                 2        8        4      778
>  216
>   -> 172.17.0.16:0                       1        4        4      389
>  216
>   -> 172.17.0.17:0                       1        4        0      389
>    0
>
> Packets are being sent from the RIP of the first node only.  From my
> understanding when using DR OutPkts should always be zero.
>
> Output from tcpdump is a bit strange (first pass):
> 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq
> 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
> 0,nop,wscale 7], length 0
> 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq
> 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
> 0,nop,wscale 7], length 0
> 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [S], seq
> 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr
> 0,nop,wscale 7], length 0
> 02:10:51.986814 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq
> 2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val
> 978483 ecr 3044575792,nop,wscale 7], length 0
> 02:10:51.986919 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq
> 2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val
> 978483 ecr 3044575792,nop,wscale 7], length 0
> 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1,
> win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 0
> 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1,
> win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 0
> *02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
> 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length
> 0*
> 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq
> 1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
> length 173
> 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq
> 1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
> length 173
> *02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq
> 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
> length 173*
> 02:10:51.987126 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174,
> win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
> 02:10:51.987128 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174,
> win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
> 02:10:51.987241 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1,
> ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
> 02:10:51.987245 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1,
> ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
> 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2,
> win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
> 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2,
> win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
> *02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
> 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0*
> 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174,
> ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
> 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174,
> ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
> *02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq
> 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length
> 0*
> 02:10:51.987527 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175,
> win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
> 02:10:51.987529 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175,
> win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0
>
> Second pass:
> 02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq
> 2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr
> 0,nop,wscale 7], length 0
> 02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq
> 2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr
> 0,nop,wscale 7], length 0
> 02:10:59.360519 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq
> 2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val
> 985857 ecr 3044583166,nop,wscale 7], length 0
> 02:10:59.360524 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq
> 2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val
> 985857 ecr 3044583166,nop,wscale 7], length 0
> 02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1,
> win 115, options [nop,nop,TS val 3044583166 ecr 985857], length 0
> 02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1,
> win 115, options [nop,nop,TS val 3044583166 ecr 985857], length 0
> 02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq
> 1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857],
> length 173
> 02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq
> 1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857],
> length 173
> 02:10:59.360751 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174,
> win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0
> 02:10:59.360753 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174,
> win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0
> 02:10:59.361058 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1,
> ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0
> 02:10:59.361062 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1,
> ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0
> 02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174,
> ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0
> 02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174,
> ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0
> 02:10:59.361331 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175,
> win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0
> 02:10:59.361334 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175,
> win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0
>
> The end result is that the packets are always coming from the first node
> and never balanced to the second node.
>
> Thanks for any further help, seems the solution is really close!
>
> Jacoby
>
>
> On Sat, Jan 18, 2014 at 12:44 AM, Julian Anastasov <ja at ssi.bg> wrote:
>
>>
>>         Hello,
>>
>> On Fri, 17 Jan 2014, Jacoby Hickerson wrote:
>>
>> > Thanks Julian!  After enabling CONFIG_IP_VS_NFCT and setting conntrack
>> to 1
>> > that resolved the problem.
>> > However, how leery should I be with it consuming memory?  Is there a
>> test to
>> > monitor this consumption?  Currently the nf_conntrack_max is set to the
>> > default: 65536
>>
>>         cat /proc/slabinfo | grep nf_conntrack
>> or 'slabtop' can show the object size used by conntracks.
>> It should be 240+ bytes. You can expect one conntrack per
>> IPVS connection. You can also see conntracks with
>> cat /proc/net/nf_conntrack | less
>>
>>         cat /proc/sys/net/netfilter/nf_conntrack_count shows
>> the current number of conntracks. You can look for 'count'
>> at __nf_conntrack_alloc() and nf_conntrack_free() to see
>> how it is implemented.
>>
>> Regards
>>
>> --
>> Julian Anastasov <ja at ssi.bg>
>>
>
>


More information about the lvs-users mailing list