[lvs-users] Port mapping with LVS-DR using fwmark

Jacoby Hickerson hickersonjl at gmail.com
Sat Jan 18 01:13:20 GMT 2014


Thanks Julian!  After enabling CONFIG_IP_VS_NFCT and setting conntrack to 1
that resolved the problem.

However, how leery should I be with it consuming memory?  Is there a test
to monitor this consumption?  Currently the nf_conntrack_max is set to the
default: 65536

Also yes the RIP and VIP are on the same box with local and non-local
clients.

Jacoby


On Fri, Jan 17, 2014 at 1:39 PM, Julian Anastasov <ja at ssi.bg> wrote:

>
>         Hello,
>
> On Thu, 16 Jan 2014, Jacoby Hickerson wrote:
>
> > I've searched Google and this mailing list but haven't quite seen the
> same
> > configuration and/or setup as mine.
> >
> > The ldirectord documentation states that port mapping on the same server
> > where the director resides is not possible other than masq, however it
> says
> > "non-fwmark".  My setup is using fwmark, however, when trying to port map
> > from port 80 to another port, the client connection hangs.  Here are the
> > exact details of my setup:
> >
> > The VIP is on the same box as the director and RIP 172.17.0.16.  This
> setup
>
>         VIPs are always on director. You mean RIP 172.17.0.16
> and VIP 172.17.0.24 are on same box? Then there are 2 cases:
> local and non-local client?
>
>         Only the masq forwarding method can change daddr and
> dport in packet and it should work even for local client and
> for local real server. Except if some recent regression happens.
>
> > works fine when no port mapping is being done, but I need to move the
> port
> > to something higher than 1024.
> >
> > virtual=172.17.0.24:80
> > real=172.17.0.16:50000 gate 100
> > real=172.17.0.17:50000 gate 100
> > service=http
> > scheduler=rr
> > protocol=tcp
> >   checktype=connect
> >   fwmark=100
> >
> > iptables:
> > iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> > --dport 80 -j MARK --set-xmark 0x64/0xffffffff
> > iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT
> > --to-ports 50000
> > iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT
> > --to-ports 50000
>
>         Do you have CONFIG_IP_VS_NFCT enabled and sysctl var
> /proc/sys/net/ipv4/vs/conntrack set to 1? IIRC, it is needed
> when packets modified by IPVS need to be inspected later by netfilter
> in POST_ROUTING or INPUT chain as is the case with the LocalNode
> feature. If "conntrack" is not enabled, IPVS marks the packets as
> untracked to avoid consuming memory for conntracks, the conntracks
> are created and destroyed with every packet.
>
>         If consuming memory is not a problem, you can find that
> conntrack=1 with tuned nf_conntrack_max is faster than conntrack=0.
>
>         Let me know if conntrack=1 does not help, so that
> we can track the problem further.
>
> > Issue:
> > curl -v 'http://172.17.0.24'
> > * About to connect() to 172.17.0.24 port 80 (#0)
> > *   Trying 172.17.0.24...
> >
> > 00:41:44.503581 IP 172.17.0.2.46099 > 172.17.0.24.80: Flags [S], seq
> > 1066084928, win 14600, options [mss 1460,sackOK,TS val 2520815062 ecr
> > 0,nop,wscale 7], length 0
> > 00:41:44.503581 IP 172.17.0.2.46099 > 172.17.0.24.80: Flags [S], seq
> > 1066084928, win 14600, options [mss 1460,sackOK,TS val 2520815062 ecr
> > 0,nop,wscale 7], length 0
> > 00:41:44.503658 IP 172.17.0.16.50000 > 172.17.0.2.46099: Flags [S.], seq
> > 824291086, ack 1066084929, win 14480, options [mss 1460,sackOK,TS val
> > 9521949 ecr 2520815062,nop,wscale 7], length 0
> > 00:41:44.503663 IP 172.17.0.16.50000 > 172.17.0.2.46099: Flags [S.], seq
> > 824291086, ack 1066084929, win 14480, options [mss 1460,sackOK,TS val
> > 9521949 ecr 2520815062,nop,wscale 7], length 0
> >
> > So the problem I'm having is that the source ip is not being translated
> by
> > iptables but sent via lvs as the RIP.  Is there a kernel option, iptables
> > option or ipvsadm option that would allow it to change it back to the
> VIP?
> >
> > Any help would be very appreciated!
> >
> > Jacoby
>
> Regards
>
> --
> Julian Anastasov <ja at ssi.bg>
>


More information about the lvs-users mailing list