lvs-dr with freebsd jailhost as realservers

Mon Apr 9 20:45:54 BST 2007

Hello All,

I've setup a two nic lvs-dr machine which is able to contact a host on 
the the local ethernet segment that is partitioned using freebsd jails 
(It works brilliantly with standalone freebsd hosts, or linux hosts).

ipvsadm has no trouble getting to my jailhosted webservers (these two 
are both jailed)

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  66.207.199.194:80 rr
  -> 66.207.199.213:80            Route   1      0          0
  -> 66.207.199.212:80            Route   1      0          0

 From the jailhost, from the raw socket, I can see all incoming traffic 
destined for the jailhosts, when I sniff traffic (ftcpdump -vv | grep 
66.207.193.249 | grep http) for a normal http session between my ip and 
66.207.199.212:80, I see this:

15:37:40.556183 IP (tos 0x0, ttl  60, id 40758, offset 0, flags [DF], 
proto: TCP (6), length: 60) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: S, cksum 0x5230 (correct), 
2098400119:2098400119(0) win 5840 <mss 1460,sackOK,timestamp 6001729 
0,nop,wscale 2>
15:37:40.556223 IP (tos 0x0, ttl  64, id 10067, offset 0, flags [DF], 
proto: TCP (6), length: 64, bad cksum 0 (->3f9)!) 
jailone.web0.beanfield.net.http > 66.207.193.249.45601: S, cksum 0x0f9f 
(incorrect (-> 0xa62c), 4171180494:4171180494(0) ack 2098400120 win 
65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 1740598 6001729,sackOK,eol>
15:37:40.558266 IP (tos 0x0, ttl  60, id 40759, offset 0, flags [DF], 
proto: TCP (6), length: 52) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: ., cksum 0xe042 (correct), 1:1(0) ack 1 
win 1460 <nop,nop,timestamp 6001730 1740598>
15:37:40.562167 IP (tos 0x0, ttl  60, id 40760, offset 0, flags [DF], 
proto: TCP (6), length: 464) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: P 1:413(412) ack 1 win 1460 
<nop,nop,timestamp 6001731 1740598>
15:37:40.562361 IP (tos 0x0, ttl  64, id 10068, offset 0, flags [DF], 
proto: TCP (6), length: 448, bad cksum 0 (->278)!) 
jailone.web0.beanfield.net.http > 66.207.193.249.45601: P 1:397(396) ack 
413 win 33304 <nop,nop,timestamp 1740604 6001731>
15:37:40.564255 IP (tos 0x0, ttl  60, id 40761, offset 0, flags [DF], 
proto: TCP (6), length: 52) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: ., cksum 0xdc07 (correct), 413:413(0) 
ack 397 win 1728 <nop,nop,timestamp 6001731 1740604>
15:37:40.735856 IP (tos 0x0, ttl  60, id 40762, offset 0, flags [DF], 
proto: TCP (6), length: 395) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: P 413:756(343) ack 397 win 1728 
<nop,nop,timestamp 6001774 1740604>
15:37:40.736054 IP (tos 0x0, ttl  64, id 10069, offset 0, flags [DF], 
proto: TCP (6), length: 544, bad cksum 0 (->217)!) 
jailone.web0.beanfield.net.http > 66.207.193.249.45601: P 397:889(492) 
ack 756 win 33304 <nop,nop,timestamp 1740771 6001774>
15:37:40.738104 IP (tos 0x0, ttl  60, id 40763, offset 0, flags [DF], 
proto: TCP (6), length: 52) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: ., cksum 0xd6e5 (correct), 756:756(0) 
ack 889 win 1996 <nop,nop,timestamp 6001775 1740771>
15:37:45.938653 IP (tos 0x0, ttl  64, id 10109, offset 0, flags [DF], 
proto: TCP (6), length: 52, bad cksum 0 (->3db)!) 
jailone.web0.beanfield.net.http > 66.207.193.249.45601: F, cksum 0x0f93 
(incorrect (-> 0x490f), 889:889(0) ack 756 win 33304 <nop,nop,timestamp 
1745772 6001775>
15:37:45.977668 IP (tos 0x0, ttl  60, id 40764, offset 0, flags [DF], 
proto: TCP (6), length: 52) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: ., cksum 0xbe3d (correct), 756:756(0) 
ack 890 win 1996 <nop,nop,timestamp 6003085 1745772>
15:37:58.148838 IP (tos 0x0, ttl  60, id 40765, offset 0, flags [DF], 
proto: TCP (6), length: 52) 66.207.193.249.45601 > 
jailone.web0.beanfield.net.http: F, cksum 0xb25a (correct), 756:756(0) 
ack 890 win 1996 <nop,nop,timestamp 6006127 1745772>
15:37:58.148857 IP (tos 0x0, ttl  64, id 10190, offset 0, flags [DF], 
proto: TCP (6), length: 52, bad cksum 0 (->38a)!) 
jailone.web0.beanfield.net.http > 66.207.193.249.45601: ., cksum 0x0f93 
(incorrect (-> 0x0a38), 890:890(0) ack 757 win 33303 <nop,nop,timestamp 
1757507 6006127>

However, when I go through the lvs vip, (66.207.199.194), the webserver 
fails to return to the request originator. Each time I put through a 
request, I see the inactive connection incrementing, but only four out 
of ten times do I see a request that references my origination ip:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  66.207.199.194:80 rr
  -> 66.207.199.213:80            Route   1      0          3
  -> 66.207.199.212:80            Route   1      0          3

15:39:56.834578 IP (tos 0x0, ttl  60, id 1645, offset 0, flags [DF], 
proto: TCP (6), length: 60) 66.207.193.249.50768 > 
66-207-199-194.beanfield.net.http: S, cksum 0x9d62 (correct), 
2230198073:2230198073(0) win 5840 <mss 1460,sackOK,timestamp 6035796 
0,nop,wscale 2>
15:39:56.834590 IP (tos 0x0, ttl  64, id 11041, offset 0, flags [DF], 
proto: TCP (6), length: 40, bad cksum 0 (->55)!) 
66-207-199-194.beanfield.net.http > 66.207.193.249.50768: R, cksum 
0x0f75 (incorrect (-> 0x35ad), 0:0(0) ack 2230198074 win 0
15:40:49.877116 IP (tos 0x0, ttl  60, id 62909, offset 0, flags [DF], 
proto: TCP (6), length: 60) 66.207.193.249.50769 > 
66-207-199-194.beanfield.net.http: S, cksum 0xccf2 (correct), 
2284632221:2284632221(0) win 5840 <mss 1460,sackOK,timestamp 6049056 
0,nop,wscale 2>
15:40:49.877132 IP (tos 0x0, ttl  64, id 11393, offset 0, flags [DF], 
proto: TCP (6), length: 40, bad cksum 0 (->fef4)!) 
66-207-199-194.beanfield.net.http > 66.207.193.249.50769: R, cksum 
0x0f75 (incorrect (-> 0x9909), 0:0(0) ack 2284632222 win 0
15:41:14.669517 IP (tos 0x0, ttl  60, id 16970, offset 0, flags [DF], 
proto: TCP (6), length: 60) 66.207.193.249.50771 > 
66-207-199-194.beanfield.net.http: S, cksum 0x4463 (correct), 
2315462431:2315462431(0) win 5840 <mss 1460,sackOK,timestamp 6055253 
0,nop,wscale 2>
15:41:14.669528 IP (tos 0x0, ttl  64, id 11563, offset 0, flags [DF], 
proto: TCP (6), length: 40, bad cksum 0 (->fe4a)!) 
66-207-199-194.beanfield.net.http > 66.207.193.249.50771: R, cksum 
0x0f75 (incorrect (-> 0x28af), 0:0(0) ack 2315462432 win 0

I'm looking to see if anyone has a working lvs-dr setup with freebsd 
jails, or possibly someone who has some suggestions on how I can break 
this flow down surgically to figure out of this is a misconfigured dr 
setup or an incompatiblity with the freebsd jails architecture.

Thanks for your time.

M