Problems with IPVS

[Mindaugas]

>>  Dumps attached on previous e-mail were done on bond0 interface which is
>> facing proxy. tcpdumps done on proxy confirms the problem.
>
> Hehe, you definitely want to use all possible features of Linux 
> networking. How is your bonding configured, ALB? There is an outstanding 
> issue with regard to packet reassembly on bond devices using ALB. It's 
> highly unlikely that you're experiencing it, though. But this could 
> explain your not perfect looking ethereal :).

  People write features to be used. :) And yes, I'm using mode=balance-alb.

>>> How many devices are we talking about including Phone and proxy?
>>
>>  Phone, SGSN/GGSN, PIX firewall (one end of GRE is there), server, proxy.
>
> Excellent, thanks. Does the PIX belong to the carrier? I presume, the IP 
> addresses after the PIX are still non-publicly routeable IP addresses?

  We are the carrier. :) And all the addresses are on private networks.


>>>>  4. phone sends HTTP GET request;
>>>>  5. proxy ACKs packet 4;
>>> Only ACK? No data?
>>
>>  Yes.
>
> Window size? adv size?

  I think you'll be able to download tcpdumps now. I'm afraid to mess 
something since I'm too far from being TCP/IP guru. :)

>>  Proxy is CentOS4 Linux server running Squid.
>
> And you see nothing unusual in your squid logs when connecting with SE 
> phones?

  No.

>> # sysctl net.ipv4.tcp_fack net.ipv4.tcp_sack
>> net.ipv4.tcp_fack = 1
>> net.ipv4.tcp_sack = 1
>
> Does disabling (just for a test) SACK change anything?

  I'll try it later, after deleting morning e-mails and dealing with other 
problems.

>>  Once again sysctl says that no. Both on LVS server and on proxy.
> What are the kernel versions? (Sorry, if this is a dupe.)

# uname -a (LVS server)
Linux leben 2.6.9-42.0.3.ELsmp #1 SMP Fri Oct 6 06:28:26 CDT 2006 x86_64 
x86_64 x86_64 GNU/Linux

# uname -a (Squid server)
Linux scorpio 2.6.9-22.0.1.ELsmp #1 SMP Thu Oct 27 14:49:37 CDT 2005 x86_64 
x86_64 x86_64 GNU/Linux

  Both runs CentOS4.

>>>>  The problem is that LVS does not pass packets 11. to 14. to phone. 
>>>> Why?
>>> Because packet 8 was FIN and LVS is not stateful with regard to TCP 
>>> sessions and retransmits.
>>
>>  But phone did not acknowledged that FIN yet?
>
> Sure, but we act on first seen FIN regarding template expiration, IIRC:
>
> http://www.drugphish.ch/~ratz/IPVS/ip__vs__proto__tcp_8c.html#a36
>
> But I'd need to check the code again. Take this with a grain of salt.

  That's out of my league. :) But I still not understand why you act on 
first FIN? Other side still not ACKed and retransmissions can happen as we 
see in our case.

>>  If I setup DNAT instead of LVS then packets 11.-14. are sent to phone. 
>> In case of LVS they are not.
>
> So you get to see packets 11-14 on the outbound interface of LVS from 
> Squid, but never on the inbound interface (direction of PIX)? This is very 
> odd!

  Exactly. I see those packets on interface bond0 (which faces proxy). But I 
do not see them on interface netwap (facing PIX).
  iptables cannot be an issue since I accept everything now.

>>  And after phone receives those packets it sends ACK to packets 6. and 7. 
>> and then to 8.
> But only for DNAT.

  Yes.

  Mindaugas