[lvs-users] Servers unable to flush data to clients

Brian Staszewski brian.staszewski at farheap.com
Fri Jun 8 19:50:33 BST 2007


----- "Joseph Mack NA3T" <jmack at wm7d.net> wrote:
| On Wed, 16 May 2007, Brian Staszewski wrote:
| 
| > Hey everyone. I implemented an LVS-NAT/ldirectord setup 
| > for a web site and I'm having some strange problems. What 
| > appears to be happening is that somehow the real servers 
| > are not always able
|        ^^^ ^^^^^^ ^^^^
| 
| sometimes they can, sometimes they can't?

Correct. More often than not everything is fine. I estimate that the problem occurs less than 5% of the time.

| 
| > to send data back to the client that 
| > made the request. When I view apache's status page 
| > (/server-status/) I'll see many connections in the "W" 
| > (Sending Reply) state, and they'll stay there until they 
| > either time out or I restart apache.
| 
| this is a strange one. This will require some tracking down.
| 
| do you have any iptables rules on the machines? If so, kill 
| them and have your firewalling done elsewhere for the 
| moment.

Stopping iptables makes the problem go away! I don't understand why, as my rules are fairly simple... let ssh, heartbeat, icmp, http, and https in, let anything out, and drop everything else:

###Begin iptables script
#!/bin/sh

# Flush all rules
iptables -F
iptables -X

# Setting default filter policy
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

# Allow unlimited traffic on loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# Allow incoming ssh, heartbeat, http only
iptables -A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp --dport 694 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# Allow web servers access to teh interweb
iptables -A FORWARD -i eth1 -j ACCEPT
iptables -A FORWARD -o eth1 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT

# Allow all internal connections
iptables -A INPUT -i eth1 -j ACCEPT

# allow icmp
iptables -A INPUT -p icmp -j ACCEPT

# Allow all outgoing traffic
iptables -A OUTPUT -j ACCEPT

# log stuff we're gonna drop
iptables -A INPUT -m limit --limit 60/minute -j LOG --log-level 7 --log-prefix "Dropped by firewall: "

# make sure nothing else comes in, anything may go out
iptables -A INPUT -j DROP

###End iptables script

| 
| do you know anything about the requests that don't get reply 
| packets? do they go through a database, are they POST's, do 
| they retrieve gifs from other servers, do they have a packet 
| size divisable by 13 (j.k. but you're going to have to find 
| a pattern and neither of us know what to look for)
| 
| is there one request that you can make that will fail often 
| enough to test the problem, even if you don't know what's 
| characteristic about it?

As far as I can tell there's no trend in the requests that don't get back to the clients. 

| does tcpdump tell you anything about the packets that aren't 
| getting through? is there a shower of icmps coming back for 
| these packets?
 
To be honest I don't know enough about tcpdump to look for strange happenings. I can answer your second question though - no, no icmp showers.

| is the default gw for the affected server(s) being flipped 
| to the backup director? (force one director to stay as the 
| active director for the tests?)

No. The primary director remains active the entire time.

| > I've tried with and without persistence, and it has made no
| difference.
| 
| persistence it to do with the forwarding, not the tcpip 
| layer.
| 
| Joe
| 
| -- 
| Joseph Mack NA3T EME(B,D), FM05lw North Carolina
| jmack (at) wm7d (dot) net - azimuthal equidistant map
| generator at http://www.wm7d.net/azproj.shtml
| Homepage http://www.austintek.com/ It's GNU/Linux!
| _______________________________________________
| LinuxVirtualServer.org mailing list -
| lvs-users at LinuxVirtualServer.org
| Send requests to lvs-users-request at LinuxVirtualServer.org
| or go to http://www.in-addr.de/mailman/listinfo/lvs-users

So, I guess that the problem lies with iptables or LVS' interaction with iptables, but I don't know why or where to look next.

Thanks tons,
Brian




More information about the lvs-users mailing list