[lvs-users] Servers unable to flush data to clients
brian.staszewski at farheap.com
Fri Jun 8 19:50:33 BST 2007
----- "Joseph Mack NA3T" <jmack at wm7d.net> wrote:
| On Wed, 16 May 2007, Brian Staszewski wrote:
| > Hey everyone. I implemented an LVS-NAT/ldirectord setup
| > for a web site and I'm having some strange problems. What
| > appears to be happening is that somehow the real servers
| > are not always able
| ^^^ ^^^^^^ ^^^^
| sometimes they can, sometimes they can't?
Correct. More often than not everything is fine. I estimate that the problem occurs less than 5% of the time.
| > to send data back to the client that
| > made the request. When I view apache's status page
| > (/server-status/) I'll see many connections in the "W"
| > (Sending Reply) state, and they'll stay there until they
| > either time out or I restart apache.
| this is a strange one. This will require some tracking down.
| do you have any iptables rules on the machines? If so, kill
| them and have your firewalling done elsewhere for the
Stopping iptables makes the problem go away! I don't understand why, as my rules are fairly simple... let ssh, heartbeat, icmp, http, and https in, let anything out, and drop everything else:
###Begin iptables script
# Flush all rules
# Setting default filter policy
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
# Allow unlimited traffic on loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
# Allow incoming ssh, heartbeat, http only
iptables -A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp --dport 694 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow web servers access to teh interweb
iptables -A FORWARD -i eth1 -j ACCEPT
iptables -A FORWARD -o eth1 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
# Allow all internal connections
iptables -A INPUT -i eth1 -j ACCEPT
# allow icmp
iptables -A INPUT -p icmp -j ACCEPT
# Allow all outgoing traffic
iptables -A OUTPUT -j ACCEPT
# log stuff we're gonna drop
iptables -A INPUT -m limit --limit 60/minute -j LOG --log-level 7 --log-prefix "Dropped by firewall: "
# make sure nothing else comes in, anything may go out
iptables -A INPUT -j DROP
###End iptables script
| do you know anything about the requests that don't get reply
| packets? do they go through a database, are they POST's, do
| they retrieve gifs from other servers, do they have a packet
| size divisable by 13 (j.k. but you're going to have to find
| a pattern and neither of us know what to look for)
| is there one request that you can make that will fail often
| enough to test the problem, even if you don't know what's
| characteristic about it?
As far as I can tell there's no trend in the requests that don't get back to the clients.
| does tcpdump tell you anything about the packets that aren't
| getting through? is there a shower of icmps coming back for
| these packets?
To be honest I don't know enough about tcpdump to look for strange happenings. I can answer your second question though - no, no icmp showers.
| is the default gw for the affected server(s) being flipped
| to the backup director? (force one director to stay as the
| active director for the tests?)
No. The primary director remains active the entire time.
| > I've tried with and without persistence, and it has made no
| persistence it to do with the forwarding, not the tcpip
| Joseph Mack NA3T EME(B,D), FM05lw North Carolina
| jmack (at) wm7d (dot) net - azimuthal equidistant map
| generator at http://www.wm7d.net/azproj.shtml
| Homepage http://www.austintek.com/ It's GNU/Linux!
| LinuxVirtualServer.org mailing list -
| lvs-users at LinuxVirtualServer.org
| Send requests to lvs-users-request at LinuxVirtualServer.org
| or go to http://www.in-addr.de/mailman/listinfo/lvs-users
So, I guess that the problem lies with iptables or LVS' interaction with iptables, but I don't know why or where to look next.
More information about the lvs-users