DoS protection strategies

Roberto Nibali ratz at drugphish.ch
Tue Apr 18 21:22:50 BST 2006


Ken,

I reckon people have different setups and thus our views seem to differ 
a bit. Without proper numbers and test conduct configurations that are 
repeatable by other persons, we'll probably compare apples to oranges. 
That said, I should nevertheless like to comment on this based on my 
limited experience with web services.

> This 150 connection limit is the default MaxClients setting in Apache, 
> which in practice should be adjusted as high as you can without Apache 
> using more memory than you want it to (e.g., 80-100% of available RAM -- 
> no need for the DoS to swap-kill your box, too).  Each Apache process 
> will use several megabytes (higher or lower depending on 32- or 64-bit 
> platforms, add-on modules, etc) so this can't be set too high.

Most of our customer's httpd show RSS between 800KB and 2MB; some of 
them it's including mod_perl or mod_php.

> Disabling KeepAlives will drop your working process count by roughly an 
> order of magnitude, and unless you're just serving static content it's 
> generally worth disabling.  But for your case of 150 idle connections, 
> it doesn't help.

Or you set your timeouts correctly, or you implement proper state 
mapping using a reverse proxy and a cookie engine.

> Netfilter has various matching modules that can limit connections from 
> and/or to specific IPs, for example:
> 
> iptables --table filter --append VIP_ACCEPT --match dstlimit --dstlimit 
> 666 --dstlimit-mode srcipdstip-dstport --dstlimit-name VIP_LIMIT --jump 
> ACCEPT

No wonder you have no memory left on your box :).

> The reason DoS attacks are so successful (especially full-handshake 
> attacks) is that something needs to be able to cache and handle incoming 
> connections.  And that is exactly where Apache is weakest -- the process 
> model is terrible at handling a high number of simultaneous, quasi-idle 
> connections.

100% agreed.

> LVS has some DoS prevention settings which you should consider 
> (drop_entry, drop_packet, secure_tcp) but they're generally only useful 
> for SYN floods.

I would not call dropping a certain amount of illegitimate _and_ 
legitimate connections to be too useful when you're running on a strict 
SLA. QoS approaches based on window scaling help a bit more.

> A full handshake will be passed on through LVS to the 
> application, and that is where the resources must be available.  And 
> given persistence, a single-IP attack will be extremely effective if you 
> only have one (or few) real servers.

Yep.

> Once a connection has been made to Apache, it will need to either 
> relegate idle connections out of process (see Apache 2.2's new event 
> MPM, not sure if it only works on idle keepalives) or limit based on IP 
> with the modules you mention.
> 
> This problem is difficult to solve completely, and I agree that solving 
> it in Apache is the least powerful, least convenient, and highest 
> overhead solution.  Given Netfilter functionality (2.6 and later), the 
> absence of throttles or connection limits in LVS isn't fatal.  But I do 
> feel that LVS could be made a more comprehensive system if it rolled in 
> even basic connection throttling/limiting

There is the overflow mode, I haven't submitted the new code yet, but 
you can already limit the amount of connections per real server. My 
2.4.x IPVS patches regarding the overflow mode are done (in case you 
want to test), since we still deploy mostly 2.4.x kernels.

Regarding throttling, I reckon you use the TC framework available in the 
Linux kernel since long before netfilter hit the tables.

>, plus a more closely 
> integrated and maintained health checking system.

How would you improve it? Suggestions are always welcome.

> And source-routing 
> support. ;)

Which you did :).

> There are commercial products available that implement heavy-duty 
> DoS/intrusion protection.

True. And have you looked at how they do it? Traffic contracts and the 
sorts, like TC for example.

>  They block the vast majority of simple 
> attacks and are crucial for any large-scale public-facing services.

Blocking or dropping is not acceptable, diverting or migrating is. The 
biggest issue on large-scale web services according to my experience is 
the detection of malicious requests.

>  But 
> a good distributed full-handshake or even valid HTTP request DoS is 
> almost impossible to fully block.

Agreed ;).

Best regards,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc


More information about the lvs-users mailing list