Apache Anti-DoS Preparation

I am NOT going to discuss here the definitions and concept. I wanted to put it down, based in what i faced before on the Subject line and how I tackled. I came up with some predictable symptoms which will definitely help some one to get some hint and how to manage this crisis.

I always believed that along with the networks and server admins, - application developer should accommodate a module based on security aspects(necessary but NOT essential), so that the prevention preparation becomes a bit more collaborative. But while trying to meet the application/product deadline line, specially in start-up kinda company, somehow this gets overlooked.

Apache being the Leading web server, its vulnerability has always been worked upon continuously. So there should not be any scope left from our side too, to fine tune this guy so that it is strong enough to combat DoS/DDoS kind of attack.

If you are a web server administrator and see the following symptoms, i would advice you to be more detail about your log analysis:

Symptoms

1. if the sites are suddenly serving slow, despite any new business marketing boost etc.
2. Lots of hanging processes, http://(Your-Apache-Admin-URL)/server-status. Read More.
   
3. You see same set of ip's requesting more than needed -
   netstat -punta | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n.
4. Server resource spike, with top command you see soaring high resource utilization.
5. You are left with very less no. of connection to serve the real traffic, with some intermittent 5xx error.

And Finally You Are Left With,

1. A hang server.
2. Web site down/ intermittent service delivery .
3. Gradually SSH also stops responding. (Lucky if you had already logged in!)
4. Need for a mandatory reboot in order to restore access to the system.

Prevention Preparation:

1. You should have a solid understanding of what is being served from your server, how much average rss memory consumtion per child, how much memory this server has, and based on that only you calculate the MPM settings. For ex.
   #ps -ylC httpd --sort:rss | awk '{ sum = sum + $8 } END { print sum/NR }' ; doing so gave me 28422.2 = approx 28 MB , Hence my MPM settings should be in proportionate to the available share of RAM for this apache web server.
   

   Thus the appropriate setting for MaxSpareServers would help apache not to over consume and ultimately ending up onsuming ethe ntire available memory resource causing a systems OOM error and hang up permanently.

2. Enable SYN COOKIES at the kernel level
   echo 1 > /proc/sys/net/ipv4/tcp_syncookies,
   along with this, lets also do a
   echo 1 > /proc/sys/net/ipv4/tcp_keepalive_probes
   echo 2 > /proc/sys/net/ipv4/tcp_synack_retries
   echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
   
   
3. Enable and Configure iptables to prevent the attack or at least work to identify the attack(this is regardless of apache specific, rather system wide)
   /sbin/iptables -N syn-fld
   /sbin/iptables -A syn-fld -m limit --limit 100/second --limit-burst 150 -j RETURN
   /sbin/iptables -A syn-fld -j LOG --log-prefix "SYN fld: "
   /sbin/iptables -A syn-fld -j DROP
4. Two must deploy tool,
   

   mod_ipconnlimit and mod_evasive module.

5. A fair knowledge of sed/awk/perl and regex shell tricks to do a quick analysis of your access logs.

- and the moment you identify some one(or even slightest suspect) drop all his request(across entire web cluster),

/sbin/iptables -A INPUT -p tcp -s --dport 80 -j REJECT ,
and later on, needless to say, don't forget to white list the same if it is NOT the culprit.

iptables -L INPUT --line-numbers
iptables -D INPUT

Handy Operations Tips:

Tips 1: If your server is behind LB, know if it is stateless or Keep-Alive. if it is a Keep-Alive one, you need to immediately Shutdown the server from LB; this will help stop further traffic to the server. If it is a stateless server , of course you are free to take this guy down and further do the fixing work. And please be quick to do that.

Tips 2: if badly affected, you might also want to increase the MaxClients Limit, to sustain. But its is NOT possible On the fly, since changing of MPM settings needs a apache restart. You need plan this put as a quick crisis Mgt.


I will write more on MPM settings for better performance in coming days. , and steps for setting up mod_evasive and mod_ipconnlimit soon..

PS: I would be more than happy to get the opportunity to discuss your challenges, it will help us to understand various other aspects too.


Thanks & Cheers!
DEBU