Too Many ESTABLISHED connection from a single IP address in Apache

Portal Home > Knowledgebase > Articles Database > Too Many ESTABLISHED connection from a single IP address in Apache

Posted by ananthan-nair, 05-15-2012, 08:41 AM
netstat -ntp |grep 80 shows too many ESTABLISHED connection from single IP address. Around 300 of them and it is not an attack and user is using a 2G connection to access our website using our mobile application. This is the case with other 2G connections also. As a result of this Apache is running out of children. Earlier it was showing too many close_wait and after enabling tcp_tw_reuse and tcp_tw-recycle and tcp-fin_timeout to 30 there is not much close_wait but the number of ESTABLISHED connections increased. We are using Ubuntu 11.04 having 48 GB ram with 2.6.38-13-generic kernel keepalive On keepalive timeout 10 max clients 1000 max-request-perchild 4000 timeout 300 I have set syn_ack to 1 and syn_retries to 2. On broadband connections there is no such issue. Connections are closing properly, but with 2G connections Apache is running out of children due to too many ESTABLISHED connections. Also i have tried setting timeout from default 300 to 30,but since our project is image hosting for mobile phones,clients couldn't upload images properly as they are getting frequent time out more over it didn't help either.Also there were a lot of 408 messages so changed it to the default 300. we also tried turning off the keepalive but sill this exist.Can anybody tell me why thus much established connection occurs from slow connections. We recreated this issue on another server running on ubuntu 10.4, 2.6.18-274.7.1.el5.028stab095.1 but there even though total children is high but its far less when compared to server running on 11.4.Can any one give me an explanation to this
Posted by UNIXy, 05-15-2012, 09:42 AM
This is an unintentional Slowloris like symptom. Either configure mod_reqtimeout or have Nginx proxy to Apache. Regards Joe / UNIXY
Posted by KMyers, 05-15-2012, 09:44 AM
Hello, Could the user be using a one of the cloud browsers (Amazon SILK, Opera, Skyfire)? These browsers optimize the websites to work over slower connections. The catch is that they proxy everything through their own servers. If you have 50 different users using Opera, they could all appear to come fromt he same IP.
Posted by ananthan-nair, 05-15-2012, 09:57 AM
I personally tried using 2G connection to access our site and still we have that issue.It might not be an attack.
Posted by ananthan-nair, 05-15-2012, 09:59 AM
we have our own mobile application to put images to server,which is behaving weirdly on slow connection,but on high speed no such issues.
Posted by UNIXy, 05-15-2012, 10:01 AM
It's not an attack. It just behaves like one. Perhaps you have way too many assets to serve than 2G can handle. Might want to trim things down. Still approaching this issue from a slowloris angle will help. Regards
Posted by ananthan-nair, 05-15-2012, 10:09 AM
I have too many ESTABLISHED connection in netstat's ouput.Its not getting disconnected even after it reaches apache time out of 5 mins not even changing its status from ESTABLISHED to finwait1 or to finwait2. i am using prefork.will using reqtimeout module alleviate the problem..
Posted by prashant1979, 05-15-2012, 10:40 AM
If there are too many connections from a single IP, you can try enabling mod_evasive Apache module which can help you prevent DDoS if at all happening in Apache.
Posted by gregBS, 05-15-2012, 11:51 AM
try using nginx before apache and then proxy the connections. it will alleviate your load.
Posted by netmultiple, 05-15-2012, 02:47 PM
mod_qos can handle slowris attack very well
Posted by ananthan-nair, 05-16-2012, 12:25 AM
No dear its not an attack...genuine users are accessing it.Only problem is that they are using 2g connection.
Posted by UNIXy, 05-16-2012, 01:38 AM
Frankly, 2G visitors aren't going to pull any faster than they can. You can buy yourself some time with Nginx but sooner than later you're going to hit a hardware and software limitation. The Linux kernel is only able to juggle with so many tasks/threads before it starts thrashing. You're going to have thousands of lingering threads/tasks waiting on network IO, which blocks. I would strongly suggest you start planning for a scale out. In other words, use multiple smaller nodes. Regards

Add to Favourites Print this Article