Server sending out spam

Portal Home > Knowledgebase > Articles Database > Server sending out spam

Posted by Adonis, 12-02-2009, 04:55 PM
A few weeks ago, i found out that my server IP got banned by Microsoft for Hotmail/Live. Dealing with their support is enormously difficult. The only way to receive mail headers (to try to find where in the server this problem is), is to enroll in their Junk Mail Reporting Progam (JMRP). And the only way you can get in there is to have the IP address on your own name, not just assigned to your server. I did however get into their SNDS. So there i found that the IP was indeed banned and flagged red for 68 straight days already. And it appears that every day, between 1000 and 2000 messages are delivered to Hotmail/Live, and that almost every day, between 1 and 3 of these hit their spam trap (honey trap). I highly doubt that any of my services send out even close to that amount to Hotmail/Live on a daily basis, so something else must go on. I have been diving into the server for many days now, and keep coming up empty handed (nothing useful in logs). However, for the past 2 months, i have been having almost daily situations where between 17:00 and 21:00 (it varies), a shitload of perl processes are started under the apache user. I can't seem to find out what starts these processes. These processes keeprunning even if apache dies, so psmon cannot restart apache. I implemented a script that kills all apache processes every 10 minutes. I have no idea if these perl processes are linked to the spam problem. There is no noticable increase in load when these run, nothing in the logs. When they run, a strace on these processes gives bak something like this: First question would be, is this possibly related to the spam problem, or a PHP-CGI problem, or something else? And the second one would be, how to find out at all what is sending all this spam to Hotmail/Live? I am willing to hire someone if that will solve it, since i clearly seem to not be able to do it Last edited by Adonis; 12-02-2009 at 04:58 PM.
Posted by partek, 12-02-2009, 05:39 PM
Judging by those commands in the strace, I would bet someone installed some spam software on your server. Since they are running under the apache user, they likely got in through a vulnerability in one of your web applications. When these are running, run the following command: pstree -p > /tmp/pstree.out Look for those perl processes in there and you should be able to figure out what is executing them. Is your documentroot and/or cgi directory writable by the apache user?
Posted by Adonis, 12-02-2009, 06:02 PM
I have not checked all 50 or so domains yet, but from what i see, these are only writable by the domain user. Since these perl processes run under the apache user, i must assume they were invoked by apache. I also wonder hy these always seem to run between 17:00 and 21:00. There are no crontabs listed under the apache user at all. Beyond that, nothing out of the ordinary is scheduled between that time. I run ASL (Atomic Secured Linux), which was installed weeks before this started happening. ASL was installed minutes after the server came live. I'd think that solution would keep most attackers outside. Also, clamscan checks, rkhunter, chkrootkit checks all come up blank. I'll check the pstree command you gave me tomorrow when these things appear again. This is driving me nuts.
Posted by partek, 12-02-2009, 06:29 PM
ASL is really only going to provide operating-system level security. If someone breaks in via a web application it can't really help you. If they're running as the apache user, then they're likely being executed by a PHP or CGI script of some sort. Check the apache logs for any scripts being executed around that time. You may also be able to find the perl script that is being executed by looking for some of the content of what you saw in the strace. You should run that find specifying the path as anywhere a user can write a file to such as the docroot, any cgi directories, and user home directories. Once you find the path to the perl script that is being run, you can then use that to find what php/cgi script is calling it.
Posted by Adonis, 12-02-2009, 08:29 PM
I just finished that, and had it scan the complete Plesk root. Nothing strange there either.
Posted by Adonis, 12-03-2009, 07:33 PM
Okay i finally found out that these perl processes are indeed that spammer: Still no trace of where it is located, who executed it, etc.
Posted by foobic, 12-03-2009, 07:43 PM
You'll probably find that the perl script is being written (or uploaded), executed and deleted in quick succession. /proc/pid/cmdline should tell you where it was, though.
Posted by Adonis, 12-03-2009, 07:53 PM
Ah i did not know that. I'll check that out the next time this spammer starts his thing. Since these processes are already gone.
Posted by trza2k, 12-09-2009, 08:39 AM
I just had a similar issue. In my case the spammer was ftping a php/perl script combo then executing the php via the webserver to start a perl spaming script then delete the files. You should be able to see wierd .pl/php files being uploaded in your ftp xfer log if this is the case. Also i noticed 2 different ip's for the same ftp account. Http logs for the domain should have the same weird .php file being accessed/run. I believe in this case the user account password was hacked from the users PC and then used to run these scripts.
Posted by mikegotroot, 01-02-2010, 02:40 PM
If the processes are owned by apache then you have a vulnerable PHP application, and you need to check to make sure that you are letting ASL disable the dangerous PHP functions on your system that make this possible. Out of the box ASL will only warn you about these and report them as vulnerabilities - but if you tell it to auto-fix these vulnerabilities it will close those holes in your system. For example, if you have ASL configured correctly you should have these functions turned off at a minimum in /etc/php.ini disable_functions = , dl , exec , passthru , pcntl_exec , pfsockopen , popen , posix_kill , posix_mkfifo , posix_setuid , proc_close , proc_open , proc_terminate , shell_exec , system , leak , posix_setpgid , posix_setsid , proc_get_status , proc_nice , show_source , escapeshellcmd , phpinfo Also, ASL has realtime malware detection, but you also have to turn that on. Check to make sure you have that turned on as that specific trace would definition be stopped by that too.
Posted by DigitalLinx, 01-02-2010, 02:54 PM
escapeshellcmd is an input sanitation function http://www.php.net/manual/en/functio...peshellcmd.php it would be like disabling mysql_escape_string What's leak()?
Posted by mikegotroot, 01-02-2010, 06:14 PM
Unfortunately too many people thing that. The two functions are nothing alike nor are the consequences of allowing them. Unlike escaping mysql - which by itself isn't a dangerous thing - shell commands are ALWAYS dangerous. So sanitizing them means you allow them, which is always bad. If you allow shell commands globally, you allow any PHP application or more importantly in this case, any injected PHP code to act as a shell. And thats the exact vector these attacks use - allow shell commands = get owned via the web. They don't even have to log in, allowing shell commands gives an attacker a shell - unauthenticated to your box. Remember, the code is often injected - so you have to ask yourself, what would I want to allow an attacker to run on my box? Allowing commands to be run unauthenticated is the worst possible thing you can do and thats how these worms work. PHP is interpretated, so the purpose of killing these functions is to limit the scope of what PHP can do - so yes, disable all shell functions even the "safe ones". Also, as things go, escapeshellcmd doesn't work very well for sanitizing shell functions (which is an oxymoron by itself). So depending on it is dangerous by itself as it doesn't guarantee the result most users think of (i.e. safe). Also there have been serious weakenesses/vulnerabilities in the function itself that would allow you to bypass other things, like openbasedir and safe_mode. Turning it off doesn't hurt because if you allow it - well you are allowing shell commands anyway. If you deny shell commands, well then you don't need this. Any really, not much out these uses this function (not Joomla, Wordpress, MT, sugarCRM, etc.) So allowing shell commands in any form is a huge huge hole by itself. Shell commands = just post your root password and get it overwith. leak() is a debug function. It tells PHP to leak a specific amount of memory, so its a function that can be used to cause a denial of service attack against the server. Last edited by mikegotroot; 01-02-2010 at 06:24 PM.
Posted by Newwebsite.com, 01-02-2010, 06:20 PM
You can also set up the firewall so that only the real mailserver can send out mail via port 25 and all other process cannot. At least that way all mail that goes out will go through your real mailserver and be logged and then you have a better chance of finding any spammer. It sounds like this spammer is bypassing your real mailserver and so therefore nothing is logged.
Posted by mikegotroot, 01-02-2010, 06:30 PM
You'll have to change the users below (qmail, postfix) to the users that your MTA uses to send mail, but these two rules will restrict outgoing port 25 to trusted users. Also, you'll need to make sure you have spamassassin or something setup to check outgoing mail for spam too. iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner qmail -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner postfix -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner root -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -j REJECT
Posted by Newwebsite.com, 01-02-2010, 06:36 PM
Thanks Mike. I was just trying to find the rules myself. I knew I had done it before but could not remember the exact statements.
Posted by mikegotroot, 01-02-2010, 07:13 PM
You may want to also block port 465 outbound too (SSL SMTP): iptables -I OUTPUT -p tcp --dport 465 -m owner --uid-owner qmail -j ACCEPT iptables -I OUTPUT -p tcp --dport 465 -m owner --uid-owner postfix -j ACCEPT iptables -I OUTPUT -p tcp --dport 465 -m owner --uid-owner root -j ACCEPT iptables -I OUTPUT -p tcp --dport 465 -j REJECT
Posted by DigitalLinx, 01-02-2010, 07:54 PM
Disabling shell commands is a safe way to go, agreed. However it is also limiting the service. Lets take a look at one of most popular content management systems mediawiki As you can see shell functions are being in order for mediawiki and some of its features to function. If you have a tight system with careful directory permissions and not allowing php code to be executed at directories where php code isn't expected such as for example user uploaded images/avatars and have tight permissions on system binaries along with open_basedir, you'll have a functioning and secure system. Those iptables commands will block *ALL* outbound e-mail including legit e-mail going out from legit client accounts, it will also cause the queue to be filled because the emails wont be delivered to the remote hosts. Care to post a reference to this leak() function I have never heard of, nor did the php interpreter please?
Posted by mikegotroot, 01-02-2010, 08:28 PM
Yes, good old wide open mediawiki. And in that case just use suhosin and only allow those functions for that vhost or go ahead and allow those functions for a more trusted domain. I agree that a properly locked down box with good permissions can prevent this, but for a hoster thats just not gonna happen if their customers don't practice good security themselves. And heres the kicker - the specific attack vector appears to be taking advantage of vulnerability in APR < 1.3.6 that allows a child thread to hijacvk port 80 as a global listener thread - hijacking more than just the domain thats been compromised but the entire server. So if you can inject code on the box, or upload it you can take advantage of this bug in APR. \ As to the rules, yes a small tweak is in order, the final rule needs to be an append not an insert: iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner mailuser -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner root -j ACCEPT iptables -A OUTPUT -p tcp --dport 25 -j REJECT And if you dont have a global rule for loopback, then add this: iptables -I OUTPUT -p tcp --dport 25 -d 127.0.0.1 -j ACCEPT [root@shrike2 ~]# su - mailuser [mailuser@shrike2 ~]$ telnet ac3.progllc.com 25 Trying 82.165.156.225... Connected to ac3.progllc.com. Escape character is '^]'. 220 rblsmtpd.local quit 221 rblsmtpd.local Connection closed by foreign host. [mailuser@shrike2 ~]$ exit logout [root@shrike2 ~]# su - user [user@shrike2 ~]$ telnet ac3.progllc.com 25 Trying 82.165.156.225... telnet: connect to address 82.165.156.225: Connection refused As to leak() heres a link: http://www.phpdig.net/ref/rn39re747.html Last edited by mikegotroot; 01-02-2010 at 08:41 PM.
Posted by DigitalLinx, 01-03-2010, 02:01 PM
Well if we're talking about such vulnerabilities, the attacker may as well exploit one of many php memory leaks/corruption bugs to override php.ini settings, not to mention bypass php completely and use a CGI script to execute commands if enabled, or perhaps SSI. But these are semi-advanced techniques and the average script kiddie which attacks random websites probably isn't aware of them. I see that leak() was deprecated and then removed from php 5.x. Nonetheless we're getting off topic, in the end daily system checks are a must if you want to have a secure system, croning chkrootkit or other 'security programs' are just not enough, since the source is available to everyone and the checks they do are trivial to bypass not to mention they do not check for everything.
Posted by Maynard, 01-03-2010, 03:23 PM
Great discussion, folks. Thanks. In addition to earlier suggestions for controlling SMTP output with iptables, I suggest jumping to a chain which logs before rejecting the illicit packets. That way you don't hide the problem from yourself
Posted by mikegotroot, 01-03-2010, 03:50 PM
Here you go: iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner qmail -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner postfix -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -m owner --uid-owner root -j ACCEPT iptables -I OUTPUT -p tcp --dport 25 -d 127.0.0.1 -j ACCEPT iptables -A OUTPUT -p tcp --dport 25 -j LOG_SMTP_OUT iptables -A LOG_SMTP_OUT -m limit --limit 1/second -j LOG --log-level info --log-prefix "Unauth-SMTP " --log-tcp-sequence --log-tcp-options --log-ip-options iptables -A LOG_SMTP_OUT -j REJECT
Posted by mikegotroot, 01-03-2010, 03:55 PM
Which ASL protects against in many ways. The script kiddies aren't the ones launching the thread hijacking attacks, so these capabilities are within the scope of this forum thread. The hijacked port 80 symptom is not a script kiddie attack, and its a clever attack on a bug in APR. You'd be suprised at home many companies still use PHP 4, so its still a good idea to tell people to disable this function. And disabling it in 5 is harmless PHP will just ignore it. Daily checks are good, but not enough. You have to harden your system based on an assumption that things will fail and parts of the system will become compromised. Defense in depth is the answer.
Posted by DigitalLinx, 01-03-2010, 04:25 PM
Define ASL? Is that some kind of php module/patch like suhosin? Not sure how exactly it would be possible to protect against memory corruption if an internal php function is vulnerable. Care to elaborate please?
Posted by mikegotroot, 01-03-2010, 04:51 PM
ASL is short for Atomic Secured Linux. Specifically, lots of different elements of ASL protect against these attacks, from the self learning RBAC, secure kernel, realtime malware protection, upload protection, self-healing vulnerability scanner, etc. You just have to turn those features on. Not to mention we publish a hardened PHP in our atomicrocketturtle.com lab that helps to prevent these things if you don't use ASL, but thats more about hardening PHP itself and backporting various security features for people that can't upgrade to the latest and greatest (and if they can upgrade to the latest and greatest including new methods to protect PHP users).

Add to Favourites Print this Article