Just noticed that incoming emails were being delayed by about 30 minutes, we host our emails on Google Apps, so that couldn’t have been the issue. Next step was to investigate the server where the DNS of our domain was being hosted. Turns out the server was under constant load (load average ~4) for the past couple of days.
Upon investigating what was causing the load I saw multiple threads using the xmlrpc.php file. There were about 20 odd threads consuming between 20% – 90% cpu. Searching online revealed that this was a distributed botnet attack trying to compromise the wordpress installation by exploiting that file.
To fix this I add the following to the global apache configuration via a pre include
<Files xmlrpc.php> Order Deny,Allow Deny from all </Files>
For now the server load has gone down but upon manually hitting the file it turns out that the wordpress installation throws a 404 page. This may still be a problem because the botnets would still be hitting that URL which in turn would still cause high cpu load.
I haven’t tested the following myself but this would be my next logical move. The following command redirects any hit to to the xmlrpc.php file to an invalid IP address thus taking the load off of our server.
RewriteRule ^xmlrpc\.php$ "http\:\/\/0\.0\.0\.0\/" [R=301,L]
One problem with the above solution is that you’ll have to add this to every properties .htaccess file individually, which is going to take time if you have 100+ wordpress installations on your server like I do.