We’ve Been Spam Bombed

25th September 2008

If you’re one of our 900 hosting customers having sporadic trouble with email for the past 36 hours, here’s the deal, and we’re pretty sure it’s totally unrelated to my public criticism of Linux Magic.

At approximately 7:45AM Pacific, one of the PDXTC shared mail servers (mail2.pdxtc.com) was determined to be in a critical state by our server monitoring system.

The PDXTC Tech Support Manager arrived at the Portland Pittock Data Center at approximately 8:45am to troubleshoot the problem.

It was determined that this problem is due to an overwhelming amount of SPAM being sent into the server from outside services.

PDXTC currently subscribes to a service by SPF that eliminates approximately 95% of all SPAM. That service from the third party went offline and let that 95% of SPAM through to the server.

That overwhelming amount of SPAM equates to approximately 1.2 million messages. The servers load when trying to deliver these messages to user inboxes was compromised and will continue to remain in a critical state until SPF can be brought online.

We have already contacted the third party vendors in regards to the SPAM problem and they are aware of the issue and are still working on a resolution. In the meantime we are currently implementing two of our own internal SPAM filter servers to take load off of the server.

The servers have been built and have been installed in our data center. We are now currently waiting on the SPAM filtering software licenses to be issued for this matter to be resolved.

The mail server has remained online through this interim and users should be able to successfully receive email. There is frequent problems with trying to send email from this server due to the high load balance of the server and the SMTP protocol timing out when user’s attempt to send email.

Another symptom user’s may also experience due to this outage is receiving duplicate email messages in their email inbox. This is caused by the load balance of the server as well.

Anytime an email is received from an outside provider the server sends a confirmation that the email has been received to the outside provider. Once the confirmation is received the outside provider stops sending the message, unfortunately with the load balance of the server confirmation notifications to outside providers are delayed causing messages to be delivered multiple times.

Please be aware the PDXTC staff is working as quickly as possible to resolve the problem and minimize any downtime by getting the server back online as soon as possible.

PDXTC understands the importance of this matter and sincerely apologizes for any inconvenience this outage may have caused. Thank you for your continued support of PDXTC.

Other stuff I've written that you might like:

1. Stop all your Spam
2. Spam reported from our mail server
3. Blocked AOL email? Here’s why…
4. How do you Stop Spam, Viruses and Phishing?
5. Portland Web Hosting Problems