Our Web Server - BlacklistedIn mid-April our primary Oakley Studio web server, hosting about 35 client websites, was blacklisted for email spamming. This blog post explains what that means, the effect it had on email delivery for some of these clients, what the potential risks are, and what measures we took to resolve this serious issue.

What Does a Web Server Have to Do with Email?

A web server, by definition, is a computer that hosts websites and serves pages to everyone visiting those websites. You wouldn’t necessarily think a web server would also be sending out email, right? So how could it be blacklisted as an email spammer? Sending and receiving email is the job of a dedicated email server. In actuality, most web servers have a basic capability of sending out email messages, and it’s a useful feature.

Diagram of a BLACKLISTED web server - listed/blocked, not listed/delivered, bypassing the blacklist, Gmail spam filters

Take for example an ecommerce website: when a customer places an order for products or services, the web server is responsible for sending…

  1. An email to the site owner, notifying them that a new order has come in.
  2. An email to the customer, confirming their order was received, perhaps with a summary of their purchase.

Those emails are sent directly to site owner and to site customer from the web server. Even non-ecommerce websites may have reasons to send email, for example…

  • A “Contact Us” page which allows anyone to send a comment or question to the site owner. Those messages are delivered via email sent directly from the web server.
  • A “Subscribe” page which allows visitors to “opt-in” to an email newsletter. Subscribers will typically receive a confirmation email with a link that needs to be clicked to complete the signup process. (This is to confirm subscribers actually provided their own email address, not somebody elses.)

Particularly with WordPress sites, there are a variety of triggers that will cause the website to send an email:

  • A membership site sends emails to site owner and new member when someone joins.
  • A discussion forum site sends an email to site owner whenever a new discussion topic is created.
  • A security plugin sends messages to site owner notifying there are plugins or themes that need to be updated.
  • An affiliate site (like Oakley Studio) sends a notification to site owner when someone signs up for service, and sends a “Welcome” email to the new client too.

Notice that in most of these examples, the recipient of the email is the site owner.

How Is Any of This Related to Spam?

It turns out that a “Contact Us” form, if not properly secured, is a perfectly good way for spammers to send spam/junk email to website owners. Of course no one is actually visiting these form pages and typing in a spam email. Instead, they use automated programs that find these pages and submit prepared content — over and over and over again. And when my clients receive these emails, they may mark them as “spam” and maybe next time such messages go directly to the “Spam” folder instead of to their Inbox.

If it’s spam, I’m sure none of our site owners care where it came from, they just want to get it out of their Inbox and hopefully not have to see any more of it.

What Is a Blacklist?

In the constantly evolving effort to combat spam, it turns out that one of the first lines of defense that many email service providers resort to is blacklists — services that list “bad servers” that are known sources of spam emails. If an email service provider sees that a message originates from one of these bad servers, they simply block it. They don’t allow it to be received by their mail server for delivery to a recipient. It’s an effective way to fight spam, but it is a pretty blunt toolany legitimate email being sent from the server may also end up being blocked from delivery.

Which is how I came to discover that our web server had been blacklisted. When a couple of our clients reported that emails they were expecting to receive never arrived, my first task in troubleshooting was to review the email logs on the server. The email logs record who (email address) is sending and who the recipient (email address) is; who our web server is communicating with (outside email servers) and whether the email was sent or not. I discovered that messages which are refused delivery get additional info in the log, explaining why the email was refused. These entries read something like: “This email was blocked from delivery. See Spamhaus.org for details.”

Spamhaus, as it turns out, is one of the bigger email blacklisting services, used by many different email service providers. (Interstingly, Gmail — probably the biggest email service provider on the planet — does NOT use blacklists. They have their own very intelligent spam filtering and don’t need to employ such a blunt tool as blacklists.) Our own partner email service provider uses the Spamhaus blacklists to fight spam.

Blacklisting effetively means that websites on our listed web server are not permitted to send email until the source of the spam emails has been determined and mitigated.

Getting Off the Blacklist

I spent the next several days working to resolve this serious issue. I determined that spam email was being sent from a few sites that had unsecured “Contact Us” forms. Those forms have now been secured with Google Recaptcha, so that only real humans can submit messages, manually, one at a time, for email delivery; no more automated “spam bots.” Not all of our client websites were being misused in this way, but obviously all sites with unsecured forms would need to have a “captcha” checkbox installed. I am still in the process of doing that for a few remaining sites.

Google Recaptcha checkbox to prove you are a human being.Subscriber signup forms also need to be secured. Most Oakley Studio clients are compiling email lists as a way to facilitate communication with their visitors. You wouldn’t think a signup form that asks only for name and email address, with no text field for any other message content, could be used to send spam. But if such a form is being abused by an automated process, it can send signup confirmations to lots of people who never actually chose to opt-in. This abuse of signup forms is justifiably considered spam, and can be largely prevented using a captcha.

Another issue with outbound emails sent directly from a web server is the Sender email address. It must match the domain name of a site on the server. Several of our clients were using “@gmail.com” addresses for outbound email, and technically these would also be considered spam. Why? Because only actual Google Gmail servers are supposed to be sending email@gmail.com messages. Any such emails sent from a non-Gmail server could be construed as spam.

All outgoing email from our web server should be sent from a domain name email address of an existing website on the server. To address this issue, I created new domain name email addresses for several clients who were using a Gmail address as their WordPress admin email.

Bypassing the Blacklisting

There was one additional thing I could do to solve the email delivery problem. I would employ a reputable plugin — “WP Mail SMTP” — that could be configured to send outbound email through our own email service provider’s mail server instead of sending directly from our web server. This enforces the requirement to use a Sender domain name email address, because our email service provider only accepts for delivery messages from accounts I set up on their mail server. This effectively bypasses the blacklisting of our web server.

Was the Server Compromised?

My biggest concern during this incident was whether the web server had been compromised in some way. Had a spammer found a way to gain access to the server and install a spambot program? If so, it would not be using the built-in email sending functions of the server — and none of those outgoing emails would be logged! Detecting a surreptitiously installed bot program requires a different approach, but is not especially difficult to do.

(Warning: Geeking out for a few seconds here.) So in addition to installing captchas, and routing emails through our designated SMTP Email Server (bypassing blacklisting altogether), and creating new domain name email addresses for Sender in all outbound messages, …  I’ve been monitoring at the server level with netstat and tcpdump to see if there is any whisper of traffic on outgoing port 25, suggesting a compromised server. I have not seen — anything at all. So it does not appear the server itself has been in any way compromised, and there is no evidence of unauthorized access to any sites on the server.

Throughout this episide, I have had excellent communication with Spamhaus, and they have provided some specific and useful advice on methods to check for an incursion. Unfortunately they were not able to provide any actual email with all the raw headers intact, which could have possibly shed more light on how our websites were being misused. They say they get requests for actual emails all the time but beyond that, they were pretty careful — you might say “cagey” — about what they chose to tell me about their listing processes.

“We get our listing intelligence from a variety of sources, and most of them don’t provide samples at all, just the diagnostics we require to create listings. Our own infrastructure does retain samples, but unfortunately nothing from your IP is actually hitting us. Which is, for reasons I won’t go into, suggestive that this more likely is a simple misconfiguration rather than something actively malicious.”

We have gone about ten days with no more spam messages detected by the blacklists, and every day improves our web server’s reputation.

Things never stay the same, and blacklisting is the new big hammer in this game of Whack-A-Mole that is the fight against spam. As a reputable website hosting firm, I need to comply with changes in the industry to ensure that the spammers can’t send their junk through our servers — whoever it goes to. Every email sent this way is potentially damaging to our domains and our brands online. Sending no email directly from Oakley Studio web servers is a goal we’re going to accomplish.