Like many technical/developer minded people, I like to control my own stuff. It's no surprise then that I run my own web server, complete with hosting web applications, personal projects and of course running my own email services. I have my own personal domains and choose to take responsibility for running everything for them. It's just how I like to do things!
Jump to late yesterday evening (21st May 2018) at around 11PM. Linode (who my VPS server is hosted with) opened an abuse ticket against my Linode server stating that it was sending spam, complete with the full copy of the email with headers that the abuse report related to. Funnily enough the email in question had actually been received from my corporate work account early in the evening, an email from [email protected]. Concerned, I obviously immediately connected via SSH to my server to investigate the report.
Whenever you have a spam complaint there’s generally a protocol you need to follow in order to identify and examine the situation. These are key ones:
Where is the spam coming from?
Has the server been compromised?
Has a specific email account been compromised?
Rate limiting the amount of email traffic during investigation is a good idea to help digest logs
Look for unusually high email volume from single users
Examine the mail queue for any items that look suspect
grep mail logs for suspicious email traffic
After some initial investigation, nothing looked out of place, but that doesn’t mean there isn’t a problem.
Mail logs are your friend for this type of investigating. Linode provided the full email with headers so I can start digesting the logs for clues.
Full email with headers Linode provided in the report:
One of the most important things you always need for email abuse reports is headers. Its a standard protocol for abuse reports to include them. These are are at the top of the gist above and basically show the more nerdy information about the email. The first interesting point was the headers contained no reference to my Linode server at all. While puzzling, headers can be faked, but usually you’d expect to see yourself in the relay list if being accused of sending said spam email, otherwise how can it be tied to you?
No matter, the headers don’t provide anything useful, but more details like the from or sender address are visible so I can now try finding these entries in my mail logs. I use Exim as my Mail Transfer Agent and Exim comes with a tool called exigrep. Basically, grep but tailored for Exim logs, much nicer than having to format plaingrep commands.
As I know the from address, I attempt to locate a matching item in the log. For this I searched the Eximmainlog file, which covers all email events, accepted, rejected etc.
exigrep "[email protected]" /var/log/exim/mainlogNo results. OK strange, how exactly has this email supposedly come from my server if there’s no log of it? Maybe the logs have rotated since? Its possible. I added * (wildcard) to the end of the file name so exigrep looks at all log files available on the file system.
exigrep "[email protected]" /var/log/exim/mainlog*Nope. Still nothing. OK this is odd. Well the envelope ID is there so lets try searching for that, that’s pretty much the absolute reference to tie any email back to the source.
exigrep "0vXAXPW2TmSNt-e-U8Y_tw" /var/log/exim/mainlog*Still nothing. OK now I’m at a loss, because I’ve got Linode saying this email originated from my server, yet the headers don’t mention my server at all and my Exim logs have no record of any email of this kind being processed. Have spammers gained access to the server and cleaned the logs? Unlikely. There was no evidence of any breach through an email account or a user account on the server itself.
Sometimes, you can easily get fixated on an area you think is the problem when its not. This turned out to be one of those situations. If you keep looking past the email headers you will find the original HTML source code that was sent in the email.
Line 206–208, has this code written as a CSS comment. It has likely come from a responsive template, belonging to Litmus and adapted by the community.
/* Remove margin on email wrapper in Android 4.4 KitKat *//* See more at: https://blog.jmwhite.co.uk/2015/09/19/revealing-why-emails-appear-off-centre-in-android-4-4-kitkat/ */div[style*="margin: 16px 0"] {margin:0 !important; font-size:100% !important;}What’s however more interesting, is I actually wrote the blog post the CSS comment features! It relates to a quirk of the Android 4.4 built in email client that had some odd off centre behaviour with email layouts. I documented the issue and proposed a slightly weird attribute selector CSS rule to target a CSS property. Inception right? It did the job though. Someone’s created some boilerplate code and documented each line it seems, they were even kind enough to reference me, but that’s probably what just burned me!
So it turns out Linode’s automated system has flagged this email as spam and tied it to me, because of the domain blog.jmwhite.co.uk being in the email and you guessed it, my domain blog.jmwhite.co.uk and indeed jmwhite.co.uk is hosted on the accused Linode server!
What may have further pushed me into the crossfire here, is my DNS for the domain jmwhite.co.uk is also hosted at Linode and obviously tied to the same account the accused Linode server is on, so it looks like a bad case of mistaken identity and an overzealous automated system getting a bit itchy on the details. Clearly Linode don’t vet their abuse reports that closely as it was clearly obvious that the headers didn’t show my involvement in this, but anyhow, situation clarified, my Linode doesn’t get terminated for spam, everyone’s happy!
Its also interesting to know that spammers at least care about the devices there spam goes to, given they are using boilerplate code to have responsive email campaigns! They didn’t get the memo about HTML/CSS comments being a potentially dangerous thing in HTML email campaigns though, but then I’d not have a story to share about a bizarre case of mistaken identity!