A case study in troubleshooting the email pipeline, chapter 2

By | September 11, 2017

A couple of days ago, I posted a saga about troubleshooting my email server and coming to the conclusion that Constant Contact was sometimes generating emails whose “Date:” headers ended with CR rather than CRLF. I was wrong. Constant Contact’s headers are fine.

Today, I noticed some additional messages in my notspam archives showing a problem similar to the ones from Constant Contact which I noticed before, but the messages I noticed today were not sent by Constant Contact.

I investigated and discovered that these messages had gotten corrupted by a bug in spamitarium: it wasn’t handling empty headers properly. So I fixed that.

After doing that, I had a brainstorm: let’s see if any Constant Contact messages have come into my inbox since I deployed my workaround a few days ago, and if so, were they handled properly. Lo and behold, there were new Constant Contact messages, and they had the concatenated header problem that I thought I’d already fixed. D’oh!

I then flailed around a bunch trying yet again unsuccessfully to reproduce the issue, before I made a fortuitous discovery: when I was working on this issue a couple of days ago, I had modified the spamitarium script called by bogofilter to save a copy of both its input and its output in /tmp, and I had forgotten to turn that off when I (thought I) was done fixing the problem! I went and looked in /tmp and found the files associated new Constant Contact message, and at long last I was able to identify the real root cause of the issue:

Date: Fri, 8 Sep 2017 13:44:33 -0400 (EDT)
From:
  "Really Long Really Long Name" <somewhat-long-address@example.com>

Remember how I mentioned three paragraphs ago that spamitarium wasn’t handling empty headers properly. Well, it interpreted the From: line shown above, which was wrapped because its contents were so long, as an empty header, and therefore it handled it incorrectly.

Why wasn’t I able to reproduce this today or when I was debugging this a couple of days ago? Because here’s what that header looked like when the message arrived in my IMAP folder:

Date: Fri, 8 Sep 2017 13:44:33 -0400 (EDT)
From:   "Really Long Really Long Name" <somewhat-long-address@example.com>

Something between the internet and my mailbox kindly “fixed” the From: header by unwrapping it, so what I was feeding into spamitarium to try to reproduce the issue was not what spamitarium was actually getting when the original message was received by my server. *sigh*

As a result of this, I’ve completely refactored the header-parsing code in spamitarium to avoid issues like this. Frankly, that code needed to be rewritten, and now it has been.

For reading this far, you get a photo of the little field mouse that my son and I saw when biking home from ice cream yesterday:

Share

One thought on “A case study in troubleshooting the email pipeline, chapter 2

  1. Pingback: A case study in troubleshooting the email pipeline – Something better to do

Leave a Reply

Your email address will not be published. Required fields are marked *