I recently encountered a bizarre problem with Mail.app, the email client included in Mac OS X, that took me quite a while to solve (with no help from AppleCare; more on that in a separate blog posting). I’m posting about the problem and its solution here so that, with any luck, the next person who stumbles upon it will be able to find this solution with a Web search.
The iMac in our house has several users on it. Each of them has Mail.app configured to talk to the same servers for incoming and outgoing mail, using essentially the same settings (aside from different usernames and passwords). One day, after Mail.app crashed several times in a row for one of the users, it suddenly stopped being able to send mail. Any attempt to send a message yielded this:
(For search engines, here’s the text in the error pop-up:
Cannot send message using the server server
The server “server” refused to allow a connection on the default ports.
Select a different outgoing mail server from the list below or click Try Later to leave the message in your Outbox until it can be sent.
Sending from: Jonathan Kamens <firstname.lastname@example.org>
[Edit SMTP Server List]
[?] [Edit Message] [Try Later] [Try With Selected Server])
I knew there was nothing wrong with the server. I run it myself, so I was able to check that the SMTP daemon was running and accepting connections. Furthermore, I was able to confirm that right before the error above occurred, Mail.app was connecting to the SMTP daemon. In addition, as noted above, sending mail was working for this user right before Mail.app crashed several times, and furthermore, sending mail was working for all the other users on the computer.
To debug this further, I clicked on the [Connection Doctor] button, told the Connection Doctor to display the details of its tests, and then told it to run the test again. In the resulting transcript, I could clearly see Mail.app connecting to the SMTP server, so at first glance it was not clear why it was claiming that it couldn’t. However, upon further examination, it was clear that something was wrong.
When an SMTP client connects to an SMTP server, the first thing it does is send the command “EHLO client.host.name“. The “EHLO” command indicates a request to start an Enhanced SMTP session (“Enhanced SMTP HeLlO“), and “client.host.name” is supposed to be the fully qualified host name of the client. If the server rejects the EHLO command, then the client sends “HELO client.host.name” instead. The “HELO” command represents a request to start a regular (i.e., not enhanced) SMTP session. Both commands are necessary because some SMTP servers don’t support Enhanced SMTP.
I could clearly see attempts by the client to send both EHLO and HELO commands to the SMTP server. I could also clearly see that for some reason the server was rejecting both commands. Closer examination revealed why. For some reason, the client.host.name sent by the client to the server had two spurious question marks in it before and after the first period in the host name. That is, instead of sending client.host.name, it was sending client?.?host.name. The server was correctly rejecting the commands from the client, because question marks are not valid characters in host names. Bizarre!
I suspected that something in Mail.app’s settings must have gotten corrupted when it crashed several times in a row, so I tried shutting down the app, Library/Preferences/com.apple.mail.plist, and restarting. It didn’t help. I tried removing all of Library/Caches. That didn’t help either. In desperation I tried removing all of Library/Preferences, and even that didn’t solve the problem (note for the latter two attempts, I had to log out the user and remove the files as root).
Finally, in what turned out to be a moment of inspiration, I shut down the app and removed several hidden files and directories from the user’s home directory:
rm -rf .CFUserTextEncoding .DS_Store .bash_history .cups .ssh
Lo and behold, when I restarted Mail.app after removing these files, sending mail was working again.
It’s hard to say for certain which of these files was the culprit, but I’m pretty sure it was either .CFUserTextEncoding or .DS_Store.
I don’t know if anyone else is ever again going to encounter this problem, but if so, then now you know how to solve it. If you do encounter it and this solution works for you, please leave a comment and let me know!