Until recently I was backing up my data into a ReiserFS filesystem being stored in the Amazon S3 cloud via s3backer. That was costing me on average a little under $5 per month in storage and transaction costs.

Then I got an offer from AppSumo for 10GB of storage for life from LetsCrate for only $25. That got me wondering… There are a bunch of cloud storage / file sharing services on the Internet right now, and just about every one of them offers some amount of storage for free. Could I find away to take advantage of all that free storage to reduce my backup costs almost to nothing?

What’s out there

Here are the cloud storage services that offer free Linux-accessible storage that I know about (if you know of others, please post a comment or email me!):

That’s 50GB of free space in the cloud (43GB if you don’t have a Windows VM), if you can figure out how to use it effectively. So, how do you use it effectively?

The backup script: crateify.pl

My answer is crateify.pl, a simple Perl script I wrote for this purpose.

Without further ado, here is its embedded Perl “POD” documentation:

NAME

crateify.pl – Package up files for backing up in the cloud

DESCRIPTION

This script packages files within a directory tree into compressed, encrypted tar “crates” that can be easily uploaded to free cloud storage accounts, providing a sort of poor-man’s cloud backup solution.

The files are packaged in chronological order, i.e., oldest files first, to minimize the frequency with which you have to rebuild crates. Files that are updated between runs of the script are repackaged in new crates.

CONFIGURATION SETTINGS

The following variables can and should be edited in the script before you use it:

$backup_dir

The directory whose contents should be crated.

Note that files with newlines in their names will not be crated.

$data_dir

The directory in which crates and associated metadata files should be stored.

$gpg_dir

The directory in which the keyring containing your GPG key (used to encrypt the crates for safe storage online) is stored.

$gpg_key

The identifier of the GPG key that should be used to encrypt the crates.

NOTE: Make sure you have copies of your public and private GPG keys backed up somewhere safe not inside a crate. If your computer crashes and you need to restore from your backup, it won’t do any good if you can’t decrypt it!

$archive_size

The (pre-compression, pre-encryption) size of each crate, in bytes. A crates can end up being much bigger than this if the last file inserted into it is large.

@exclude

Regular expressions (relative to the root of $backup_dir) of directories and files to be excluded from crating.

Here’s a trick I use to find out what’s taking up space in my crates:

 cd $backup_dir
 sed -e 's/ [0-9]*$//' $data_dir/crate-##### | xargs -d '\n' ls -lSr

This lists the files in the specific crate, in size order, so you an see what’s taking up a lot of space. I do this whenever my nightly backup report email tells me that a larger than expected crate was built.

Note that I personally do not back up my “live” hard drive, but rather a mirror hard drive maintained with rdiff-backup. Therefore, most of the files I would not want/need to crate are already excluded from my $backup_dir, which is why my @exclude list is so short.

If you do backup of your live hard drive, then make sure you exclude cloud storage directories, e.g., ~/Dropbox, especially if you store crates in them! Otherwise, you’ll create a loop where each time you create new crates in a backup, your old crates will be included in them, which would obviously be Very Bad.

If you specify both @exclude and @include, then @include is applied first and @exclude is applied to what’s left.

@include

Regular expressions (relative to the root of $backup_dir) of directories and files to be included from crating.

If you specify both @exclude and @include, then @include is applied first and @exclude is applied to what’s left.

OPTIONS

–crates=#

Produce (at most) te specified number of crates, rather than just one new crate, which is the default.

This is faster when you want to produce multiple crates, since it won’t have to rescan the entire backup directory for each one.

–full

Create enough crates to hold everything that currently needs to be crated.

–scan

Update meta-data files (see below) without building any new crates.

–quiet

Don’t print warnings about updated or deleted files in existing crates.

COMPACTING CRATES

The early crates you build will probably be relatively static, assuming that you have a lot of old data that isn’t likely to change anymore.

However, over time your crates will accumulate files that are obsolete because they’ve been deleted or updated versions have been packed into newer crates. Each time you run it, the script prints warnings about such files.

You will probably want to occasionally “compact” your crates to remove such obsolete files. To do this, simply remove the corresponding crate-##### files from $data_dir, and the corresponding compressed, encrypted tar files from wherever you put them, and the script will repack the files that were in those crates the next time you run it.

META-DATA FILES

The script creates the following meta-data files:

crate-#####

Listings of the files in each crate. The script needs these to work, so you should leave them in $data_dir even if you move the crates themselves into the cloud.

deleted

A list of the crated files that have been deleted since they were crated.

updated

A list of the crated files that have been updated since they were crated, i.e., files that have obsolete versions in one or more crates, and will also, if your crates are up-to-date, have a current version in one crate.

packing_list

Temporary file created and used while packing crates. It should not exist between successful runs of the script, but you shouldn’t create a file with this name in $data_dir or it’ll get overwritten.

excludes

A list of all te files in all of the crates, intended to be used to exclude those files from some other backup system.

Suppose you want to use this script to back up your old, static files that never change, but you’d rather use some other backup system to back up frequently changing files. To do that, you would tell the other backup system to exclude the files listed in $data_dir/excludes.

For example, if you use rsync to backup frequently changing files to a remote filesystem, then you can tell it to “–exclude-from $data_dir/excludes”.

WHERE TO PUT THE CRATES

The crates you build with this script obviously don’t do much good as a backup if they sit on the same drive as the files being backed up. Here are some examples of what you can do with them to turn them into a real backup.

• Stick an extra hard drive (internal or external) into your system and put your crates on it. This won’t do you much good if your house burns down or somebody steals your computer, but it’ll at least protect you against drive failure.
• Make a deal with a friend — he lets you use unused space on his hard disk to scp your crates to every night when you back up, and vice versa.
• Free cloud storage! See http://blog.kamens.us/ for a list of cloud storage platform which will give you a total of 50GB of free storage just for asking. You can store a lot of crates in 50GB!
• Upload them to Amazon S3 or some other commercial cloud storage service.

Personally, I have uploaded most of my crates, the ones containing older files that change rarely if ever, by hand to free accounts on SkyDrive and LetScrate. Then, my nightly backup puts new crates in my Dropbox folder, so they get synchronized to the cloud automatically. Occasionally, I compact the Dropbox crates as described above and move some of the compacted crates SkyDrive or LetsCrate as needed.

What if a crate is too big?

You probably have some really huge files (home videos, anyone?) that you want to back up. Since this script doesn’t split files between crates, any crate containing a really huge file is going to be really huge itself.

Depending on where you store your crates, this may present a problem, since some cloud storage services limit the size of uploaded files.

The easiest solution is to split big crates before uploading it. For example:

  split -b 50000000 -d crate-#####.tar.bz2.gnupg crate-#####.tar.bz2.gnupg. && \
  rm crate-#####.tar.bz2.gnupg

The name of the crate is specified to the “split” command a second time with a period at the end of it as the file-name prefix for the split files that are produced.

If you ever need to restore from a split crate, you can cat all of the split files directly into gpg, something like this:

  cat crate-#####.tar.bz2.gnupg.* | gpg | tar xj

DOING A RESTORE

If you can’t figure out on your own how to restore from the crates produced by this script, then you probably shouldn’t use it. CrashPlan is a pretty nice service, and it’s very inexpensive.

Having said that…

To restore from a set of crates, you decrypt and untar all the crates in order (preferably as root, so that read-only, updated files can be overwritten) and then remove the ones listed in the “deleted” file.

Alternatively, if you just need to restore a specific file, you can look through the crate-##### files in reverse order to find the file you want, and then extract it from the corresponding crate.

WHAT THIS SCRIPT ISN’T

This script isn’t really intended to preserve historical versions of files or to allow you to recover files that were deleted long ago. It sort of does that if you never compact your crates, but that’ll eat up a lot of extra storage space for files that change regularly.

Therefore, if you want access to a historical record of your files, as opposed to an emergency recovery snapshot of what you’ve got on disk right now, this probably isn’t the right tool for you.

AUTHOR

This script was written and is maintained by Jonathan Kamens <jik@kamens.us>.

Please let me know if you have questions, comments or suggestions!

OTHER FREE BACKUP SOLUTIONS

I won’t lie to you… It takes work to set up and use this script for backups. If you’re the kind of do-it-yourselfer who likes stuff like this, great, but if not, you might be asking yourself, “Are there other options for backing up my Linux box for free?”

There are probably quite a few of them, but if you have one that’s you’re favorite please free to email me email me and I’ll add it here, but here’s the one I like…

CrashPlan

CrashPlan (http://crashplan.com/), which I’ve mentioned elsewhere in this document, will let you back up an unlimited amount of data to their servers for $3.00 per month. This is neat, but they’ll also let use their easy-to-use software for free to back up your data to your own server instead of theirs.

“How is that free?” you’re asking? Well, if you can find a friend with an Internet connection (who doesn’t?) and some extra hard drive space (hard drives are really cheap nowadays!), you can back up your system on his hard drive, and vice versa. Both of you need to install the CrashPlan software on your systems and open up your firewalls to allow access to it, and that’s it. You can configure CrashPlan to limit the amount of bandwidth it uses so it won’t max out your Internet connection (in fact, it comes configured that way by default). The one caveat is that if you ever do need to do a restore, it’ll probably take longer from your friend’s computer than it would from something in the cloud, since most home Internet connections have a slower uplink speed than downlink.

DONATIONS

This script is and always will be free for you to use or modify as you see fit. Having said that, it took me time to write the script, and it takes me time to support the people using it. So if you do use it and save yourself some money, please consider showing your appreciation by sending me a donation at http://blog.kamens.us/support-my-blog/. Any donation, large or small, is appreciated!

COPYRIGHT

Copyright (c) 2011 Jonathan Kamens.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

VERSION

$Id: crateify.pl,v 1.35 2012/01/04 13:07:30 jik Exp $

The current version of this script should always be available from http://stuff.mit.edu/~jik/software/crateify.pl.txt.

I followed the instructions on the Fedora Web site for upgrading using yum, and 2,978 downloaded packages and several hours later, I had a Fedora 15 system. But the fun was only beginning. Here, for the benefit of others who follow me, are the problems I encountered after the upgrade and what I did to fix them. Much of this I learned from other helpful people on the web, but some of it I had to figure out for myself.

• First and foremost, my network performance went totally to hell immediately after the upgrade. With the F14 kernel, I was getting about 7 megabit/s on my cable-modem connection, while after the upgrade, I was getting around 500 kilobit/s, i.e., 1/14th of the pre-upgrade speed. It appears to be a problem with kernel support for my network adapter, a Realtek RTL8168c/8111c NIC that uses the r8169 kernel driver. I filed a bug about it, and then went out today and bought a Netgear GA311 PCI network adapter (RTL8110s using the same r8169 kernel driver), which so far appears to be working just fine.
  Note that before I replaced the network adapter, I was able to improve network performance a bit by reducing the MTU on the ethernet device to 1000 (“sudo ifconfig eth0 mtu 1000″).
• Grub no longer lets me hit ESC or any other key while it’s doing its boot countdown to bring up a menu of kernels to choose from. If I want to boot a different kernel, I have to edit /boot/grub/grub.conf and reboot. I still haven’t figured out how to fix this, but I did file a bug about it.
• The files in my desktop folder weren’t showing up on my desktop. To address this I had to run gnome-tweak-tool (you may need to “yum install gnome-tweak-tool” if you don’t have it) and enable “Have file manager handle the desktop” in the “File Manager” settings panel. While I was at it, I made some other settings changes in gnome-tweak-tool:
• I reduced the default fonts from 10pt to 11pt, since things got a little bigger after the upgrade and I wanted to fit more text on the screen.
• I enabled “Show date in clock” in the “Shell” panel.
• I changed “Arrangement of buttons on the titlebar on the “Shell” panel to “All” because I wanted maximize and minimize buttons in addition to the default close button.
• I wanted the Delete key to delete files, but alas, the default Nautilus binding for deleting files changed from Delete to Ctrl-Delete in F15. I found these instructions for fixing this, but I had to tweak them slightly. In particular:
• Run gconf-editor, not dconf-editor as the other web page suggests.
• Find org > gnome > desktop > interface > can-change-accels and check the checkbox next to it.
• Open a Nautilus window, e.g., double-click the “home” icon on your desktop. Note that it took me a while to figure out that you must do this in a window, i.e., you can’t do it by right-clicking on a file on the desktop. Yes, this is counter-intuitive.
• Select a file in the Nautilus window.
• Open on the File menu and move your mouse down to “Move to Trash”.
• Hit the Delete key to remove the old Ctrl-Delete key binding. Hit it again to create a new key binding, just for Delete.
• Uncheck the check-change-accels checkbox and close gconf-editor.
• For some reason I’m not sure of, I had nfs-utils and ypbind, neither of which I need or use, installed, and after the upgrade, systemd was trying to start ypbind and idmapd from nfs-utils. I removed both packages with “yum remove”.
• For some bizarre reason, a bunch of my contacts in Pidgin moved from the groups they were in before to “Orphans” at some point during the upgrade. I had to move them all back into their correct groups one by one. Ugh.
• No weather applet! Must have weather applet! A little googling led me to:
• run “git clone https://github.com/simon04/gnome-shell-extension-weather.git”;
• follow the instructions in README.md for building and installing the extension;
• run “python weather-extension-configurator.py” to configure the extension settings described in README.md, which is easier than using the gsettings commands there; and
• type Alt-F2, “r”, Enter to restart the GNOME shell and activate the extension. Thanks very much to the authors of this wonderful extension!
• I had a launcher file (i.e., a “*.desktop” file) created in GNOME 2 that I wanted to add to my GNOME 3 favorites. Unfortunately, it appears that to add a launcher to your favorites it has to be in /usr/share/applications (well, actually, I probably could have put it in ~/.local/share/applications, but I didn’t think of that at the time). So I copied it into /usr/share/applications, launched it by moving my cursor to the “hot spot” in the upper left corner of the screen and typing the name of the launcher; and then when it was running, right-clicked on it in the favorites bar and selected “Add to Favorites”.
• I wanted to be able to easily clear my desktop, like the “Show Desktop” button I had in my GNOME 2 panel. I found these instructions for fixing this.
• The MySQL daemon wasn’t starting up properly after the upgrade. Apparently one of the ib_logfile* files in /var/lib/mysql was truncated or something. I did “sudo rm /var/lib/mysql/ib_logfile*” and then “systemctl start mysqld.service” worked just fine.
• The ddclient program which I use to update my dynamic DNS record with OpenDNS.com and DynDNS.org was generating an error on shutdown because it was failing to create its PID file properly on startup. I fixed its init script to address this and submitted a patch here.
• I desperately wanted an old-style dock visible all the time with my running applications in it. Fortunately I’m not the only one… I did “sudo yum install gnome-shell-extensions-dock” and then logged out and logged back in, and presto, I’ve got a dock.
• In F14, I used the mail-notification program to generate notifications about new email. It stopped working after the upgrade so I thought it wasn’t supported in F15. Turns out I just had to weak it a bit.
• launch gnome-session-properties;
• select “Mail Notification” and click “Edit”; and
• remove “–sm-disable” from the command field.
• The cron daemon wasn’t starting properly after the upgrade. I’m not sure why, but I fixed it by doing “sudo systemctl enable crond.service” and “sudo systemctl start crond.service”.
• I had an old-style init script in /etc/init/foo.conf that looked like this:
  start on runlevel 5
  stop on runlevel [!5]
  exec /home/jik/scripts/foo.pl
  respawn
  I had to replace it with /lib/systemd/system/foo.service which looks like this:

  [Unit]
  Description=Foo server
  After=network.target
  
  [Service]
  ExecStart=/home/jik/scripts/foo.pl
  Restart=always
  
  [Install]
  WantedBy=multi-user.target

  Then I did “sudo systemctl enable foo.service” and “sudo systemctl start foo.service”.

• In F14, you could rely on the GNOME session launching script to execute the contents of your .bash_profile file, so that environment variables you set in .bash_profile would be inherited by all child processes. This doesn’t work in F15, so you have to set environment variables in .bashrc.

I’ll continue to update this list as new things come up.

It took a day or so to get used to, but I think the GNOME 3 UI is growing on me.

Emboldened by that, I decided to take a stab at fixing two bugs in the core Thunderbird code that have been driving me crazy. That, too, required a steep learning curve, but in the end, I was able to submit fixes for two bugs, one quite old and one new in Thunderbird 3.1, affecting a whole bunch of people:

• It was impossible to remove attachments from some MIME messages, including MIME messages generated by the Mac Mail client (Mozilla bug #351224). This bug has been reported by at least 30 different people and was first reported almost four years ago. Fixing it required rewriting pretty much an entire module within C++ source code for Thunderbird.
• Thunderbird was incorrectly inserting a couple extra spaces at the beginning of some sent email messages (Mozilla bug #564737). This bug was first reported just a few months ago and has already been reported by at least 56 different people. This bug is in the core code that is shared between all Mozilla applications, which means that the fix will impact Firefox, Seamonkey, etc. as well as Thunderbird.

Needless to say, there are other things I should have been working on when I got distracted by fixing these bugs. But I’d almost forgotten how rewarding it is to be able to contribute to open-source software in ways that benefit a lot of people.

The old version is not compatible with Thunderbird 3, and its author and maintainer appears to have abandoned it.

I’d love for him to integrate my changes into his version and resume maintaining it, but in the meantime, for the sake of making it available to people, I’ve released the new version myself.

Here’s a picture:

Interestingly, it’s been less than two days since I released it, and it’s already been downloaded by 71 people. Sweet!

This is a significant problem, since Microsoft Outlook, which has by far the biggest market share of any email client, is one of the clients that displays Mailman footers incorrectly.

Many people have complained about this problem to the maintainers of GNU Mailman, but they have declined to address it.  I don’t agree with their reasoning, but it is of course their prerogative as the volunteer maintainers of free software to decide that they’d rather maintain some sort of vision of purity in their code rather than actually make the it do what their users want it to.  Jan Ploski also has some interesting thoughts about this.

Fortunately, it’s our prerogative as users to fix it ourselves if they don’t .  Adrian Bye did this with a patch to Mailman way back in 2005, but the maintainers rejected his patch and it’s now out-of-date and incompatible with the current stable Mailman release.  Others have hacked together site-specific solutions using mimedefang, but no one has implemented a generic solution that can be deployed on top of a standard Mailman installation.  Until now, that is.

I’ve just released a script that can be deployed easily into a mimedefang installation to automatically reformat outbound Mailman messages to insert the footer into the text and/or HTML bodies of the message rather than as a separate attachment.  All you need to do to use it is install it into your mimedefang installation using the provided instructions, then modify the msg_footer setting inside Mailman to add a couple of special tokens which tell the script to reformat your footers.

Share and enjoy!  And hey, if you find this useful, maybe you can show your appreciation.

I don’t just do this for the heck of it.  I look for things in my archived email on a regular basis.  Most of the time I’m only looking back a few months or perhaps a couple of years, but I do occasionally find it necessary for one reason or another to go digging through the really old stuff.  To make that feasible,  I need to be able to quickly search hundreds of thousands of messages.

Over the years I have imposed a lot of requirements on acceptable solutions for this problem:

• I want to be able to store the email in a compressed form, so the search engine needs to understand how to decompress the email archives when indexing or retrieving messages from them.
• To achieve decent compression, the email has to be stored in files that hold multiple messages, rather than storing each message in its own file.  Therefore, the search engine needs to understand how to break up the mailbox files into separate messages, index them separately, and retrieve them separately during searches.
• I don’t want to use a proprietary or binary mailbox format — I want to be able to look at the mailboxes in a text editor and manipulate them easily with tools such as Perl.  I used to store my email in BABYL Files, but now I use mbox format (which Thunderbird and Eudora also use for local folders).
• I don’t want to be locked into a GUI — I need to be able to update the index, do searches, and retrieve results through the command line.

For many years, I was unable to find any actively maintained open-source software package that satisfied all of these requirements.  I was therefore stuck using freeWAIS 0.5, one of the very first Internet search engines, which was developed and released by Thinking Machines Corporation over 20 years ago.  I was an active developer on the project; my efforts were focused on making the indexing code faster and less of a disk-hog and fixing a myriad of bugs and memory leaks (freeWAIS was written in C; its primary authors, all of them Lisp programmers, were so used to automatic garbage collection that they were very bad about cleaning up after themselves).  Every since the freeWAIS project went defunct, I’ve maintained my own personal version of the code, layering hack on top of hack to keep it compiling and running on new versions of Linux.  It was gross, but it was good enough.

That is, it was good enough until I upgraded my Linux box at home about a month ago and went from 32-bit Linux to 64-bit Linux.  The freeWAIS code is very dependent on things like the size of an integer, the size of “time_t” and “off_t”, etc.  Furthermore, to say that the code is not particularly clean or portable would be a gross understatement.  When I rebuilt it for 64-bit, it stopped working.  After spending several hours trying unsuccessfully to yet again nurse it back to health, I decided to take another look around to see if any new search engines that would do what I need had come onto the scene.

I was delighted to discover mairix, a package written and maintained by Richard Curnow which bills itself as “a program for indexing and searching email messages stored in maildir, MH or mbox folders.”  Wow, someone went and wrote exactly the tool I needed.  w00t!

Well, actually, it’s not exactly the tool I needed, because when I set it up last night I discovered some minor issues with its parsing of mbox files.  But I fixed those issues and sent Richard my patches, and I hope they’ll be incorporated into the next release so I don’t have to go down the maintain-my-own-software road again .  (If you’re trying to use version 0.21 of mairix to index mbox files, email me and I’ll send you my patches.)

Mairix indexed my >400,000 email messages in 12 minutes (3.5GHz CPU, 7,200 RPM SATA hard drive).  The mairix index consumes only 144MB of space, despite the fact that my email archives take up 1.1GB compressed. It takes mairix less than 0.2 seconds to do an AND search for two search terms and save the 8 matching messages into an mbox.  That is simply incredible.

One of the ways Richard made mairix so fast is by using another tool he wrote, dfasyn, which he describes as “a tool for building general deterministic finite automata (DFAs) given a description as a non-deterministic finite automaton.”  You use a high-level syntax to describe the various legal state transitions, and dfasyn compiles that into some big numeric arrays and a very trivial function which transitions between machine states using nothing but pointer and integer arithmetic.  The nerd in me thinks this is très cool.

I’m posting about mairix both to let other people know about this great tool, and to give Richard the kudos he deserves for implementing it.

Share and enjoy!

On the other hand, organizing the copies of message I’ve sent is less straightforward.  Visiting my “Sent Items” folder every time I send a message and filing the message I’ve just sent is gross.  What I’ve done in the past is to leave all the copies in “Sent Items” and periodically file all of them one at a time.  This is time-consuming and error-prone, and what’s more, it could be at least partially automated — if I file a message I received in a particular folder, then something ought to be smart enough to figure out that the response I sent should go into the same folder.

After spending over an hour this morning filing “Sent Items” messages, my annoyance threshold was finally exceeded and I decided to automate the process as much as I could.  The result is file-sent-items.pl.  In a nutshell, this script reads each message in your “Sent Items” folder (or whatever it’s called), grabs the message IDs in the “Message-ID”, “Resent-Message-ID”, “In-Reply-To” and/or “References” headers, looks for a message with one of those IDs in your other folders, and if one is found, moves the sent message into its folder.  There’s a bit more too it than that, of course, which you can find out by reading the comment at the top of the script and running it with the “–help” option to get a usage message.

Of course, this script won’t be able to file messages that aren’t part of conversations already filed into other folders, but it still reduces by a lot the amount of filing you have to do by hand.

Share and enjoy!

You can configure Vonage to email you about voicemail messages.  The email contains the actual voicemail message as an audio attachment, but what it doesn’t have, inexplicably, is the name of the caller pulled from caller ID.  This has annoyed me ever since we switched to Vonage, and I recently finally got annoyed enough to finally do something about it.  I’m posting my solution here on the off chance that it’ll be useful to others.

To take advantage of this hack, you have to have the ability to filter the contents of your incoming email, e.g., with procmail.  If you don’t know what that means, then you should probably stop reading now. :-/

My solution consists of two scripts and two configuration changes.  First, the scripts:

1. The script vmail-cid.pl fetches the caller ID information for recent calls from www.vonage.com and saves it in a CSV file.
2. The script vonage-vmail-filter.pl reads the aforementioned caller ID CSV file, as well as a CSV file you’ve exported from your Outlook contacts, and uses the information in those files to filter a voicemail notification email message on stdin and send the (possibly modified) notification to stdout.  You can configure the script which of the two CSV files to read, i.e., you can take advantage of either the caller ID information from vmail-cid.pl, or the Outlook export CSV, or both.

Now, the configuration changes:

1. You need to set up a scheduled task, cron job, or whatever to run vmail-cid.pl periodically to keep the caller ID CSV file up-to-date.
2. You need to tie vonage-vmail-filter.pl into your email delivery, e.g., by editing your .procmailrc file (for which there is an example in a comment at the top of the script), sieve configuration, or whatever.

When all of this is done properly, then every voicemail notification you receive will be updated with the caller ID or Outlook Contacts name for the calling phone number, if it’s available, before it lands in your mailbox.

Click here to download a zip file containing the scripts.  See the comments at the top of the scripts for additional details.  As always, please feel free to send me any questions, comments and suggestions you might have.

Enjoy!

Suppose you’re happily working in a Cygwin window, and you need to load one of the files you’re working on into a Windows app. You could select File > Open… in the app and then browse to the ten-levels-deep Cygwin folder to find the file you need, or instead you could put this in your .bashrc:

e() { explorer $(cygpath -w $(pwd)) & }

Then, in the Cygwin window, just type “e <ENTER>”, and an Explorer window will pop up, opened to the current directory. You can now double-click on the file, open it from the right-click menu, or drag and drop into a running app.

A 30-minute 1920×1080 WMV is very large, like around 50MB.  We want it to be smaller before we publish it.  One way to make it smaller is to simply scale it down, but when the entire 1920×1080 canvas is shrunk down to a reasonable size, text which was perfectly legible before becomes unreadable.  Since the text is usually a crucial part of the demo, that’s unacceptable.

Usually, the presenter isn’t using the whole screen at any given time.  Rather, he’ll be focused on the content in one particular window for an extended period of time, and that window often takes up only around half of the screen.  Therefore, cropping the WMV to the active area is a possible way to the video file smaller without sacrificing legibility.  There’s just one problem — what happens when the presenter jumps around from window to window during the demo?

There are various expensive commercial video editors which will let you crop different segments of the video differently and then glom them all together into one video at the end.  However, I don’t like using commercial software when I can do it for free (this is probably about half on principle and half out of stubbornness).  Therefore, I set out to figure out how to do this with free tools.  The result is a script called wmv-to-panned-mov.pl.

This script uses the excellent free tool ffmpeg to slice and dice the input video with the appropriate cropping and produces a much smaller MOV file (i.e., QuickTime video) as its output.  You feed the script a CSV file indicating the start times, end times, and cropping areas of the various slices, and it does the rest.  There’s more information in the comment at the top of the script about how to use it.

Please comment here to let me know if you find this useful.  If the time I spent writing this script saves you some time or money, please consider sending a little something to my tip jar (but if you don’t, that’s OK too).