Archive for the ‘Free software’ Category

Hack of the day: Pulseaudio / Bluez: Switch headset between A2DP and HSP automatically

Tuesday, April 23rd, 2013

CGI script for RSS feed of comments on Hacker News posting

Thursday, January 24th, 2013

Perhaps I just don’t grok the zen of Hacker News, but I just don’t get why the site doesn’t provide RSS feeds of comments on postings.

I also don’t get why nobody has written something to provide this. Or, at least, if it has been written, I couldn’t find it (perhaps my Google mojo just isn’t high enough).

http://api.ihackernews.com/ gets part of the way there, but it’s an API, not an RSS feed, and it’s broken and has been for quite a while.

So I wrote a script to generate such a feed.

You can see it in action at http://jik3.kamens.brookline.ma.us/cgi-bin/hn-comment-feed.cgi?id=5107045 (which I will have to remove if there is so much interest in this Show HN that people clicking on the link overwhelms my tiny little VPS).

(If you want to comment about this, please do on the Hacker News posting about it rather than here.)

 

Free Linux cloud backups for cheap bastards

Saturday, November 19th, 2011

The last time I looked, there were no options I was happy with for backing up my Linux PC in the cloud for a reasonable price. That may have changed, and perhaps if I were less stubborn I’d pay CrashPlan a few bucks per month to back up my system,  but I feel compelled to build my own backup infrastructure for three reasons: (1) I want complete control over it; (2) I worry about a company backing up my data going belly-up and leaving me high and dry; and (3) I’m a cheap bastard.

Until recently I was backing up my data into a ReiserFS filesystem being stored in the Amazon S3 cloud via s3backer. That was costing me on average a little under $5 per month in storage and transaction costs.

Then I got an offer from AppSumo for 10GB of storage for life from LetsCrate for only $25. That got me wondering… There are a bunch of cloud storage / file sharing services on the Internet right now, and just about every one of them offers some amount of storage for free. Could I find away to take advantage of all that free storage to reduce my backup costs almost to nothing?

(more…)

Fedora 14 -> Fedora 15 upgrade notes

Thursday, July 7th, 2011

For some reason lost in the mists of time, I decided at about 7am yesterday that I just had to upgrade my Fedora 14 desktop to Fedora 15. Sometimes I get these weird ideas…

I followed the instructions on the Fedora Web site for upgrading using yum, and 2,978 downloaded packages and several hours later, I had a Fedora 15 system. But the fun was only beginning. Here, for the benefit of others who follow me, are the problems I encountered after the upgrade and what I did to fix them. Much of this I learned from other helpful people on the web, but some of it I had to figure out for myself.

  • First and foremost, my network performance went totally to hell immediately after the upgrade. With the F14 kernel, I was getting about 7 megabit/s on my cable-modem connection, while after the upgrade, I was getting around 500 kilobit/s, i.e., 1/14th of the pre-upgrade speed. It appears to be a problem with kernel support for my network adapter, a Realtek RTL8168c/8111c NIC that uses the r8169 kernel driver. I filed a bug about it, and then went out today and bought a Netgear GA311 PCI network adapter (RTL8110s using the same r8169 kernel driver), which so far appears to be working just fine.
    Note that before I replaced the network adapter, I was able to improve network performance a bit by reducing the MTU on the ethernet device to 1000 (“sudo ifconfig eth0 mtu 1000″).
  • Grub no longer lets me hit ESC or any other key while it’s doing its boot countdown to bring up a menu of kernels to choose from. If I want to boot a different kernel, I have to edit /boot/grub/grub.conf and reboot. I still haven’t figured out how to fix this, but I did file a bug about it.
  • The files in my desktop folder weren’t showing up on my desktop. To address this I had to run gnome-tweak-tool (you may need to “yum install gnome-tweak-tool” if you don’t have it) and enable “Have file manager handle the desktop” in the “File Manager” settings panel. While I was at it, I made some other settings changes in gnome-tweak-tool:
    • I reduced the default fonts from 10pt to 11pt, since things got a little bigger after the upgrade and I wanted to fit more text on the screen.
    • I enabled “Show date in clock” in the “Shell” panel.
    • I changed “Arrangement of buttons on the titlebar on the “Shell” panel to “All” because I wanted maximize and minimize buttons in addition to the default close button.
  • I wanted the Delete key to delete files, but alas, the default Nautilus binding for deleting files changed from Delete to Ctrl-Delete in F15. I found these instructions for fixing this, but I had to tweak them slightly. In particular:
    • Run gconf-editor, not dconf-editor as the other web page suggests.
    • Find org > gnome > desktop > interface > can-change-accels and check the checkbox next to it.
    • Open a Nautilus window, e.g., double-click the “home” icon on your desktop. Note that it took me a while to figure out that you must do this in a window, i.e., you can’t do it by right-clicking on a file on the desktop. Yes, this is counter-intuitive.
    • Select a file in the Nautilus window.
    • Open on the File menu and move your mouse down to “Move to Trash”.
    • Hit the Delete key to remove the old Ctrl-Delete key binding. Hit it again to create a new key binding, just for Delete.
    • Uncheck the check-change-accels checkbox and close gconf-editor.
  • For some reason I’m not sure of, I had nfs-utils and ypbind, neither of which I need or use, installed, and after the upgrade, systemd was trying to start ypbind and idmapd from nfs-utils. I removed both packages with “yum remove”.
  • For some bizarre reason, a bunch of my contacts in Pidgin moved from the groups they were in before to “Orphans” at some point during the upgrade. I had to move them all back into their correct groups one by one. Ugh.
  • No weather applet! Must have weather applet! A little googling led me to:
    • run “git clone https://github.com/simon04/gnome-shell-extension-weather.git”;
    • follow the instructions in README.md for building and installing the extension;
    • run “python weather-extension-configurator.py” to configure the extension settings described in README.md, which is easier than using the gsettings commands there; and
    • type Alt-F2, “r”, Enter to restart the GNOME shell and activate the extension. Thanks very much to the authors of this wonderful extension!
  • I had a launcher file (i.e., a “*.desktop” file) created in GNOME 2 that I wanted to add to my GNOME 3 favorites. Unfortunately, it appears that to add a launcher to your favorites it has to be in /usr/share/applications (well, actually, I probably could have put it in ~/.local/share/applications, but I didn’t think of that at the time). So I copied it into /usr/share/applications, launched it by moving my cursor to the “hot spot” in the upper left corner of the screen and typing the name of the launcher; and then when it was running, right-clicked on it in the favorites bar and selected “Add to Favorites”.
  • I wanted to be able to easily clear my desktop, like the “Show Desktop” button I had in my GNOME 2 panel. I found these instructions for fixing this.
  • The MySQL daemon wasn’t starting up properly after the upgrade. Apparently one of the ib_logfile* files in /var/lib/mysql was truncated or something. I did “sudo rm /var/lib/mysql/ib_logfile*” and then “systemctl start mysqld.service” worked just fine.
  • The ddclient program which I use to update my dynamic DNS record with OpenDNS.com and DynDNS.org was generating an error on shutdown because it was failing to create its PID file properly on startup. I fixed its init script to address this and submitted a patch here.
  • I desperately wanted an old-style dock visible all the time with my running applications in it. Fortunately I’m not the only one… I did “sudo yum install gnome-shell-extensions-dock” and then logged out and logged back in, and presto, I’ve got a dock.
  • In F14, I used the mail-notification program to generate notifications about new email. It stopped working after the upgrade so I thought it wasn’t supported in F15. Turns out I just had to weak it a bit.
    • launch gnome-session-properties;
    • select “Mail Notification” and click “Edit”; and
    • remove “–sm-disable” from the command field.
  • The cron daemon wasn’t starting properly after the upgrade. I’m not sure why, but I fixed it by doing “sudo systemctl enable crond.service” and “sudo systemctl start crond.service”.
  • I had an old-style init script in /etc/init/foo.conf that looked like this:
    start on runlevel 5
    stop on runlevel [!5]
    exec /home/jik/scripts/foo.pl
    respawn

    I had to replace it with /lib/systemd/system/foo.service which looks like this:

    [Unit]
    Description=Foo server
    After=network.target
    
    [Service]
    ExecStart=/home/jik/scripts/foo.pl
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    Then I did “sudo systemctl enable foo.service” and “sudo systemctl start foo.service”.

  • In F14, you could rely on the GNOME session launching script to execute the contents of your .bash_profile file, so that environment variables you set in .bash_profile would be inherited by all child processes. This doesn’t work in F15, so you have to set environment variables in .bashrc.

I’ll continue to update this list as new things come up.

It took a day or so to get used to, but I think the GNOME 3 UI is growing on me.

 

I guess I’m now a Mozilla core developer, too

Friday, July 23rd, 2010

About a month ago, I dived into the world of Mozilla add-on development by adopting the abandoned Thunderbird “Send Later” add-on and porting it to Thunderbird 3.1. The learning curve was pretty steep, and it took a lot more work than I expected to stabilize the add-on, but I think it was worth it, considering that in the two weeks since I released it, almost 2,000 people have downloaded it and at least 444 of them are using it.

Emboldened by that, I decided to take a stab at fixing two bugs in the core Thunderbird code that have been driving me crazy. That, too, required a steep learning curve, but in the end, I was able to submit fixes for two bugs, one quite old and one new in Thunderbird 3.1, affecting a whole bunch of people:

  • It was impossible to remove attachments from some MIME messages, including MIME messages generated by the Mac Mail client (Mozilla bug #351224). This bug has been reported by at least 30 different people and was first reported almost four years ago. Fixing it required rewriting pretty much an entire module within C++ source code for Thunderbird.
  • Thunderbird was incorrectly inserting a couple extra spaces at the beginning of some sent email messages (Mozilla bug #564737). This bug was first reported just a few months ago and has already been reported by at least 56 different people. This bug is in the core code that is shared between all Mozilla applications, which means that the fix will impact Firefox, Seamonkey, etc. as well as Thunderbird.

Needless to say, there are other things I should have been working on when I got distracted by fixing these bugs. But I’d almost forgotten how rewarding it is to be able to contribute to open-source software in ways that benefit a lot of people.

I guess I’m a Mozilla add-on developer now

Sunday, July 11th, 2010

I just released a port of the “Send Later” Mozilla Thunderbird add-on for Thunderbird 3.1+.

The old version is not compatible with Thunderbird 3, and its author and maintainer appears to have abandoned it.

I’d love for him to integrate my changes into his version and resume maintaining it, but in the meantime, for the sake of making it available to people, I’ve released the new version myself.

Here’s a picture:

Interestingly, it’s been less than two days since I released it, and it’s already been downloaded by 71 people. Sweet!

Solving the GNU Mailman MIME message footer problem

Tuesday, May 25th, 2010

If you administer a GNU Mailman installation, you are probably aware that message footers don’t always work quite right: if a message submitted to a list is entirely plain text with no attachments, then the footer is fine, but if it’s formatted in HTML or has attachments, then the footer is added to the message as a separate message part, and some email clients display it as an attachment which must be clicked on to view, rather than displaying it as part of the message text.

This is a significant problem, since Microsoft Outlook, which has by far the biggest market share of any email client, is one of the clients that displays Mailman footers incorrectly.

Many people have complained about this problem to the maintainers of GNU Mailman, but they have declined to address it.  I don’t agree with their reasoning, but it is of course their prerogative as the volunteer maintainers of free software to decide that they’d rather maintain some sort of vision of purity in their code rather than actually make the it do what their users want it to.  Jan Ploski also has some interesting thoughts about this.

Fortunately, it’s our prerogative as users to fix it ourselves if they don’t :-).  Adrian Bye did this with a patch to Mailman way back in 2005, but the maintainers rejected his patch and it’s now out-of-date and incompatible with the current stable Mailman release.  Others have hacked together site-specific solutions using mimedefang, but no one has implemented a generic solution that can be deployed on top of a standard Mailman installation.  Until now, that is.

I’ve just released a script that can be deployed easily into a mimedefang installation to automatically reformat outbound Mailman messages to insert the footer into the text and/or HTML bodies of the message rather than as a separate attachment.  All you need to do to use it is install it into your mimedefang installation using the provided instructions, then modify the msg_footer setting inside Mailman to add a couple of special tokens which tell the script to reformat your footers.

Share and enjoy!  And hey, if you find this useful, maybe you can show your appreciation. :-)

My new favorite open-source software: mairix by Richard Curnow

Friday, January 8th, 2010

I am an email pack-rat.  I have saved just about every email message I’ve sent or received at work or at home in the past twenty years (excluding messages sent to public mailing lists).  That’s a lot of email, over 400,000 messages.

I don’t just do this for the heck of it.  I look for things in my archived email on a regular basis.  Most of the time I’m only looking back a few months or perhaps a couple of years, but I do occasionally find it necessary for one reason or another to go digging through the really old stuff.  To make that feasible,  I need to be able to quickly search hundreds of thousands of messages.

Over the years I have imposed a lot of requirements on acceptable solutions for this problem:

  • I want to be able to store the email in a compressed form, so the search engine needs to understand how to decompress the email archives when indexing or retrieving messages from them.
  • To achieve decent compression, the email has to be stored in files that hold multiple messages, rather than storing each message in its own file.  Therefore, the search engine needs to understand how to break up the mailbox files into separate messages, index them separately, and retrieve them separately during searches.
  • I don’t want to use a proprietary or binary mailbox format — I want to be able to look at the mailboxes in a text editor and manipulate them easily with tools such as Perl.  I used to store my email in BABYL Files, but now I use mbox format (which Thunderbird and Eudora also use for local folders).
  • I don’t want to be locked into a GUI — I need to be able to update the index, do searches, and retrieve results through the command line.

For many years, I was unable to find any actively maintained open-source software package that satisfied all of these requirements.  I was therefore stuck using freeWAIS 0.5, one of the very first Internet search engines, which was developed and released by Thinking Machines Corporation over 20 years ago.  I was an active developer on the project; my efforts were focused on making the indexing code faster and less of a disk-hog and fixing a myriad of bugs and memory leaks (freeWAIS was written in C; its primary authors, all of them Lisp programmers, were so used to automatic garbage collection that they were very bad about cleaning up after themselves).  Every since the freeWAIS project went defunct, I’ve maintained my own personal version of the code, layering hack on top of hack to keep it compiling and running on new versions of Linux.  It was gross, but it was good enough.

That is, it was good enough until I upgraded my Linux box at home about a month ago and went from 32-bit Linux to 64-bit Linux.  The freeWAIS code is very dependent on things like the size of an integer, the size of “time_t” and “off_t”, etc.  Furthermore, to say that the code is not particularly clean or portable would be a gross understatement.  When I rebuilt it for 64-bit, it stopped working.  After spending several hours trying unsuccessfully to yet again nurse it back to health, I decided to take another look around to see if any new search engines that would do what I need had come onto the scene.

I was delighted to discover mairix, a package written and maintained by Richard Curnow which bills itself as “a program for indexing and searching email messages stored in maildir, MH or mbox folders.”  Wow, someone went and wrote exactly the tool I needed.  w00t!

Well, actually, it’s not exactly the tool I needed, because when I set it up last night I discovered some minor issues with its parsing of mbox files.  But I fixed those issues and sent Richard my patches, and I hope they’ll be incorporated into the next release so I don’t have to go down the maintain-my-own-software road again :-).  (If you’re trying to use version 0.21 of mairix to index mbox files, email me and I’ll send you my patches.)

Mairix indexed my >400,000 email messages in 12 minutes (3.5GHz CPU, 7,200 RPM SATA hard drive).  The mairix index consumes only 144MB of space, despite the fact that my email archives take up 1.1GB compressed. It takes mairix less than 0.2 seconds to do an AND search for two search terms and save the 8 matching messages into an mbox.  That is simply incredible.

One of the ways Richard made mairix so fast is by using another tool he wrote, dfasyn, which he describes as “a tool for building general deterministic finite automata (DFAs) given a description as a non-deterministic finite automaton.”  You use a high-level syntax to describe the various legal state transitions, and dfasyn compiles that into some big numeric arrays and a very trivial function which transitions between machine states using nothing but pointer and integer arithmetic.  The nerd in me thinks this is très cool.

I’m posting about mairix both to let other people know about this great tool, and to give Richard the kudos he deserves for implementing it.

Share and enjoy!

Hack of the day: auto-filing IMAP “Sent Items” folder

Wednesday, December 30th, 2009

I file my email in different IMAP folders, grouped by topic rather than the “received messages in one folder, sent in another” model imposed nowadays by most mail clients.  It’s easy to keep the messages I receive organized by topic — when I’m done handling a message, I simply move it into the appropriate folder.

On the other hand, organizing the copies of message I’ve sent is less straightforward.  Visiting my “Sent Items” folder every time I send a message and filing the message I’ve just sent is gross.  What I’ve done in the past is to leave all the copies in “Sent Items” and periodically file all of them one at a time.  This is time-consuming and error-prone, and what’s more, it could be at least partially automated — if I file a message I received in a particular folder, then something ought to be smart enough to figure out that the response I sent should go into the same folder.

After spending over an hour this morning filing “Sent Items” messages, my annoyance threshold was finally exceeded and I decided to automate the process as much as I could.  The result is file-sent-items.pl.  In a nutshell, this script reads each message in your “Sent Items” folder (or whatever it’s called), grabs the message IDs in the “Message-ID”, “Resent-Message-ID”, “In-Reply-To” and/or “References” headers, looks for a message with one of those IDs in your other folders, and if one is found, moves the sent message into its folder.  There’s a bit more too it than that, of course, which you can find out by reading the comment at the top of the script and running it with the “–help” option to get a usage message.

Of course, this script won’t be able to file messages that aren’t part of conversations already filed into other folders, but it still reduces by a lot the amount of filing you have to do by hand.

Share and enjoy!

Gross hack of the day: adding names to Vonage voicemail email notifications

Tuesday, December 29th, 2009

I’ve been using Vonage for telephone service for a couple of months, and I’m quite happy with it.  Their feature set is quite competitive, and I’m paying them significantly less than I’d be paying Verizon, Comcast or RCN for equivalent service.  (Shameless plug: if you’d like to give Vonage a try, send me email and I’ll send you a referral.  Both you and I will get a free month of service.)

You can configure Vonage to email you about voicemail messages.  The email contains the actual voicemail message as an audio attachment, but what it doesn’t have, inexplicably, is the name of the caller pulled from caller ID.  This has annoyed me ever since we switched to Vonage, and I recently finally got annoyed enough to finally do something about it.  I’m posting my solution here on the off chance that it’ll be useful to others.

To take advantage of this hack, you have to have the ability to filter the contents of your incoming email, e.g., with procmail.  If you don’t know what that means, then you should probably stop reading now. :-/

My solution consists of two scripts and two configuration changes.  First, the scripts:

  1. The script vmail-cid.pl fetches the caller ID information for recent calls from www.vonage.com and saves it in a CSV file.
  2. The script vonage-vmail-filter.pl reads the aforementioned caller ID CSV file, as well as a CSV file you’ve exported from your Outlook contacts, and uses the information in those files to filter a voicemail notification email message on stdin and send the (possibly modified) notification to stdout.  You can configure the script which of the two CSV files to read, i.e., you can take advantage of either the caller ID information from vmail-cid.pl, or the Outlook export CSV, or both.

Now, the configuration changes:

  1. You need to set up a scheduled task, cron job, or whatever to run vmail-cid.pl periodically to keep the caller ID CSV file up-to-date.
  2. You need to tie vonage-vmail-filter.pl into your email delivery, e.g., by editing your .procmailrc file (for which there is an example in a comment at the top of the script), sieve configuration, or whatever.

When all of this is done properly, then every voicemail notification you receive will be updated with the caller ID or Outlook Contacts name for the calling phone number, if it’s available, before it lands in your mailbox.

Click here to download a zip file containing the scripts.  See the comments at the top of the scripts for additional details.  As always, please feel free to send me any questions, comments and suggestions you might have.

Enjoy!