Jaro-Winkler in PL/pgSQL

We needed to do a fuzzy text match in PostgreSQL, but the fuzzy search algorithms that are available in the extensions that come in the Debian Postgres packages are pretty crude. Jaro-Winkler distance is available in the extension pg_similarity, but that requires doing make; make install, which makes installation cumbersome our development and production environments. An alternative is installing postgresql-plpython and using the Jaro-Winkler distance from the jellyfish library, but that reqires installing the package and enabling the pg_plpythonu extension on all databases. So I took some time and ported the algorithm to PL/pgSQL, which does not require any extensions:

-- Ported from jellyfish/_jellyfish.py

CREATE OR REPLACE FUNCTION jaro_winkler(ying TEXT, yang TEXT)
    RETURNS float8 AS $$
DECLARE
    ying_len integer := LENGTH(ying);
    yang_len integer := LENGTH(yang);
    min_len integer := GREATEST(ying_len, yang_len);
    search_range integer;
    ying_flags bool[];
    yang_flags bool[];
    common_chars float8 := 0;
    ying_ch TEXT;
    hi integer;
    low integer;
    trans_count integer := 0;
    weight float8;
    i integer;
    j integer;
    jj integer;
    k integer;
BEGIN

    IF ying_len = 0 OR yang_len = 0 THEN
        RETURN 0;
    END IF;

    search_range := (GREATEST(ying_len, yang_len) / 2) - 1;
    IF search_range < 0 THEN
        search_range := 0;
    END IF;
    FOR i IN 1 .. ying_len LOOP
        ying_flags[i] := false;
    END LOOP;
    FOR i IN 1 .. yang_len LOOP
        yang_flags[i] := false;
    END LOOP;

    -- looking only within search range, count & flag matched pairs
    FOR i in 1 .. ying_len LOOP
        ying_ch := SUBSTRING(ying FROM i for 1);
        IF i > search_range THEN
            low := i - search_range;
        ELSE
            low := 1;
        END IF;
        IF i + search_range <= yang_len THEN
            hi := i + search_range;
        ELSE
            hi := yang_len;
        END IF;
        <<inner>>
        FOR j IN low .. hi LOOP
            IF NOT yang_flags[j] AND
                 SUBSTRING(yang FROM j FOR 1) = ying_ch THEN
               ying_flags[i] := true;
               yang_flags[j] := true;
               common_chars := common_chars + 1;
               EXIT inner;
            END IF;
        END LOOP inner;
    END LOOP;
    -- short circuit if no characters match
    IF common_chars = 0 THEN
        RETURN 0;
    END IF;

    -- count transpositions
    k := 1;
    FOR i IN 1 .. ying_len LOOP
        IF ying_flags[i] THEN
            <<inner2>>
            FOR j IN k .. yang_len LOOP
                jj := j;
                IF yang_flags[j] THEN
                    k := j + 1;
                    EXIT inner2;
                END IF;
            END LOOP;
            IF SUBSTRING(ying FROM i FOR 1) <>
                    SUBSTRING(yang FROM jj FOR 1) THEN
                trans_count := trans_count + 1;
            END IF;
        END IF;
    END LOOP;
    trans_count := trans_count / 2;

    -- adjust for similarities in nonmatched characters
    weight := ((common_chars/ying_len + common_chars/yang_len +
               (common_chars-trans_count) / common_chars)) / 3;

    -- winkler modification: continue to boost if strings are similar
    IF weight > 0.7 AND ying_len > 3 AND yang_len > 3 THEN
       -- adjust for up to first 4 chars in common
       j := LEAST(min_len, 4);
       i := 1;
       WHILE i - 1 < j AND
             SUBSTRING(ying FROM i FOR 1) = SUBSTRING(yang FROM i FOR 1) LOOP
           i := i + 1;
       END LOOP;
       weight := weight + (i - 1) * 0.1 * (1.0 - weight);
    END IF;

    RETURN weight;

END;
$$
LANGUAGE plpgsql;

Failed escape

I've been using Ubuntu since the spring of 2005, that is for over 8 years. I appreciate the Debian underpinnings, predictable releases, general fit and finish. However, now I'm using Gnome Shell instead of Unity and I feel that there's not much value provided to me in over plain Debian Testing. Moreover, I'm unconvinced in the general direction of Ubuntu is taking for the past couple of years, the "Not Invented Here" mentality of developing competitors to other Open Source projects (Unity, indicatorors, Mir). So I decided to switch.

I downloaded a Debian Testing (wheezy) netboot 23 MB mini.iso image, set it up to boot with GRUB from a USB flash drive, made a copy of my system partition, and rebooted. Long story short, the installer failed on the base system install stage, something about unable to authenticate the busybox package. I've spent a couple of hours fumbling around trying to get past the error, downloaded a different version of the wheezy netboot image, to no avail. I also tried the 200 MB mini-CD image, but that failed with "Could not mount the CD-ROM" being booted off a flash drive.

After a while I just restored the backup, updated the new root FS UUID in /etc/fstab and /boot/grub/grub.cfg and went on with life. On Ubuntu. I'm still on Precise, probably worth trying an upgrade to Raring. I'll keep the backup of the Precise system partition, though.

App Camp 2011, first day

So, App Camp kicked off at about 6pm on Friday with a greeting from Andrius Kubilius, the prime minister of Lithuania, followed by Rimantas Žylius, the minister of economy. The politicians were wearing jeans and jackets, no ties already. Kubilius reminded us on numerous occasions that it's the XXIst century, confessed that he wouldn't be able to make a mobile app, and asked for an app that would solve the budget deficit issue. The MC commented that the PM is not as tech-naive as he presents himself, as he's using both and iPad and a Samsung Galaxy Tab, because, as he tells, the apps are different on these. Žylius reminded us that everything is possible, if even he could become a cabinet minister.

Several short keynotes by foreign guests and local mobile community tycoons were followed by the app pitches. Each team had 60 seconds to present their idea to attract collaborators. Most of them have been pretty good, the presenters were successful in conveying their idea and getting people interested.

There were several recurring themes: 3 pitches for apps for fishermen, 3 pitches for loyalty system apps (with the promise of ridding your wallet of all these loyalty cards), a coulple of GPS adventure games geared to get geeks of their monitors, a couple of variations on the where to go for lunch theme.

Other app ideas that I found noteworthy: an egg cooking simulator for small kids, rules of the road reader & quiz, onboard Java IDE, location-based hitchhiking hookup, mobile app based on the sviesoforai.lt traffic congestion IS, a mobile app to report places of environmental concern for green NGOs to take care of, GPS taxi meter to tell if you're getting ripped off, a mobile confession app that uploads the audio to a CMS for a priest to listen to and grant penance over the air. A game to learn the map by navigating Lithuania and Europe by choosing the direction by the names of the adjacent towns. The prize for the best pitch (Samsung Galaxy S II) was awarded to the guy operating a clothing swap web site, who promissed his collaborators access to a database of 100k girls in Lithuania with names, profile photos, clothes sizes, and contact details :) That's the team I joined!

Then there was time for networking over soft drinks and light snacks. Later, beer appeared, too. It appeared that a lot of the app ideas were proposed by marketers willing to promote the idea and looking for devs to implement it for free. Undecided developers were floating from one circle to the next, looking for a good app idea to invest their weekend in.

My clothes swap team consists of 4 developers and a designer. At 9am on Sunday the development began with brainstorming over coffee and cinnamon buns. We were driven off the premises at 10pm. We have a working app with the main functionality fleshed out, but perhaps it's too early for screenshots yet. If interested, the code is on GitHub: https://github.com/alga/AppCamp16. My team mates who have some iOS experience, but have not dealt with Android before, commented how easy developing for Android is.

The whole event is held in the airy, light, inspiring spaces of the National Gallery of Art, one of my favourite buildings of all. The venue was a substantial factor in my decision to come and participate. I'm very glad I came, and not only because of the building. I'm looking forward to the next day (which is quite soon already, time to catch some sleep!)

Ubuntu Oneiric FTW :(

I was used to having uptimes in the order of weeks on the laptop. A couple of suspends, resumes every day, hooking up and disconnecting from the external monitor. Everything worked without a hitch on Ubuntu during the last couple of years. I was being conservative and used Gnome 2 without compositing most of that time. But Ubuntu 11.10 Oneiric is different. Gnome 3 used to crash 5 times a day, Unity has a ~50% chance of surviving a day or two. Most often things seem to crash after the screensaver has kicked in. Yay Ubuntu. Might be time to look elsewhere. Linux Mint or good ole Debian.

Make use of the Caps Lock key

So, there is this Caps Lock key on all keyboards, sitting untouched in polite society. Why not put it to good use? Here are the commands that map the Find symbol to the Caps key on Linux. Stick them somewhere in the X session startup:

xmodmap -e 'remove Lock = Caps_Lock'
xmodmap -e 'keycode 0x42 =  Find'

Then I map the key to the keyword completion command in Emacs like this. The second command is needed for Emacs in a terminal window:

(global-set-key [find] 'dabbrev-expand)
(global-set-key "\M-[1~" 'dabbrev-expand)

"Version Control by Example" -- first impressions

One summer night when browsing Reddit or HackerNews, I came across an offer of a free book about VCS'es in return for filling in a simple survey. I thought, what the heck, and did it. They promised to dispatch it within 2 or 3 months. This morning an envelope arrived.

The book is printed on a heavy, coated paper, which imparts a sense of quality. Certain design and editing aspects, though, give away it being a self-published work. In particular, I find the coloured borders of pages are a bit too noisy for my taste, and punctuation like "?!?!?" would have never gotten through a professional editor.

The author, Eric Sink, curiously, is the founder of AbiWord open source project. He's been is in VCS business for 15 years. His company started by offering tools for SourceSafe, and is now promoting their own open source (Apache licensed) source control management system Veracity.

Obviously, promoting Veracity is the motive of giving away books for free. However, along with Veracity, the author gives an equal standing and an honest review of Subversion, Mercurial and Git, along with a general introduction into version control concepts and some sage advice about best practices in version control. The book presents each VCS by going through an example of two developers collaborating on a small and silly C program, which is easy to follow and presents the peculiarities of each VCS in a natural setting.

I'm sure this will become my handbook tome when dealing with Mercurial and Git. I have long fantasized about having a nice dictionary of command equivalents in different VCS'es (clone/branch, fetch/pull/update), and the book has such a table in an appendix. However, sadly, Bazaar is missing from this comparison and the rest of the book, although it is mentioned as one of the three most popular DVCSes in the introduction.

Altogether, I'm thankful for the gift of a book that has, doubtless, taken a lot of time and effort to make.

Updating Samsung Galaxy S SCL GT-I9003 to Gingerbread in Ubuntu

So, I came across the news on the net that Android Gingerbread 2.3.4 update has been made available by Samsung to GT-I9003 phones. Apparently, it was being rolled out on a country or region basis via Samsung's Kies application. First I tried updating the phone with Kies in Virtualbox, but that didn't quite work, I could not get Kies to connect to the device. (Might be the reason was that the phone was not in Kies USB mode). Then I tried booting the OEM Windows Vista on the laptop. Vista had worked after I copied the partition from another hard drive, but now it refused to boot. Apparently, system could come up from hibernation, but depended on the hard drive unique ID in order to boot. Some web browsing, several reboots and trials later, I repaired the Vista boot configuration with the help of a warez Vista DVD. The laptop's recovery partition is useless -- basically it offers restoring from an external backup made earlier or a "factory reset". No Windows recovery, no command prompt.

So, turns out Samsung Kies sucks on real Windows as much as it does on Virtualbox. Granted, it did connect to my phone, but I could get more out of it than a backup of the 7 contacts in my phone's memory (the rest are synch'ed with Google). But backing up the 300 photos I have made Kies show a progress bar for 15 mins and then choke. And worst of all, there was no update of any kind offered! I wasted about 10 minutes staring at progress bars waiting for it to install, and then some 30 more minutes staring at progress bars waiting for the backups to fail.

So then I looked again at flashing the phone myself. I got the firmware from samfirmware.com, found the instructions for flashing with the leaked proprietary Samsung Windows flasher called Odin, got the Open Source flasher Heimdall (Debian packages worked fine), found an encouraging blog post on how to use it.

The firmware from Samfirmware is a RAR archive (password: samfirmware.com) containing a tar:

-rwxrwx--- 1 alga alga 191658005 2011-09-19 01:23 I9003XXKPH_I9003XXKPH_I9003SERKPH.rar
-rw-rw-r-- 1 alga alga 353699840 2011-08-10 09:53 I9003XXKPH_I9003XXKPH_I9003SERKPH_HOME.tar

Inside the tar, there are the firmware images:

$ tar xvvf I9003XXKPH_I9003XXKPH_I9003SERKPH_HOME.tar
-rwxr--r-- jonghoon/na-cdma-sw 262144 2011-08-09 15:19 boot.bin
-rwxr--r-- jonghoon/na-cdma-sw 1310720 2011-08-09 15:19 Sbl.bin
-rwxr--r-- jonghoon/na-cdma-sw 618496 2011-08-09 15:19 param.lfs
-rwxr--r-- jonghoon/na-cdma-sw 5787648 2011-08-09 15:19 normalboot.img
-rwxr--r-- jonghoon/na-cdma-sw 324751360 2011-08-09 15:19 system.rfs
-rwxr--r-- jonghoon/na-cdma-sw 12582912 2011-08-08 06:23 modem.bin
-rwxr--r-- jonghoon/na-cdma-sw 8378368 2011-08-09 15:20 cache.rfs

Apart from these, you need a partition table image (*.pit, latona_20110114.pit in my case), that can be downloaded from the Samfirmware site, too.

The phone needs to be placed in download mode. Everyone on the internet says this can be done by holding down Volume Down, Home, and Power buttons at the same time, but this did not work for me. An alternative is to use adb:

$ adb reset download

Then the flash command is as follows:

$ heimdall detect
Device detected
$ heimdall flash --pit latona_20110114.pit --primary-boot boot.bin --secondary-boot Sbl.bin --normal-boot normalboot.img --cache cache.rfs --modem modem.bin --param param.lfs --system system.rfs

About 2 tense minutes of progress counters uploading these files, a long reboot, and presto! The phone boots up into Android 2.3.4 Gingerbread!

The heimdall flash command can be run with the --repartition flag or without it. Repartitioning is needed if the partition layout is different, or, as one forum post suggests, a filesystem type changed. However, it seems that the partition layout is the same for all GT-I9003 versions (Samfirmware suggests the same pit file), so just flashing the partition images is OK.

Looking at other firmware archives, it looks like the number of images and their names vary. You just need to match up the files you have with the heimdall option flags (in other words, partition names of your phone). The heimdall print-pit can be useful here, it prints out the actual partition layout of the device.

The nice thing about flashing Android updates is that none of the user data was overwritten -- the system lives on separate partitions from installed programs and their data. Although, of course, there is a risk that something goes wrong, so it's wise to back up data before flashing.

Now, speaking of Gingerbread, it seems that most of the goodies added in it Samsung had already forward-ported in their Froyo firmware. I hope that they fixed the stability issues. My phone used to crash often with a black screen on Froyo when sitting idle.

Android FTW

Finally, I got an Android phone. It's a Samsung Galaxy S SCL. Got it at the end of Friday, had the whole weekend to play.




Small, posh box. Quality accessories. A charger the size of a matchbox, practically just the plug. Earbuds with rubbery thick wires that do not tangle up. The device is slick, the UI is fast and smooth.

The UI is all translated, with a couple of amusing bugs. One of them is that the Bluetooth menu is, in fact, Italian, not Lithuanian (it, lt, big deal!) "Impostazioni Bluetooth. Ricerca dispositivi." Then there are some usual cases where the translator did not quite get what they're translating. "Build number" in the system information is translated as "Sukurti numerį" (literally "create a number").

The Android Market is amazing. I found everything I needed: an ssh client, a PasswordSafe program, a fb2/html/txt book reader, an OpenStreetMap based vector map program with address search and routing, an app for WindGuru and Windfinder, a virtual drum kit like my bro had on iPhone4, etc. Usually there are more that one choice for each particular type of app. But I'm going to write my own fuel log app anyway, as the one that does everything I want and a bit more is castrated in the free version.

Turns out, with Android it's even harder to hide your data from Google. Just enter your Google account creds once, say in the Google Talk app, and look, your address book is sync'ed with your Google account too! It can all be turned off, of course, but still. That's my current tinfoil hat conspiracy theory as to why Google invested in Android -- to gob up everyone's address books!

I tried avoiding feeding all my data to Google. I played with SyncEvolution for two hours trying to fish out the data from my Symbian phone. In the end it worked, I had the phonebook on the laptop as a directory with 350 vcards. Now it's just a matter of adding the new device, and pyushing it up there, right? Bzzzt, what do you mean Android not supported!? Turns out it's true. Android has no SyncML support built in. There are apps that might or might not work over the network, but Bluetooth, here and now, is out of the question.

I went to the Android Market and looked for a phonebook import app. Found one that did exactly what I needed -- filled the address book from a directory with vCards. A couple of minutes, and here we have it! But wait, all non-ascii characters got mangled, double-UTF-8-encoded. Good job the app is GPL, bzr repo available at the author's web site. Another half-hour, and I have the SDK all set up with Eclipse integration. A quick look at the source, a couple of recompiles and reruns, one offending line commented out, and the phonebook is all imported. Only later I found that the standard phone book app has the vCard import/export too, perhaps even without bugs! The patch for the import app is still to be sent.

Anyway, now I'm diving into Android dev tutorials and reading Bloch's Effective Java.

Mobile Monday Vilnius (Nov 2010)

A couple of hours ago I participated in the Mobile Monday event. It was quite cramped, also it was quite fun. There were four 15 minute presentations. Here's a summary:

Omnitel: smarphones are, like, selling, and, like, their market share is, like, increasing. Over 50% of the phones we sell nowadays "have an operating system".

Microsoft: Windows Phone 7. We realised that our mobile OS offering is crap, so we took a fresh start. We're going to give a shot at the exercise we failed at back in 1985: try to imitate what Apple is doing! See, it's all touchy and slidy and aesthetically appealing! Apps are the key, so please go and download our SDK for free! Castrated Visual Studio, emulators, templates, everything is free! Then you can submit apps to our app store, we'll review them, and then you will be able to make moneys! The platform you use to develop apps is a mix of our browser bling platform and our gaming console toolkit!

Then there was a 15 minute breather break, as the air got quite stuffy.

Nokia: Yeah, well, we realised that we had a lot of stuff (a slide with logos for Java, Python, Qt, Maemo, Meego, Symbian, and some more) so we decided to get our story straight: Qt (pronounced "cute")!!! It's the new Java, write once, run anywhere!!! Symbian, Meego, it's all irrelevant, Qt is what you'll develop your apps in! Also, we have an app store we call OVI. We will sell your Qt apps there. It's really cool, the customer is billed on their mobile bill, we have agreements with most of the providers in the world. It will cost you more, but with no credit cards involved, customers have a lower barrier to buy. Also, we encourage and invite local apps and local services, create a great local app, we will put in on the featured page for your locale, and you'll be RRRRRich!

Some guy: Mobile web. A lot of people use web on their mobile. Uhhh. So we should develop mobile versions of our sites. But having http://m.* in front of URLs kinda sucks, and people disregard these, just go to the main site and pan and zoom. CSS3 and HTML5 is the answer. Apple has a lot of nice demos of that! But wait, they're only drafts now. And Javascript is the answer! Thanks, bye.

Then there was a panel with the presenters FINALLY taking questions.
Audience: MS guy, what are the hidden costs in your platform?
MS guy: it's free! Download with one click, install, develop your content, submit!
Audience: but come on now, how much does it cost in reality?
MS guy: I'm telling you, it's really free.
Audience: What about VisualStudio?
MS guy: We give you a special version we call "Visual Studio Express" for free! Well, if you want to experiment with the look&feel, you need to cash out for the full blown one, but to summarize it's free.
Audience: Nokia, the Symbian Foundation has fallen apart, what do you make of it?
Nokia guy: Symbian, Shmimbian, it's all irrelevant, Cute is the answer!
Audience: What about the horrible signing process?
Nokia guy: we fixed it! If you submit an app for review to our OVI store, we'll sign it for you! And Qt is the new thing!
Audience: what about the market share? How much market do you gather your plaform has?
Nokia guy: Frankly, I don't know and don't care. And as I said, platforms are a thing of the past, Qt is the king.
MS: We hope we will get some market share! PLZ help, but sorry, you need to get someone abroad to buy you a phone and then submit an app for you. Anyways.
Omnitel: Weeeel, obviously, S60 is by far the moooost popular smartphone platform in Lithuania, then there is Apple, and Android is rising, and RIM is also showing promise...

Then the questions were over and there was some shmoozing over wine and finger food. By then it was pas 8pm, so the crowd dissipated quickly.

Freeing up some space on Ubuntu

I tried upgrading to Ubuntu Maverick Meerkat, but, as it often happens, the upgrade manager brushed me off saying I need to clear up about 970 MB of disk space on the root filesystem.
Time to buy a new hard drive, but in the meantime I still want to upgrade with what I have.

apt-get autoremove freed up some space, but more than an order of magnitude less than the gig I needed to find.

deborphan has always been a great tool to seek out the unneeded packages that use up disk space. If you run it without any arguments, it reports the libraries that are not required by any application. So you can safely remove every package deborphan reports unless you know you need it (for your own development, proprietary software you installed manually, etc.). When deborphan reports nothing, it's time to lax up the requirements:

   deborphan -anz | sort -n

This lists all packages that are not required by other packages, even if they aren't libraries, and sorts them in a descending order by size. Old kernel images show up in this list near the bottom. There's usually more low-hanging fruit at the end of the list that can help you free up several hundred megs of space. Here you need to be careful not to delete something that you need from the system, as with these flags deborphan lists even packages that are apps rather than libraries, and even includes those that are suggested or recommended by other packages.

We're still several hundred megs short, but there are no obvious large packages to remove. The space is taken up by hundreds of relatively small ones. An effective technique I have found for this situation is start up Synaptic, order packages by origin, then go through restricted, multiverse and universe, sort packages by install status and size, and remove all the small packages you know nothing about, and various utilities you installed and used once. Sometimes it will threaten to remove a program that you need, then you need to back out. After you apply the changes, it's a good idea to close Synaptic and let apt-get autoremove and deborphan have a go again. If you're still not there, start up Synaptic again and prune Universe some more.

Presto! I got my gig (and more!) of free space by removing old kernels (480 MB), sacrificing Flightgear, QCad parts library, and lots of small utilities from Universe (900 MB).

EuroPython 2010!

Yay, my talk Zope Components for the Win got accepted to EuroPython! I'm going, and probably taking my wife and a baby too!

upicasa -- an upload script for PicasaWeb

So, as planned last time, I developed my PicasaWeb script a bit further. After I got home from a niece's first birthday party, I had some photos I wanted to upload, so I added the support for listing and creating albums. Then today I moved the authentication credentials out of module globals into a combination of a config file and interactive email/password query. With my password out of the source code it became ready for the initial import into a version control repository.

I chose Mercurial as a VCS just to try it out. Thus, Bitbucket was a natural place to host the repository. Unfortunately, yesterday evening it was inaccessible. Mercurial feels really fast and convenient.

Bitbucket was a total blast to get going. I registered in with my Launchpad OpenId, created a new repository for my project, uploaded my SSH pubkey, and was able to push my initial version within two or three minutes. Way to go, dudes! Very clear, simple, and good looking. A stark contrast with Launchpad, which leaves me confused whenever I try to do something that should be really simple. Here's the link to the upicasa hg repo on Bitbucket.

So, anyway, I wrote a setup.py script, a README file, polished the script a bit and released it to PyPI: upicasa 0.1. Writing the setup.py and the README in order to package the script took about the same time, if not more, than writing the script in the first place, but that's how it goes with Open Source, doesn't it?

The more I use PicasaWeb the more I like it. It has a nifty semi-automatic face recognition feature. It makes tagging people on photos pass the threshold of convenience where it becomes practial. Finally I'll be able to find a photo of someone I know I have. Full screen album display feature is also very nice.

So, now I have a warm fuzzy feeling of having done a useful weekend programming project. Tried something new. Learned a bit. Now it will be great to see whether other people will find it useful.

Sometimes Google is great

Last weekend we have been invited to a neighbour's 5th birthday party. It has taken place at a kids' entertainment park in a shopping centre. I took my camera, and the neighbours asked me to share the pictures.

So, I decided I want to post the pictures at PicasaWeb. It's web galleries are quite convenient and featureful, look like a good place to dump large amounts of photos someone wants you to share. Facebook, in contrast, scales them down to obscenely small sizes. Since I already have a Google account, I just had to click the Register button, and I got into my gallery with my Blogger avatar already in it. I created a new album, but the upload form for adding pictures was so 1990's. It had 5 file upload input fields. Even if new ones would appear as needed, it meant that I would have to select each and every photo individually.

I started looking for a tool for automatic upload into Picasa, but a stand-alone tool for that doesn't seem to exist in Ubuntu. There is, however, a python-gdata package with a Python API to Google online services. I googled "python gdata" and got the documentation in Google Code as the first hit. It has a section on PicasaWeb. 15 minutes later the photos were already on the way. Here is the script:

#!/usr/bin/env python
"""upicasa.py -- an upload script for PicasaWeb"""
import sys
import gdata.photos.service
import gdata.media
import gdata.geo

EMAIL = 'xxxx.xxxx@gmail.com'
PASSWORD = 'XXXXX'

def main():
    gd_client = gdata.photos.service.PhotosService()
    gd_client.email = EMAIL
    gd_client.password = PASSWORD
    gd_client.source = 'alga-upicasa-1'
    print "Authenticating..."
    gd_client.ProgrammaticLogin()

    ## I used this to fish out the album id pasted below
    #albums = gd_client.GetUserFeed(user=EMAIL)
    #for album in albums.entry:
    #    print 'title: %s, number of photos: %s, id: %s' % (album.title.text,
    #                                                       album.numphotos.text,
    #                                                       album.gphoto_id.text)
    #return

    album_id = '123123123123132123'
    album_url = '/data/feed/api/user/%s/albumid/%s' % (EMAIL, album_id)
    for photo in sys.argv[1:]:
        print "uploading", photo
        photo = gd_client.InsertPhotoSimple(album_url, 'New Photo',
                                            '', # title
                                            photo, content_type='image/jpeg')


if __name__ == '__main__':
    main()

It was really fun and empowering to get the photos programmatically uploading into online albums in such a short time. Top marks to Google for providing this simple, sensible way to manage data on Google. No app registration, no developer keys, no nonsense, just the Google login and password are needed to get started. I think I'll develop this script a bit to allow for managing the albums from command line and uploading photos into them, then add it to Python Package Index. Perhaps a simple GUI could also be useful.

The Future

I've touched it. Nokia N900 is awsome. The shape and size is that of a large phone, like the early Symbian smartphones. But it is very slick and is packed with features. It has a slide-out keyboard, a camera with a sliding cover, a leg to stand on the table, a Micro USB connector for power and data, a hardware key lock, several hardware buttons, etc.

The software side, Maemo 5, is also awesome. Looks very slick, feels responsive and ergonomic. And polished. But it's an open platform, based on Linux, X11, QT, GTK+, D-Bus. It looks like head-on competition to iPhone. Reminds me that companies the size of Nokia have the resources pull off almost anything. But it's really great that Nokia is doing it right -- in a way that is Open Source, community based, and iterative.

Linux counter

Look what I have:



A five-digit, 11-years-old Linux counter number!

Goodbye, Palm V

Recently, my Palm died on me due to me neglecting to charge it and brought with it more than a year's worth of statistics of my car's fuel useage, petrol prices, and mileage. Surprisingly from today's standpoint, Palm's memory is not flash and has to be powered in order to keep its contents. If the battery discharges, the device returns to the factory state.

I have owned and used a Palm Vx since 2001. For the last couple of years I have been using it for only two applications: storing passwords in Keyring and keeping tabs on my cars' fuel usage with Fuel Log. Both of these programs are GPL licensed. I've migrated passwords to pwsafe on the laptop a couple of months ago. Now that the fuel log is gone, there's nothing binding me to it.
Sounds like a good opportunity to retire the Palm at last.

I will always have fond memories of this little device. A truly portable PDA, less than 1 cm thick, the weight that does not strain a shirt pocket, battery charge good for several weeks of standby time or 20 hours straight of reading of e-books (tried that). The screen is laughably low-res by today's standards (160x160 pixels). Nowadays most mobile phones have a higher resolution screen. However it was very comfortable on the eyes, and the green inverse backlight was very pretty, was very comfortable in darkness, but made the screen pretty much useless in twilight.

Oh, the thousands of pages of books I have read on this thing! It was so convenient. I had Palm with me at all times and could delve into the book I was reading at the time at each opportunity: a long line in the supermarket, in the doctor's lobby, in bed at night, sunbathing at the lake, anywhere! I could look up a rare word by just clicking on it and switching to the dictionary app. I could not agree more with everything John Siracusa has to say about the Palm e-book experience in his excellent article on the history of e-books.

Palm had a great set of PIM applications: Notes, Address book, To Do. These were really excellent and served their purpose very well.

Miraculously, my Palm's Lithium battery is still working, about 9 years from production! Other things have obsoleted the device: my current laptop (Thinkpad X61s) does not have neither RS-232 nor IrDA ports, and these two are the only communication channels on the Palm. The 8 MB of memory were enough 7 years ago to keep several books and an assortment of useful programs, but are laughable today. But even in its heyday, getting online not too comfortable: if there was no table around, you had to balance the Palm and the mobile phone on your knee with their infrared ports aligned, and the 9600 bps dialup on GSM was slow by any standards, and there was no web browser, at least a free software one.

Later offerings from Palm Inc. were faster, brighter, more capable, more powerful, and more expensive, but lost to Palm V in several important respects: size, weight, and battery life.

So, anyway, I need a new fuel log application, and I know the device where I want it. My S60 mobile phone. I also have it with me at all times, it's always on and ready to roll, and it holds its charge for many^H^H^H^Hseveral days. And i'm going to write it in Python.

Hardware problems with OLPC XO-1: my 2 cents

Problem: my kid's XO wouldn't turn on. The power light would go on, but the screen would stay dark. And it wasn't just the backlight. There was no startup sound as well. Powering on while holding D-pad left did start the hardware diagnostics, but the machine turned off in the middle of the test and wouldn't come back up.

On the OLPC wiki I found excellent information on startup diagnosis, but my symptoms weren't there. The next step was to use the excellent disassembly instructions and look for a connection shaken loose or some mechanical damage.

XO is really beautiful inside. Looks very sturdy, also it looks simple and sophisticated at the same time. Mostly everything is in the top part, behind the screen. There are only ~9 wires coming from the bottom: 2 from the battery, another small 2 wire connector (battery temperature sensor or something like that) and a 5-wire keyboard cable. It took me a minute or two to identify the problem.

Solution: remove the 2 cents coin from the SD card slot!

P.S. Also, I checked out the firmware easter egg (power up while holding D-pad right). Nice!

The joys of Linksys WRT 54GL

On Friday there was a thunderstorm. Usually during the storms the surge protection in my radio link antenna's power-over-ethernet power supply trips up and needs to be reset by disconnecting the PSU from the mains for around 10 minutes. This time the usual measure didn't suffice. The lights on the adapter came up, but still there was no connectivity. I reported the problem to the ISP's tech support. The next day the engineer rang up and said that he can connect to the dish at my side.

I connected a laptop directly to the uplink cable, configured the static IP address. Lo and behold, the pings go through. Apparently the uplink Ethernet port on my OpenWRT router got fried.
Luckily, this router does not just have an uplink port and four lan ports, it has a programmable switch and an open firmware (OpenWRT 7.09 "Kamikaze") lets me tap into it.

I looked at /etc/config/network, and found the following lines at the top:

#### VLAN configuration
config switch eth0
option vlan0 "0 1 2 3 5*"
option vlan1 "4 5"


vlan0 is the internal lan, 0-3 are the external lan ports, and 5 is the host port. vlan1 connects the uplink port (4) to the router (5). The asterisk probably means packet tagging. As the router internally has just one physical ethernet port, the internal and external interfaces are virtuals employing the same physical interface. Their packets are distinguished by tags.

So, I changed the config thus:


#### VLAN configuration
config switch eth0
option vlan0 "0 1 2 5*"
option vlan1 "3 4 5"


I rebooted the router and plugged the blue uplink cable into the first LAN port (the numbering of the holes on the router is backwards). Tada! Here's my Internet connection again!

Huawei E220 USB HSDPA modem on Linux

Tested on Ubuntu Hardy. When you plug the modem in, things happen:


[ 1872.829603] usb 5-2: new full speed USB device using uhci_hcd and address 11
[ 1872.988741] usb 5-2: configuration #1 chosen from 1 choice
[ 1872.998189] usb-storage: probe of 5-2:1.0 failed with error -5
[ 1872.998226] airprime 5-2:1.0: GSM modem (1-port) converter detected
[ 1872.998441] usb 5-2: GSM modem (1-port) converter now attached to ttyUSB0
[ 1873.006403] usb-storage: probe of 5-2:1.1 failed with error -5
[ 1873.006437] airprime 5-2:1.1: GSM modem (1-port) converter detected
[ 1873.006615] usb 5-2: GSM modem (1-port) converter now attached to ttyUSB1
[ 1873.057365] scsi30 : SCSI emulation for USB Mass Storage devices
[ 1873.058803] usb-storage: device found at 11
[ 1873.058810] usb-storage: waiting for device to settle before scanning
[ 1876.078727] usb-storage: device scan complete
[ 1876.082294] scsi 30:0:0:0: CD-ROM HUAWEI Mass Storage 2.31 PQ: 0 ANSI: 2
[ 1876.117573] sr0: scsi-1 drive
[ 1876.117674] sr 30:0:0:0: Attached scsi CD-ROM sr0
[ 1876.117754] sr 30:0:0:0: Attached scsi generic sg1 type 5


As you can see, we get two serial ports and a CD-ROM drive. The second serial port and the CD-ROM drive can be ignored. PPP works straight away with the following config:


noauth
connect "/usr/sbin/chat -v -f /etc/chatscripts/e220"
debug
/dev/ttyUSB0
230400
crtscts
defaultroute
noipdefault
user ignored
remotename whatever
ipparam whatever
usepeerdns


Here's the e220 chatscript (put in into /etc/chatscripts, replace the "****" with your pin code, replace "omnitel" with your provider's APN):


# Chat file for Huawei E220 HSDPA USB modem
ABORT BUSY ABORT 'NO CARRIER' ABORT 'NO ANSWER' ABORT DELAYED
'' AT
OK AT+CPIN="****"
OK ATX3
OK 'AT+CGDCONT=1,"IP","omnitel"'
OK ATDT*99***1#
CONNECT ""


New phone

I got a new phone on Thursday, Nokia 6120 classic. It is my first Series 60 phone. I have been a loyal Nokia user for around four years now, mostly because of the ubiquity and universality of Nokia chargers. During the whole time they were producing GSM handsets, Nokia has changed the charger connector only two times. Each time the change was motivated by miniaturization.

Series 60 3rd edition is very nice. The UI metaphor is different from Series 40, but it's quite intuitive and comfortable. Surprisingly, there's no stopwatch app, but of course there already are too many Stopwatch apps for S60 on the web. Should I write another one? :-) There's a snag that this phone is not yet supported by Gnokii/Gammu/Wammu, so I had to install the Nokia PC suite on Windows in order to transfer my contacts from my old Nokia 6230i. Know what? It was not flawless. Getting the contacts from the 6230i was easy (if we ignore the constant Vista disk grinding), not so with restoring the backup on the 6120c. It took Nokia PC suite 3 attempts to create the Bluetooth pair, install the drivers on the PC and on the phone, and connect to the phone.

The phone has all the 3G technologies: UMTS, HSDPA. During my first Internet connection test on a late Thursday evening speedtest.net indicated 1.2 Mbps down and about 100 kbps up with a 3.5G (HSDPA) connection. On a Saturday night though the speed fluctuated between 300-600 kbps on average, and it took a couple of attempts to establish an HSDPA connection, I would get EDGE instead. I'm using the Bitė GMS network. We'll see how things will go with Omnitel.

Yesterday I installed Python on the phone. I set up and played with a Bluetooth console. The Hardy bluez-utils are broken (see Ubuntu bug 211525), so I had to install bluez-utils from Sid. The Bluetooth console is much more convenient than banging in Python code on the phone keyboard, but it does not have readline.

I got an impression that Nokia wants people to use Python on Series 60 very much. They try very hard to make it easy, provide all sorts of docs, even multimedia tutorials. Probably they see the ease and accessibility of Python as a path to widespread platform adoption. The Python API has easy access to most of the phone's functionality, including the GUI toolkit, camera, bluetooth, networking, messaging, contacts and so on. I have an impression that cooking a homebrew contacts backup/restore application in Python will be pretty trivial. I cannot start hacking away as Python looks for modules on the flash card, and I haven't bought a microSD card yet. On a positive note, it's nice that this time Nokia didn't include a cheap claustrophobic-sized card. They're useless after you buy a real card, but hard to throw away, so they contribute to cruft accumulation in life.

Unfortunately, Nokia only provides the platform SDK for Windows, but there are ways to use the GNU toolchain in combination with the SDK and some Wine.
The SDK is needed in order to package Python scripts as SIS installable files. Even then, I'm not sure it will work, as Series 60 3rd ed. only accepts packages signed by a registered developer. There should to be a way to play with your own software, no idea yet what's involved.