Why does Firefox use Sqlite?

Apparently the latest version of Firefox has a problem with Linux: http://shaver.off.net/ explains the details, but the short of it is that Firefox 3 uses Sqlite for bookmarks and history, Sqlite in turn uses fsync() to ensure data integrity, and Linux fsync writes ALL disk buffers when fsync() is called and of course that can lead to long, long pauses where your computer is completely non-responsive. The issue will be fixed; again, that post has the details, but I have a more basic question:

Why on earth does Firefox need a database for bookmarks and history?

Am I nuts? Am I the only person who can't imagine who Joe Firefox User could possibly have enough bookmarks or browsing history that you need a database to manage the data?

Sheesh: I have a lot of bookmarks - too many, in fact and I need to clean them out. I have plenty of browsing history, too, but it's simply inconceivable to me that anyone would think they'd need anything but simple text files for this. I've written about this before at Text vs. Binary Data formats and I revisited that today with yet another test. This time I created a million line text file with random lines of 47 characters. That's 47 million bytes - I would thank that's more than enough to store my bookmarks and history. The code I used was just this:

#!/usr/bin/perl
$x=0;
while ($x < 1000000) {
 $y=rand(100000);
 printf "%0.6d Padding representing stored data %0.6d\n", $y,$x;
 $x++; 
}

I redirected that to a file and then searched it using the same horribly inefficient code I used the last time I griped about this foolishness:

#!/usr/bin/perl
$found=0;
$now=time();
$fnow=$now;
while (@ARGV) {
 $found=0;
 $x=shift @ARGV;
 open(O, "foop");
 while (<O>) {
   if (/^$x / ) {
     $found++;
   }
 }
$now=time();
$elapsed=$now - $fnow;
print "Found $found matches in $elapsed seconds \n"; 
}

Remember: that's a horrible search. We're not presorting, we haven't built any sort of auxiliary index to help us out, and we're using an interpreted language and not even optimizing that. Yet that silly code can find anything you want in that 47 million byte file in less than a second. We could cut that time by 30% just by changing "if (/^$x / ) { " to "if (/^$x /o ) { ", but my point here is not to write better Perl code - it's to show that flat text access is NOT slow on modern hardware. Heck, plain old "grep" can zip through that file like lightning:

$ time grep ^017581 foop
017581 Padding representing stored data 301404
017581 Padding representing stored data 346784
017581 Padding representing stored data 366993
017581 Padding representing stored data 402525
017581 Padding representing stored data 453018
017581 Padding representing stored data 510494
017581 Padding representing stored data 562309
017581 Padding representing stored data 705912
017581 Padding representing stored data 838930
017581 Padding representing stored data 888413

real 0m0.105s
user 0m0.047s
sys 0m0.036s

The Firefox developers seem rather adamant that they do need a database - in response to one user who questioned this as I do, they responded:

Browser history needs to be pretty fast, since it's consulted for link colouring and CSS rule values; we have to be very careful whenever we touch history because it can easily lead to performance regressions. You probably know that from your own work on browsers. The speed of even just URL autocomplete in FF2 (no multi-word, no adaptive, no title searches, no tags, no bookmark inclusion) was a common source of complaints for users with large histories, and we are very glad to be rid of it. I love that you think that bookmarks are a bigger performance issue than browsing history; just the smile I needed today!

OK, maybe I'm wrong. Maybe Joe Average Firefox User has millions of bookmarks and keeps his history for years on end. Maybe. That's hard for me to imagine, though.. I think this might be just yet another case of people not really thinking about the tools they need to use. By the way, I DETEST autocomplete..

Yet they insist:

Browser history, as used by Firefox 3, is exactly a large set of data that needs to be heavily indexed and queried in a number of different ways. Earlier designers weren't doing as much with history, and users didn't have as much of it; likewise with bookmarks.

Well, OK.. but then again, does history really NEED to be indexed six ways from Sunday? I don't need that, but if most users want it, and really do keep their history so long that Firefox needs a database.. well, there it is, right?

I'd rather use Google.

Comments /Web/firefox-sqlite.html


Sat May 31 07:06:01 2008: Subject:   drag


I'd rather just use epiphany with webkit support. :)
I've always believed that web browsers should just be web browsers. If you want some advanced functionality then it should be done as a extension to your browser so that other people who don't want/need that feature don't have to deal with the decrease in performance and security that is a side effect of code bloat.



Sun Jul 6 11:59:19 2008: Subject: Full ack   anonymous


Congratulations for the harsh critique on this sqlite stuff in FF3. I just stumbled over it because I am keeping all bookmarks in a plain html file in a SVN repository (as well as the rest of the FF config) because I am using serval different machines and want the same config on all of them.

Binary formats for configuations are a misconcept -- it is as plain as that!

Michael,
who *never* used the FF history

Add your comments

Why no "Digg this!" etc.?

Unix and Linux Support

Enter your email address for automatic notification of new posts here
(be sure to whitelist 'feedburner.com' if you use spam filtering)

17413-080530-912710-88

Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more. We appreciate comments and article submissions.

Publishing your articles here