Why does Firefox use Sqlite?

Apparently the latest version of Firefox has a problem with Linux: http://shaver.off.net/ explains the details, but the short of it is that Firefox 3 uses Sqlite for bookmarks and history, Sqlite in turn uses fsync() to ensure data integrity, and Linux fsync writes ALL disk buffers when fsync() is called and of course that can lead to long, long pauses where your computer is completely non-responsive. The issue will be fixed; again, that post has the details, but I have a more basic question:

Why on earth does Firefox need a database for bookmarks and history?

Am I nuts? Am I the only person who can't imagine why Joe Firefox User could possibly have enough bookmarks or browsing history that you need a database to manage the data?

Sheesh: I have a lot of bookmarks - too many, in fact and I need to clean them out. I have plenty of browsing history, too, but it's simply inconceivable to me that anyone would think they'd need anything but simple text files for this. I've written about this before at Text vs. Binary Data formats and I revisited that today with yet another test. This time I created a million line text file with random lines of 47 characters. That's 47 million bytes - I would thank that's more than enough to store my bookmarks and history. The code I used was just this:

#!/usr/bin/perl
$x=0;
while ($x < 1000000) {
 $y=rand(100000);
 printf "%0.6d Padding representing stored data %0.6d\n", $y,$x;
 $x++; 
}

I redirected that to a file and then searched it using the same horribly inefficient code I used the last time I griped about this foolishness:

#!/usr/bin/perl
$found=0;
$now=time();
$fnow=$now;
while (@ARGV) {
 $found=0;
 $x=shift @ARGV;
 open(O, "foop");
 while (<O>) {
   if (/^$x / ) {
     $found++;
   }
 }
$now=time();
$elapsed=$now - $fnow;
print "Found $found matches in $elapsed seconds \n"; 
}

Remember: that's a horrible search. We're not presorting, we haven't built any sort of auxiliary index to help us out, and we're using an interpreted language and not even optimizing that. Yet that silly code can find anything you want in that 47 million byte file in less than a second. We could cut that time by 30% just by changing "if (/^$x / ) { " to "if (/^$x /o ) { ", but my point here is not to write better Perl code - it's to show that flat text access is NOT slow on modern hardware. Heck, plain old "grep" can zip through that file like lightning:

$ time grep ^017581 foop
017581 Padding representing stored data 301404
017581 Padding representing stored data 346784
017581 Padding representing stored data 366993
017581 Padding representing stored data 402525
017581 Padding representing stored data 453018
017581 Padding representing stored data 510494
017581 Padding representing stored data 562309
017581 Padding representing stored data 705912
017581 Padding representing stored data 838930
017581 Padding representing stored data 888413

real 0m0.105s
user 0m0.047s
sys 0m0.036s

The Firefox developers seem rather adamant that they do need a database - in response to one user who questioned this as I do, they responded:

Browser history needs to be pretty fast, since it's consulted for link colouring and CSS rule values; we have to be very careful whenever we touch history because it can easily lead to performance regressions. You probably know that from your own work on browsers. The speed of even just URL autocomplete in FF2 (no multi-word, no adaptive, no title searches, no tags, no bookmark inclusion) was a common source of complaints for users with large histories, and we are very glad to be rid of it. I love that you think that bookmarks are a bigger performance issue than browsing history; just the smile I needed today!

OK, maybe I'm wrong. Maybe Joe Average Firefox User has millions of bookmarks and keeps his history for years on end. Maybe. That's hard for me to imagine, though.. I think this might be just yet another case of people not really thinking about the tools they need to use. By the way, I DETEST autocomplete..

Yet they insist:

Browser history, as used by Firefox 3, is exactly a large set of data that needs to be heavily indexed and queried in a number of different ways. Earlier designers weren't doing as much with history, and users didn't have as much of it; likewise with bookmarks.

Well, OK.. but then again, does history really NEED to be indexed six ways from Sunday? I don't need that, but if most users want it, and really do keep their history so long that Firefox needs a database.. well, there it is, right?

I'd rather use Google.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Comments




Sat May 31 07:06:00 2008: Subject:   drag


I'd rather just use epiphany with webkit support. :)
I've always believed that web browsers should just be web browsers. If you want some advanced functionality then it should be done as a extension to your browser so that other people who don't want/need that feature don't have to deal with the decrease in performance and security that is a side effect of code bloat.





Sun Jul 6 11:59:19 2008: Subject: Full ack   anonymous


Congratulations for the harsh critique on this sqlite stuff in FF3. I just stumbled over it because I am keeping all bookmarks in a plain html file in a SVN repository (as well as the rest of the FF config) because I am using serval different machines and want the same config on all of them.

Binary formats for configuations are a misconcept -- it is as plain as that!

Michael,
who *never* used the FF history



Mon Oct 27 16:57:11 2008: Subject:   anonymous


I totally agree with your post. Moreover, there should be an option to use databases or text files. Additionally, bookmarks should be independent of other features such as history, etc. Many users like Firefox because they can be in control of basic features, files, etc; this does not follow the trend and the essential and basic features get hidden behind complex engines which are alien to the average user. The bridge between the browser and the user is getting destroyed as new versions come.

This reminds me of the registry appearance under NT. While it is justifiable for an OS, it is not for for a browser.

Marty



Tue Feb 17 22:08:14 2009: Subject: Agreed.   Lns
http://logicalnetworking.net

I can see both sides of the table, but it frustrates me that, if the FF devs went so far as to implement ANY sql backend for things like history/bookmarks/homepage/etc., that they didn't make it PORTABLE toward other SQL backends. Sure, sqlite is serverless, very fast, runs on cellphones, blah blah blah...but what about large network installations? I administrate a few LTSP networks, where all users live on a single server. I would *LOVE* to be able to globally manipulate users' bookmarks, homepages, etc. in my favorite SQL db editor. This is a huge benefit, but alas...sqlite is set to 'exclusive' mode in FF, thus can't be touched by anything else while it's in use. This makes it in no way useful for multi-user installations. In addition, I was at LEAST able to give people global bookmarks via a hack-ish script that copied bookmarks.html into users' ~/.mozilla profile dirs, but since sqlite took THAT over too, it became obsolete. The SQLITE decision hasn't done much but limit my own administration abilities in multi-user environments. It'd be nice to separate the SQL from SQLITE and allow choice for different setups.



Tue Feb 17 22:16:08 2009: Subject:   TonyLawrence


Ayup - it's always dumb to lock up data that could be useful elsewhere. I suppose users might protest that they don't WANT their bookmarks made available, but that's not a decision developers should make.





Sat Mar 7 14:51:02 2009: Subject: I strongly disagree   henri


The only argument I can understand is that old style bookmarks made it easy for an admin to push bookmarks onto his/her userbase (but I'm pretty sure there still is a way with sqlite.)
Myself, like many others, stopped using the bookmarks menus and use the awesome bar, stopped storing bookmarks into folders (or only do it in rare cases) and instead use tags.

My typical way of browsing would be to type a few keywords in the url bar and scan the results until the page I'm looking for appears (these results come from both the history and the tagged bookmarks). To work smoothly this has to be FAST. This I do maybe 100 times a day, and there is absolutely no way suitable speed could be obtained without a really efficient way to scan the bookmarks db. This could have been done while still retaining the bookmarks.html, but I just don't see the point in practice.

To Michael: If you *never* used your history, then I'm pretty sure you are not using your browser in the most efficient manner :) After a few days I just got rid of my bookmarks toolbar and forgot about it. Read about the awesome bar.




Sat Mar 7 16:09:02 2009: Subject:   TonyLawrence


"there is absolutely no way suitable speed could be obtained without a really efficient way to scan the bookmarks"

That's the point. A database is NOT an efficient way to scan small data sets.

---

---

Sun Apr 12 18:55:59 2009: Subject: Clarity and maintainability suffers....   StephenBungay


I ran full speed into this idiocy today and Firefox is NO LONGER my browser and I will NOT be recommending it in future.

Yes, one CAN use a database for bookmarks, the question is SHOULD it be done, and there better be a damn good reason to justify the need to replace a simple effective and eaisly maintainable system with a complex, effective, not eaisly maintainable system.

Never use a complex system where a simple system will do the job.




Sun Apr 12 21:02:51 2009: Subject:   TonyLawrence


The more often this stupid "awesome bar" tries sending me to places I don't want to go, the more I hate it.

If I start typing "bar", I want to go somewhere that BEGINS with those letters, not somewhere that has them somewhere in the middle or end. I *definitely* don't want a page that has "bar" in its title.

The "awesome bar" is the "I'm an idiot" bar.

I'm not ready to quit Firefox yet, but it definitely is ticking me off. If Safari or Chrome could restore my sessions, I might just leave..



Sat Apr 18 15:30:42 2009: Subject:   TonyLawrence


OK, I've had it with this stupid Awful Bar.

Disable it with about:config and set browser.urlbar.maxRichResults to 0

This morning I wanted to go to Macosxhints.com

Of course it immediately brings up the hundreds of other times I've been to internal pages there, when all I want is just the plain home page. So I keep typing but then I accidentally hit a bad letter: "macosxho" instead of "macosxhi" - and of course THAT sends the Stupid Bar off searching for something that has "macosxho" in it which locks me up for a few seconds until the idiot thing gives up and I can back space.

Disabled. Moron idea. "Nuff said.



Fri May 1 11:33:00 2009: Subject:   TonyLawrence


I found that a happier setting is to put browser.urlbar.maxRichResults at 6. That's been enough to find what I want easily without slowing down searching through thousands of matches.

But Firefox woes continue. I run no addons (even took out NoScript), no extra toolbars, yet it still freezes now and then. It doesn't freeze as often, and it's apparent that NoScript was the worst offender. But yesterday it froze and then would not start, not with "Previous session" or new. I threw away Preferences, no luck. I tried starting it with " -safe-mode" - still would not start.

I threw it in the trash, d/l'd a new copy wth Safari and it was fine.

Firefox on Mac definitely still has issues.



Sun Jun 14 08:59:17 2009: Subject:   dgutson


Firefox now competes with an ATM machine. It's not a banking system. It's just a BROWSER.

I'm seriously considering to develop a spin off without sqlite. There was a --no-places or similar argument for the ./configure, but it was removed.


Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar