APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed
RSS Feeds RSS Feeds











(OLDER) <- More Stuff -> (NEWER) (NEWEST)
Printer Friendly Version
->
-> Why does Firefox use Sqlite?


Why does Firefox use Sqlite?




Apparently the latest version of Firefox has a problem with Linux: http://shaver.off.net/ explains the details, but the short of it is that Firefox 3 uses Sqlite for bookmarks and history, Sqlite in turn uses fsync() to ensure data integrity, and Linux fsync writes ALL disk buffers when fsync() is called and of course that can lead to long, long pauses where your computer is completely non-responsive. The issue will be fixed; again, that post has the details, but I have a more basic question:

Because more than one person has misunderstood, I need to stress this: I don't care about the fsync() issue. Things like that can be fixed or worked around. I only mentioned it because it exposed the use of Sqlite.

Why on earth does Firefox need a database for bookmarks and history?

Am I nuts? Am I the only person who can't imagine why Joe Firefox User could possibly have enough bookmarks or browsing history that you need a database to manage the data?

Again because of poor reading skills by some commentors, I need to stress that I'm not trashing Sqlite. I'm not trashing ANY database. I'm complaining about a specific case.

Sheesh: I have a lot of bookmarks - too many, in fact and I need to clean them out. I have plenty of browsing history, too, but it's simply inconceivable to me that anyone would think they'd need anything but simple text files for this. I've written about this before at Text vs. Binary Data formats and I revisited that today with yet another test. This time I created a million line text file with random lines of 47 characters. That's 47 million bytes - I would thank that's more than enough to store my bookmarks and history. The code I used was just this:

#!/usr/bin/perl
$x=0;
while ($x < 1000000) {
 $y=rand(100000);
 printf "%0.6d Padding representing stored data %0.6d\n", $y,$x;
 $x++; 
}
 

I redirected that to a file and then searched it using the same horribly inefficient code I used the last time I griped about this foolishness:

#!/usr/bin/perl
$found=0;
$now=time();
$fnow=$now;
while (@ARGV) {
 $found=0;
 $x=shift @ARGV;
 open(O, "foop");
 while (<O>) {
   if (/^$x / ) {
     $found++;
   }
 }
$now=time();
$elapsed=$now - $fnow;
print "Found $found matches in $elapsed seconds \n"; 
}
 

Remember: that's a horrible search. We're not presorting, we haven't built any sort of auxiliary index to help us out, and we're using an interpreted language and not even optimizing that. Yet that silly code can find anything you want in that 47 million byte file in less than a second. We could cut that time by 30% just by changing "if (/^$x / ) { " to "if (/^$x /o ) { ", but my point here is not to write better Perl code - it's to show that flat text access is NOT slow on modern hardware. Heck, plain old "grep" can zip through that file like lightning:

$ time grep ^017581 foop
017581 Padding representing stored data 301404
017581 Padding representing stored data 346784
017581 Padding representing stored data 366993
017581 Padding representing stored data 402525
017581 Padding representing stored data 453018
017581 Padding representing stored data 510494
017581 Padding representing stored data 562309
017581 Padding representing stored data 705912
017581 Padding representing stored data 838930
017581 Padding representing stored data 888413

real	0m0.105s
user	0m0.047s
sys	0m0.036s
 

The Firefox developers seem rather adamant that they do need a database - in response to one user who questioned this as I do, they responded:

Browser history needs to be pretty fast, since it's consulted for link colouring and CSS rule values; we have to be very careful whenever we touch history because it can easily lead to performance regressions. You probably know that from your own work on browsers. The speed of even just URL autocomplete in FF2 (no multi-word, no adaptive, no title searches, no tags, no bookmark inclusion) was a common source of complaints for users with large histories, and we are very glad to be rid of it. I love that you think that bookmarks are a bigger performance issue than browsing history; just the smile I needed today!

OK, maybe I'm wrong. Maybe Joe Average Firefox User has millions of bookmarks and keeps his history for years on end. Maybe. That's hard for me to imagine, though.. I think this might be just yet another case of people not really thinking about the tools they need to use. By the way, I DETEST autocomplete..

Yet they insist:

Browser history, as used by Firefox 3, is exactly a large set of data that needs to be heavily indexed and queried in a number of different ways. Earlier designers weren't doing as much with history, and users didn't have as much of it; likewise with bookmarks.

Well, OK.. but then again, does history really NEED to be indexed six ways from Sunday? I don't need that, but if most users want it, and really do keep their history so long that Firefox needs a database.. well, there it is, right?

I'd rather use Google.

This could be helpful: Vacuum Firefox databases for better performance, now with no restart.

That suggests opening Tools-> Error Consiole and typing:

Components.classes["@mozilla.org/browser/nav-history-service;
1"].getService(Components.interfaces.nsPIPlacesDatabase).
DBConnection.executeSimpleSQL("VACUUM");
 

He notes:

Press Evaluate. All the UI will freeze for a few seconds while databases are VACUUMed.

Note however that the procedure optimizes the Places database only, but this is precisely where you will get the most significant performance improvements.

Yeah - because that stuff never should have been put in a database to start with!




If this page was useful to you, please help others find it:  





100 comments




More Articles by - Find me on Google+



Click here to add your comments
- no registration needed!




Sat May 31 07:06:00 2008: 4272   drag


I'd rather just use epiphany with webkit support. :)
I've always believed that web browsers should just be web browsers. If you want some advanced functionality then it should be done as a extension to your browser so that other people who don't want/need that feature don't have to deal with the decrease in performance and security that is a side effect of code bloat.







Sun Jul 6 11:59:19 2008: 4395   anonymous


Congratulations for the harsh critique on this sqlite stuff in FF3. I just stumbled over it because I am keeping all bookmarks in a plain html file in a SVN repository (as well as the rest of the FF config) because I am using serval different machines and want the same config on all of them.

Binary formats for configuations are a misconcept -- it is as plain as that!

Michael,
who *never* used the FF history



Mon Oct 27 16:57:11 2008: 4695   anonymous


I totally agree with your post. Moreover, there should be an option to use databases or text files. Additionally, bookmarks should be independent of other features such as history, etc. Many users like Firefox because they can be in control of basic features, files, etc; this does not follow the trend and the essential and basic features get hidden behind complex engines which are alien to the average user. The bridge between the browser and the user is getting destroyed as new versions come.

This reminds me of the registry appearance under NT. While it is justifiable for an OS, it is not for for a browser.

Marty



Tue Feb 17 22:08:14 2009: 5459   Lns
http://logicalnetworking.net
gravatar
I can see both sides of the table, but it frustrates me that, if the FF devs went so far as to implement ANY sql backend for things like history/bookmarks/homepage/etc., that they didn't make it PORTABLE toward other SQL backends. Sure, sqlite is serverless, very fast, runs on cellphones, blah blah blah...but what about large network installations? I administrate a few LTSP networks, where all users live on a single server. I would *LOVE* to be able to globally manipulate users' bookmarks, homepages, etc. in my favorite SQL db editor. This is a huge benefit, but alas...sqlite is set to 'exclusive' mode in FF, thus can't be touched by anything else while it's in use. This makes it in no way useful for multi-user installations. In addition, I was at LEAST able to give people global bookmarks via a hack-ish script that copied bookmarks.html into users' ~/.mozilla profile dirs, but since sqlite took THAT over too, it became obsolete. The SQLITE decision hasn't done much but limit my own administration abilities in multi-user environments. It'd be nice to separate the SQL from SQLITE and allow choice for different setups.



Tue Feb 17 22:16:08 2009: 5460   TonyLawrence

gravatar
Ayup - it's always dumb to lock up data that could be useful elsewhere. I suppose users might protest that they don't WANT their bookmarks made available, but that's not a decision developers should make.







Sat Mar 7 14:51:02 2009: 5624   henri

gravatar
The only argument I can understand is that old style bookmarks made it easy for an admin to push bookmarks onto his/her userbase (but I'm pretty sure there still is a way with sqlite.)
Myself, like many others, stopped using the bookmarks menus and use the awesome bar, stopped storing bookmarks into folders (or only do it in rare cases) and instead use tags.

My typical way of browsing would be to type a few keywords in the url bar and scan the results until the page I'm looking for appears (these results come from both the history and the tagged bookmarks). To work smoothly this has to be FAST. This I do maybe 100 times a day, and there is absolutely no way suitable speed could be obtained without a really efficient way to scan the bookmarks db. This could have been done while still retaining the bookmarks.html, but I just don't see the point in practice.

To Michael: If you *never* used your history, then I'm pretty sure you are not using your browser in the most efficient manner :) After a few days I just got rid of my bookmarks toolbar and forgot about it. Read about the awesome bar.






Sat Mar 7 16:09:02 2009: 5625   TonyLawrence

gravatar
"there is absolutely no way suitable speed could be obtained without a really efficient way to scan the bookmarks"

That's the point. A database is NOT an efficient way to scan small data sets.



Sun Apr 12 18:55:59 2009: 6140   StephenBungay

gravatar
I ran full speed into this idiocy today and Firefox is NO LONGER my browser and I will NOT be recommending it in future.

Yes, one CAN use a database for bookmarks, the question is SHOULD it be done, and there better be a damn good reason to justify the need to replace a simple effective and eaisly maintainable system with a complex, effective, not eaisly maintainable system.

Never use a complex system where a simple system will do the job.






Sun Apr 12 21:02:51 2009: 6143   TonyLawrence

gravatar
The more often this stupid "awesome bar" tries sending me to places I don't want to go, the more I hate it.

If I start typing "bar", I want to go somewhere that BEGINS with those letters, not somewhere that has them somewhere in the middle or end. I *definitely* don't want a page that has "bar" in its title.

The "awesome bar" is the "I'm an idiot" bar.

I'm not ready to quit Firefox yet, but it definitely is ticking me off. If Safari or Chrome could restore my sessions, I might just leave..





Sat Apr 18 15:30:42 2009: 6221   TonyLawrence

gravatar
OK, I've had it with this stupid Awful Bar.

Disable it with about:config and set browser.urlbar.maxRichResults to 0

This morning I wanted to go to Macosxhints.com

Of course it immediately brings up the hundreds of other times I've been to internal pages there, when all I want is just the plain home page. So I keep typing but then I accidentally hit a bad letter: "macosxho" instead of "macosxhi" - and of course THAT sends the Stupid Bar off searching for something that has "macosxho" in it which locks me up for a few seconds until the idiot thing gives up and I can back space.

Disabled. Moron idea. "Nuff said.



Fri May 1 11:33:00 2009: 6298   TonyLawrence

gravatar
I found that a happier setting is to put browser.urlbar.maxRichResults at 6. That's been enough to find what I want easily without slowing down searching through thousands of matches.

But Firefox woes continue. I run no addons (even took out NoScript), no extra toolbars, yet it still freezes now and then. It doesn't freeze as often, and it's apparent that NoScript was the worst offender. But yesterday it froze and then would not start, not with "Previous session" or new. I threw away Preferences, no luck. I tried starting it with " -safe-mode" - still would not start.

I threw it in the trash, d/l'd a new copy wth Safari and it was fine.

Firefox on Mac definitely still has issues.



Sun Jun 14 08:59:17 2009: 6501   dgutson

gravatar
Firefox now competes with an ATM machine. It's not a banking system. It's just a BROWSER.

I'm seriously considering to develop a spin off without sqlite. There was a --no-places or similar argument for the ./configure, but it was removed.






Sat Jul 18 11:54:50 2009: 6643   ChristianSmith

gravatar
SQLite is not designed to be a heavy database engine, it's more of an extensible file format implementation, that happens to have a SQL interface.

I only wish they'd move the cache, as well as browser history and bookmarks to the database file. That way, all but the current displayed pages can be kept in the database file, reducing the Firefox heap footprint massively (I assume firefox's footprint is so big because of caching.)

Having a SQL interface makes so many things so much easier. Want to search the history using some other variable. Just use a different WHERE clause. No manual coding of searches, sorts, indexing etc. Makes the code more robust, and SQLite keeps the data in the file consistent.



Thu Oct 1 04:16:57 2009: 7026   anonymous

gravatar
You guys are too trusting. Firefox stores your information in a database so its primary customers like Google and Amazon can access it. Like Google, FF keeps every bit of data about where you have been. Thats why it needs a a database. I know this because despite the fact that I have the privacy settings all checked to erase everything, Amazon accessed the sqlite files and knew who I was after I closed down FF and then opened it up again. No matter what your privacy settings are, FF keeps the your information in the sqlite files and no doubt the information there is a source of funding for Mozilla. My places.sqlite file is 330K despite the fact I have the "erase history" privacy box checked. Rather sneaky because you think you have erased all your browsing data but you haven't. But then, they are just Google's lackey boys, now aren't they?



Thu Oct 1 11:06:30 2009: 7027   TonyLawrence

gravatar
You are misunderstanding completely.

Amazon knew who you were because of a cookie - and they only knew because you told them in the first place.







Wed Oct 7 23:41:32 2009: 7115   anonymous

gravatar
You missed the point entirely.

I had all the privacy settings on including "erase cookies." Additionally, when Amazon recognized me when it shouldn't have (because Firefox should have erased all the cookies), I checked. All the cookies were in fact erased from the normal cookie file we all know and love. There was a cookie alright, but it was buried in the sqlite file where most users would not know how to find it.

The point is that Firefox leads you to believe you are erasing cookies when in fact some secret cookies for special friends of Mozilla are buried where you can't see them or erase them (unless your one of the very few people that know where to find them)

Do you understand now?



Wed Oct 7 23:45:49 2009: 7116   TonyLawrence

gravatar
OK, that's different - and more disturbing.



Thu Oct 22 03:10:20 2009: 7333   intuited

gravatar
I'm no expert but I'm guessing from the complexity of the places.sqlite schema set:
 
$ sqlite3 places.sqlite .schema
CREATE TABLE moz_anno_attributes (id INTEGER PRIMARY KEY,name VARCHAR(32) UNIQUE NOT NULL);
CREATE TABLE moz_annos (id INTEGER PRIMARY KEY,place_id INTEGER NOT NULL,anno_attribute_id INTEGER,mime_type VARCHAR(32) DEFAULT NULL,content LONGVARCHAR, flags INTEGER DEFAULT 0,expiration INTEGER DEFAULT 0,type INTEGER DEFAULT 0,dateAdded INTEGER DEFAULT 0,lastModified INTEGER DEFAULT 0);
CREATE TABLE moz_bookmarks (id INTEGER PRIMARY KEY,type INTEGER, fk INTEGER DEFAULT NULL, parent INTEGER, position INTEGER, title LONGVARCHAR, keyword_id INTEGER, folder_type TEXT, dateAdded INTEGER, lastModified INTEGER);
CREATE TABLE moz_bookmarks_roots (root_name VARCHAR(16) UNIQUE, folder_id INTEGER);
CREATE TABLE moz_favicons (id INTEGER PRIMARY KEY, url LONGVARCHAR UNIQUE, data BLOB, mime_type VARCHAR(32), expiration LONG);
CREATE TABLE moz_historyvisits (id INTEGER PRIMARY KEY, from_visit INTEGER, place_id INTEGER, visit_date INTEGER, visit_type INTEGER, session INTEGER);
CREATE TABLE moz_inputhistory (place_id INTEGER NOT NULL, input LONGVARCHAR NOT NULL, use_count INTEGER, PRIMARY KEY (place_id, input));
CREATE TABLE moz_items_annos (id INTEGER PRIMARY KEY,item_id INTEGER NOT NULL,anno_attribute_id INTEGER,mime_type VARCHAR(32) DEFAULT NULL,content LONGVARCHAR, flags INTEGER DEFAULT 0,expiration INTEGER DEFAULT 0,type INTEGER DEFAULT 0,dateAdded INTEGER DEFAULT 0,lastModified INTEGER DEFAULT 0);
CREATE TABLE moz_keywords (id INTEGER PRIMARY KEY AUTOINCREMENT, keyword TEXT UNIQUE);
CREATE TABLE moz_places (id INTEGER PRIMARY KEY, url LONGVARCHAR, title LONGVARCHAR, rev_host LONGVARCHAR, visit_count INTEGER DEFAULT 0, hidden INTEGER DEFAULT 0 NOT NULL, typed INTEGER DEFAULT 0 NOT NULL, favicon_id INTEGER, frecency INTEGER DEFAULT -1 NOT NULL);
-- plus a roughly-equal number of lines creating indexes and triggers

that it would have been significantly more complex, and probably noticeably more lag-inducing, to implement this using plain text files. Running a grep for a single RE on a plain text file is bound to be quick: they've been refining that process for a few decades now. Doing the equivalent of a DB join on multiple text files is likely to be a lot slower and use way more memory unless you've indexed the text files in some sort of "database".

I'm also guessing that one of the main reasons for doing this was to speed up development of what is by far the most popular cross-platform browser. It's no insignificant challenge to develop a browser capable of supporting modern features like, for example, "Internet Banking", that the majority of people using web browsers expect because these are things that modern web browsers do. Keeping stuff in a database not only speeds up complex queries but also provides lots of powerful and scaleable access strategies to busy developers. This means they get to spend less time doing infrastructure development, and more time adding features, fixing bugs and security holes (have you reported this amazon cookie issue to Mozilla?), and planning for the future.

As an added bonus, people comfortable writing SQL queries can use them to get useful data from a detailed body of information that includes every time they visited every URL they've ever visited with that browser profile, optionally adding the specification of bookmark keywords, time visited, total number of visits since your cat had kittens, etc., into the mix. To do that using text files would I think be pretty time consuming for everybody involved. With sqlite3 it works like this:
 
$ # make a copy so it won't be locked
$ cp ~/.mozilla/firefox/pr0f113h45h.default/places.sqlite .

$ function ffrecent { sqlite3 places.sqlite "SELECT DISTINCT places.url FROM moz_places AS places LEFT JOIN moz_historyvisits AS visits ON visits.place_id = places.id WHERE visits.visit_date > $(date +%s -d "$1")000000 ORDER BY visits.visit_date DESC"; }

# get a count of unique url's that I've visited since the 10th
$ time ffrecent 2009-10-10 | wc
1156 1156 122983

real 0m0.306s
user 0m0.248s
sys 0m0.040s


For those not so comfortable writing SQL but adept in grep wizardry or just patient enough to scan through a long text file, it's less easy to get data out. However I suspect that it's more likely that somebody will have written an extension that does what they want, because it's much easier, and pleasant, and more interesting, to do using the database model of history recording. For example there's an extension called I think xmarks that provides synchronization of bookmarks across different browser profiles, and may do what you(? somebody..) wanted in terms of providing a default set of them to all of a site's users.

As far as using standard industry tools goes, it's certainly possible to convert a sqlite3 dump to one that can be loaded into a different DB, though it might take a bit of googling and/or cash to get it working. There is a bit of an ecosystem of tools developed specifically for working with sqlite3 databases.



Thu Oct 22 04:30:21 2009: 7334   Intuited

gravatar
PS I'm not totally sure that that's _exactly_ what that SQL code does; but it does seem to be in compliance with the 80/20 rule.



Thu Oct 22 11:44:55 2009: 7336   TonyLawrence

gravatar
I think your examples are way beyond what a browser should be. If somebody wants crap like that, bolt whatever dumb code is needed on top of a simple, "do what needs to be done and only what needs to be done" browser.

Wasn't that the whole POINT of extensions?






Thu Oct 22 12:11:27 2009: 7337   intuited

gravatar
I see where you're coming from... that's why I use elinks for most of my web browsing. But I do find it useful, actually in particular for web development, to be able to type a word into the address bar and have it quickly give me popular results for that keyword. This is great for hopping around in Drupal without having to go through the Admin page every time.
Keeping things simple has its benefits, but when those added levels of intelligence are helpful to, or even required by, the common web surfer, it becomes a lot more complicated or inefficient to add them into each extension that wants to use them... you would either end up with a bunch of different extensions all implementing similar functionality in different ways, or some sort of extension dependency system. While this last would be Awesome, it would likely end up being a lot more complicated to implement than just building more stuff into the browser core, especially when in doing so you can take advantage of a standard database system that's readily available. It also helps avoid having the end result be a mess of hacks of hacks.
Presumably other people use this functionality too, or at least that's Mozilla's opinion on the matter. Personally even if I didn't use advanced history queries at all I'd still rather that it be in there, if only to prevent Firefox from being slowly obsoleted by more capable browsers, and not having a standard that you can get on whichever platform.



Thu Oct 22 12:13:53 2009: 7338   TonyLawrence

gravatar
I think your usage of elinks proves my point :-)



Thu Oct 22 12:34:54 2009: 7339   intuited

gravatar
For that to be true... I would have to be everyone 8'>



Thu Nov 19 20:46:42 2009: 7569   anonymous

gravatar
Developers know better how to implement software. I can't understand why users keep arguing with developers about such inner workings.



Thu Nov 19 21:38:59 2009: 7570   TonyLawrence

gravatar
I'm not exactly a "user".

But even if I were, this is a bad case of feature bloat and an example of applying the wrong technology to a perceived need. The need doesn't exist for most users and the badly applied technology interferes with use.













Sat Nov 28 22:43:38 2009: 7675   BlueRaja
http://www.blueraja.com/blog
gravatar
Type "about:config" in your address bar
The database is not just used for bookmarks and history; it stores *all* the settings for firefox, including cookies, passwords, and even settings for all its extensions, some of which need the power and speed of a database. With all that data that needs to be stored, it makes sense to use a common database format which is both fast and widely available. I would have made the same decision.



Sun Nov 29 00:28:36 2009: 7679   TonyLawrence

gravatar
I don't agree that those "need the power and speed of a database". As I showed here and in other articles, plain text files are fast enough for quite large data sets and if you add external indexing, should be fast enough for almost anything.






Sun Nov 29 16:51:28 2009: 7691   BigDumbDinosaur
http://bcstechnology.net
gravatar
Developers know better how to implement software.

Oh really? Then, please explain to all us clueless computer jocks why Microsoft Windows is such a security disaster.

I don't know about you, Mr. or Ms. Anonymous, but in my computer career, which probably spans more time than you've been alive, I've written more than a half million lines of code, much of it assembly language (C hadn't even been invented when I started). I can attest from personal experience that developers often don't know how to best implement software. Some are more perceptive in that regard than others, but all developers will periodically adopt coding practices that are dubious. The use of SQLite in Firefox is, in my opinion, one of those cases.



Thu Jan 28 01:09:07 2010: 7959   anonymous

gravatar


Text files? Are you insane?

First of all, SqlLite is a fantastic way to store data. It's lightweight and very has efficient memory usage. New tables can be added on and also allows web sites to utilitze the storage engine.

Judging by this post, you sound like VB programmer who likes to solve all thier problems with 'simple text files'

Your also trying to predict the behaviour of millions of people. Some people may want thier history 6 'ways' from Sunday. Some people may choose to save ALL thier history for years. Try looking up a years worth of data in a text file compared to a DB. DB's have indexes remember?

dumbass...



Thu Jan 28 01:17:03 2010: 7960   TonyLawrence

gravatar


OK, I'm a dumbass.

But for this need, text files are faster. Period.



Thu Jan 28 02:31:44 2010: 7962   BlueRaja

gravatar


"But for this need, text files are faster. Period."
....WHAT!?



Thu Jan 28 03:31:47 2010: 7963   TonyLawrence

gravatar


So far I've let you make useless noise. That ends now - if you have nothing more to say than moronic comments like that, I won't allow your posts. If you want to argue that a database is needed, and can offer more than the arguments already noted above, fine, have it at. But if you are just going to be a jackass, you are cut off. Got it?



Mon Feb 15 15:01:29 2010: 8092   anonymous

gravatar


Personally, I was perfectly happy with the way Mozilla did things prior to SQLite.

I've tried my best to disable most of the new features which I don't need in FF 3.xxx, but I'm not sure I've managed to kill off persistent cookies in the dBase. By using FF 3.5.7 portable (not up to trying out 3.6 just yet), I've removed the problem of history and bookmarks being combined by not having history at all. But that is a short-term solution.

Pretty much out of ideas on how to fix what I don't like in FF.








Mon Feb 15 22:26:59 2010: 8093   anonymous

gravatar


There are always other browsers to look at using, such as Epiphany or Chrome. Firefox is obviously the most "supported" browser, but IMHO that's just as bad as Adobe Flash being a "supported" plugin. Everything should be standard and browser-agnostic. What if we could only drive on certain highways with certain cars?



Thu Feb 25 16:05:41 2010: 8139   anonymous

gravatar


To suggest that text files would be faster to perform (a) indexing against, and (b) complex joins on, which is what one would be doing to pull together atomic data, stored in an object oriented fashion, for use in FF, is amazing.

One wonders why Oracle and other RDBMS solutions are so popular, or why a library collates the text books, and builds indices of key data.

As to FF, linux and the fsync() problem, it just needs someone to use PRAGMA synchronous = NORMAL, as opposed to FULL.



Thu Feb 25 16:12:55 2010: 8140   TonyLawrence

gravatar


I don't "suggest".

People over use databases when they are actually slower than simple text files would be. You can easily prove that to yourself if you want.

That doesn't say that databases don't have a place for large or complex data sets.

I DO "suggest" that the average browser user doesn't need or want that complexity or that large a data set. THAT part is open to argument; that databases are misused is really not.







Fri Feb 26 03:29:03 2010: 8145   anonymous

gravatar


Quote "Why on earth does Firefox need a database for bookmarks and history?"

Quote "...it's simply inconceivable to me that anyone would think they'd need anything but simple text files for this"



Fri Feb 26 03:36:57 2010: 8146   harbinger

gravatar


Why don't you just use Chrome instead?



Fri Feb 26 03:37:26 2010: 8147   TonyLawrence

gravatar


Yes. You can cut and paste. You apparently can't comprehend what you are reading, but you are able to cut and paste it.

Go back and READ what I wrote. Unless you have something more intelligent to say, we're done discussing it.



Fri Feb 26 03:42:48 2010: 8148   TonyLawrence

gravatar


Why don't you just use Chrome instead?

????

How does whatever browser I happen to use change my questioning why Firefox uses sqlite for bookmarks?

In fact, I do use Chrome - and Safari, and Firefox.. but that has nothing to do with my questioning the FF developers reasoning. Even if I NEVER used Firefox, it's still a valid question to ask.







Sat Feb 27 09:13:54 2010: 8152   Asmodeus

gravatar


The post dated Thu Feb 25 16:05:41 2010 states some reasonable (and salient) facts, and you bully them for using the word "suggest", when the inference in your article is clear and present, as the subsequent post at Fri Feb 26 03:29:03 2010 tries to highlight.

A number of other posts also raise an eyebrow at text files being faster than a database, though at least one congratulates your "harsh critique".

As a defense contractor of 22 years, I will bat for FF and SQLite, not because the author of SQLite was also a defense contractor, or that the guys at Mozilla have obviously got 'something' right in FF, that it is a workable cross platform, extensible browser, but because the whole point of putting the application together the way it has been is for a better end-user experience, to provide for indexing, data integrity, even a query API that others can plug-in to.

As an aside, the article really ought to point out that the way fsync has been implemented is also flawed, in terms of I/O write barriers and journalled file systems, so any application that tries to mitigate data-loss using fsync, will run into the same problem.



Sat Feb 27 09:26:39 2010: 8153   anonymous

gravatar


What a wonderful lecture http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/278.ogg



Sat Feb 27 13:11:59 2010: 8155   TonyLawrence

gravatar


I "bullied" them?

:-)

Your comments:

the whole point of putting the application together the way it has been is for a better end-user experience,

is exactly right. Your comment

raise an eyebrow at text files being faster than a database

is moronic because it ignores the size of the data being searched. A database has overhead, indexing has overhead - if your searching needs are large enough, the advantages outweigh the disadvantages. When they are very small, the disadvantages can be quite annoying.

Even when the database approach is faster, it still doesn't always make sense to use it. If my search needs are to find something now and then (like seaching bookmarks), finding it in less than 1/10th of a second is pointless - I can't notice any improvement.

So you, database lover that you are, load up my ram with something I don't need. Your database can crash and screw up my data and if I want to access my data outside of your program, I have to know what database you used, your tables, your logic.. you've made my life harder because you don't understand that text files are better for some applications.

Do you feel "bullied"?








Sat Feb 27 13:44:37 2010: 8156   TonyLawrence

gravatar


I really think that most of the problem here is stupid people who think I'm saying text files are ALWAYS better than a database. Because they are apparently too dumb to read plain text (oh the irony), they don't understand that I'm not saying that at all so they get all upset and crazed about opinions that I don't have.

Sqlite is a nice lightweight database. If you NEED a nice, lightweight database, Sqlite is great.

When you DON'T need a database, using Sqlite or ANY database, is stupid. It can make things slower sometimes and it ALWAYS makes life more difficult if I need or want access to the data from outside the app. If you need it, fine. When you don't, it pisses me off that you used it.

In my opinion, a browser doesn't need a database for browser history. There's just no reason for it - again, my opinion.

From this point forward, any comment made by another jackass who plainly doesn't understand that I'm not saying "never use a database" will just be ignored. We have enough stupid comments here already; I don't need more.







Mon Mar 22 20:25:14 2010: 8257   anonymous

gravatar


I really don't care about either side of this issue. I just want sqlite removed from my PC. It's a hog, it slows everything down, and I just do not need it. I have 12 bookmarks and 15-20 browser history items. Firefox needs a database like I need another head.

Mozilla - get real!






Wed Apr 7 20:47:26 2010: 8381   anonymous

gravatar


Hi men,

I have been advocating about using databases without even thinking of it for years, despite I never wrote anything about it. Thank you for being in the crusade with me. If you stop by columbus OH, send me an email, you got a free beer!



Sun Apr 18 22:57:09 2010: 8431   anonymous

gravatar


Not all Gecko browsers subscribe to the SQLIte Gods.

The latest K-Meleon 1.6 (KM), sitll in Alpha with Gecko 1.9.1 engine, uses stub files where you can't absolutely get rid of the dBase beast. KM prefers to use the older technology of text files, including Wallet and of course uses its different beat philosophy with its windows scripting language macros in place of software emulation (XUl/XPI). For those of you who use Linux, KM runs under Wine and there are unofficial versions that work directly in Linux, minus some features of course.

I wonder if down the road that Mozilla will rethink its use of SQLite or is it tied into using it because of Google and the white/black list that Firefox employs?







Mon Apr 19 12:19:49 2010: 8434   anonymous

gravatar


I would like to humbly suggest that since the average user knows nothing of the details of the system, he or she is unlikely to see any value in erasing the browser history, and is unlikely to try or even want to do so. Unless firefox is to delete the user's old history, which does not seem like a very good idea, the default option will be to have that history accrue over many years.



Wed Jun 9 21:54:11 2010: 8679   Foxontherun

gravatar


I may be wasting my time with this response because you clearly reserve the right to censor postings based on one of your responses to someone with whom you disagreed, but here it goes anyways.

You don't like sqlite in Firefox, but you also use Safari & Chrome... sorry to tell you they also use sqlite.

You say a grep of a text file is faster than sqlite. How so? Grep has to parse the entire file every single time. sqlite is indexed so no need to parse the data each time it's accessed. How often is it accessed? When you click on your bookmarks tab, in the case of a flat text file it needs to be parsed each and every time in order to structure it in a hierarchy. In the case of sqlite, the data is already indexed.

When you are browsing and want to return to a site you were on earlier and pull up your history tab, with sqlite there is no additional parsing as it's indexed. With a flat text file it would have to be re-parsed each and every single time. Did you ever look at the history.dat file from FF2 and earlier? How are you at parsing the Mork file format? Not an efficient way at all to store the browser's history. There is no credible argument to support that a flat text file would be more efficient than sqlite files for a browser.

To the one that said Firefox saves all your history and cookies and such even when you clear it - wrong. I've done extensive R&D on Firefox over the course of the past 4 years (so pre-sqlite and post-sqlite). When you elect to clear your history, cookies, etc, it gets cleared short of permission issues at your file level.

To the person who only has 15 bookmarks and a handful of pages in history, you are the exception. Unless you contain your browsing sessions to very few sites, and only a few pages per site, you will have a lot more history than what you claim to have.

You may not wish to avail of all the benefits that can be leveraged through the address bar as a result of indexing bookmarks and history, or the ability to have tags or keywords associated to bookmarks. But others are using these features, and in larger numbers than the few here complaining about sqlite.

You don't want to use sqlite, you don't have to do so. But you'll have to either go back to FF2 to achieve this, or maybe use lynx if you are prepared to abandon GUI browsing in the process.

The arguments being put forth here clearly demonstrate the lack of knowledge that the people making them have about browsing and sqlite. And it shows a complete lack of ability to recognize that the performance and features that arise out of using sqlite in Firefox benefits the greater percentage of users with minimal detriment to those that don't want it.

The only posting that I've seen where I can see the performance problems with it was where the FF user profile was on a server and the journaling feature of sqlite caused a lot of disk activity on the server. This is not an issue for home users certainly, or any install where the user files are stored on the local machine where the browser is running.

Like I said, I've invested considerable time over the past 4 years examining FIrefox, and even posted the database schema complete with relationships between files in the various tables in places.sqlite to the mozilla wiki site (about 3 years ago now). I've been a regular subscriber and contributor to the Firefox support group where I've seen many issues posted by users. Very, very few have been people complaining about sqlite. The few that did related to the fact that there was no longer a bookmarks.html file (but that is easily addressed with a preference in about:config to update bookmarks.html on exit).

Invest a bit of time in better understanding exactly what is happening behind the scenes vs how it used to work behind the scenes in FF2 (which I've done in both cases) and you'll most likely change your views. You may still not see the need for sqlite based on your usage of the browser. But last time I checked, Firefox was not written for a specific user but for the greater good of the 30+ million people that use it.

The performance impact of sqlite for the very, very light users vs a text file is not observable by the end user. The performace impact of sqlite for the average to heavy users vs a text file would be very noticeable.



Wed Jun 9 22:25:51 2010: 8680   TonyLawrence

gravatar


I only censor ridiculous comments. Yours doesn't even begin to approach, that, sorry.

However, I did not for a minute suggest that you'd actually use grep. That was simply to illustrate that you don't need a database to achieve speed.

I appreciate your comments and particularly note your mention that other browsers also use sqlite. All I know is that Firefox performance, somewhat dismal to begin with, decreased when the sqlite announcement was made.

I quit using Firefox months ago. IMNSHO, it has reached the point of being unusable.

If that has nothing to do with sqlite, fine, it has nothing to do with sqlite, but my opinion remains unchanged: there is no need whatsoever to use a database for bookmarks and browser history. Your arguments present nothing that changes my mind about that.








Thu Jun 10 00:03:59 2010: 8682   Foxontherun

gravatar


Fair enough, you have no value for it based on how you use the browser. I have no problem with that. Everybody has different needs. But your inability to see past your own needs and see how using a database allows functionality that would be impractical at best if not impossible with a flat text file is where I have a problem. And because of that fact, there is no point for me to take the time to explain the inner workings of the database files and the functionality that can be provided as a result of using database files vs flat text files. If you allowed yourself to take an honest look at it and learn how things work behind the scenes and the functionality that is leveraged as a result of using databases vs flat text files and how some of those functionalities are appreciated and benefit different large segments of the user base then we could have a real discussion on this issue. But you apparently have not taken the time to fully learn how they are being used (i.e. what info is being stored, and what functionalities are achieved by it), or considered the greater benefit offered to its users that appreciate those features not possible in a flat text file situation.

You say I have offered nothing to convince you of the benefits of sqlite files vs flat text files. You have also failed as you have done nothing to convince me of the reverse. The difference being that my position is based on 4 years of digging into the inner workings of these files (pre & post sqlite) and my ability to see how others may benefit from features that I may have no need for, whereas your opinion does not expand beyond your own needs (which flaws your argument because you are trying to argue A vs B for you specifically, ignoring the needs of the remaining users) and so far appears to lack the depth of knowledge of FF data files in pre & post sqlite days (if you possess it, you have not articulated it).



Thu Jun 10 01:16:23 2010: 8683   TonyLawrence

gravatar


I wouldn't expect to convince you. You are apparently involved with the development, so of course you think this is all wonderful.

You are wrong, but you'll never see that. My expectation is that in years to come there won't be enough FF users left to bother developing for, but that is a different issue, and who knows: you might fix the clunky old thing yet. I doubt it, because all comments I've seen from developers show me that none of you have a clue how sad the poor Fox has become, but there is always hope.








Thu Jun 10 01:33:12 2010: 8684   TonyLawrence

gravatar


By the way, I can't make out whether you are actually involved in development or simply highly familiar with the code.

If the former, I find it somewhat amusing that I am being lectured by someone actually responsible for the memory leaking, crash prone, regularly freezing piece of bloatware that we call FireFox. What clever programmers you all must be!

The only thing that has kept the user base is the restoration of tabs and the thousands of extensions. Amusingly, Chrome has tab restoration, but it really doesn't even need it. Of course it is broken in other ways, but at least it doesn't crash like FF.



Thu Jun 10 01:39:48 2010: 8685   Foxontherun

gravatar


Actually I'm not a developer, just someone with a real thirst for understanding how things work under the hood. I'm not even a programmer.

Your example of searching a large file very quickly is over simplified. Imagine a user with 500 bookmarks (not unheard of) with layers of folders in the hierarchy of the bookmarks. In order to display that hierarchy of bookmarks properly each time the user clicks on the bookmark pull down menu using a flat text file, the application would have to read in the entire file and parse it out to figure out the folder/subfolder structure, and sort all the entries. The user continues to navigate and 5 minutes later clicks on the bookmark pull down menu again. The process restarts of re-parsing the entire flat text file once again. Repeat this overhead each and every single time the user clicks on the bookmarks pull down menu and you'll start to see the impact that extends way beyond simply searching the file for a string of text.

In a database environment, when the user clicks on the pull down bookmarks menu it's simply a matter of database queries to display the bookmarks. And with proper use of indexes that becomes quite efficient.

The same can be said for the 5,000+ entries in your history (but again, if your usage is to never avail of your history then no point explaining the benefits of it as they won't apply to you, but they will apply to millions of users that do use it).

In a flat text file you'd have to parse the history and sort by date/time each and every single time the user accesses the history. In a database scenario you don't have all that parsing overhead.

In a flat text file scenario how do you deal with selected removal of some of the content (i.e. delete or even editing of a bookmark)? You either end up with a void in the file, or re-write the entire file omitting what was deleted. In sqlite, everything is transaction driven. It's simply a transaction to remove an entry from the database. At appropriate times the database gets compacted to remove the blank records left from deleting an entry.

An entry in a database gets indexed quickly and then is easily searchable or parsed. An entry in a flat text file much be searched each and every single time and must be fully parsed each and every single time.

You have severely oversimplified how a browser works by examining simply how quickly you can search through a flat text file with grep. You are omitting a lot of other processing that must take place to make that information useful to the end user. Processing that is much more CPU and disk intensive in a flat text file environment than in a database environment.

Your needs might be able to be adequately met in a plain text file environment. But you'd have a browser with pretty bland features that would be overlooked by the vast majority of users in favour of browsers that offer rich features thanks to a better file system than simply flat text files. You think FF will lose its users with the current development path, but they'd lose a ton more with your development path as it would not be able to compete with the features of other browsers that many users appreciate.



Thu Jun 10 01:45:25 2010: 8686   TonyLawrence

gravatar


You aren't even a programmer?????

And you presume to lecture me and tell me that I'm wrong in an area I probably know 100 times more about than you do?

Your clueless description of how a flat file bookmark file would be handled shows your complete ignorance. Unlike you, I know both how databases work AND how to handle a flat text file.

Begone, pretender. You've made a fool of yourself and you are no longer welcome to post here.







Thu Jun 10 02:05:34 2010: 8687   foxontherun

gravatar


Ah, the censoring begins... I am no longer welcome here, yet you didn't address my points. My logic on how a flat file would be processed may not accurately represent the algorithm that would be used, however it does not negate the fact that your argument was founded on purely the fact that a grep search was fast enough to be suitable (for your needs). It failed to consider the additional overhead of processing such a file to render bookmarks or history pull down menus or sidebar menus, or providing the indexing functionality that is offered in FF 3.x vs using a database file to achieve this.

Instead of simply attempting to ridicule me and shut me out simply because you feel that your superior programming knowledge trumps everything else without offering any credible supporting arguments, share your wisdom with me and others. Educate us on how a flat text file can achieve the same efficiencies as a database when it comes to parsing and indexing. How it can efficiently update the file and its indexes to achieve comparable end user features that is achieved through database design.

I may not be a programmer, but that does not mean I am incapable of understanding programing principles. I have dug into FF source code at one point to figure something out and had no difficulty understanding the logic of the subroutine I was examining. The fact that it's very well documented is certainly very helpful.



Thu Jun 10 03:38:20 2010: 8688   foxontherun

gravatar


I don't know if you are away from your system so have not had the chance to vet my earlier response, or if you've chosen to file it in the bit bucket. I guess at the risk of being wrong I can draw my own conclusions if my previous comment does not show up after a day or so that you've elected to broaden your censorship criteria.

On a side note, when I said I am not a programmer I never said I have never written code, or that I never took programming courses. You made assumptions (right or wrong - my background should not preclude me from participating) simply because I stated I am not a programmer. Certainly more weight must be given to the more knowledgeable/experienced/educated individual in absence of reason to the contrary. But if that is the benchmark, then I most likely have 100 times more knowledge about FF than you do considering I've invested in excess of 1000 hours researching the topic over the past 4 years (why is not important - I have my reasons for leaving it at that recognizing that it obviously diminishes the credibility that can be given to my statements, but suffice to say have I absolutely no association to the Firefox development team or even Mozilla Foundation). So who trumps whom in this situation - the programmer or the researcher? Well neither in my opinion. Each point stands on its own merit regardless of our background. It's either right or it's wrong (or somewhere in the middle). If we get into opinions, then yes, our knowledge, experience, and education will bear relevancy on the weight given to the opinion.

Your first response to me was that you only censor ridiculous comments. My logic may have been flawed (although in absence of clarification on your part I really can't know for certain now can I), but far from ridiculous. The point still remains that parsing (and re-parsing) a flat text file will result in more overhead that a few sql queries against a database unless you know of a way that it can be accomplished more efficiently than what you've articulated so far (scanning the entire file with a while loop searching for a matching string).

So if you choose to censor my future postings as you appear to suggest in your response to me stating that I am not a programmer then you are not censoring only ridiculous comments as you originally claimed. Thus you will have dictated an end to the discussion for no other reason than your opinion about my knowledge, experience, and education based on the comment that I am neither a developer nor a programmer. Someone can state they are not a cabinet maker, but that doesn't mean they've don't dabble in cabinet making. But not to an extent that they would qualify themselves as a cabinet maker, especially given it's not how they earn a living. Because of that does it automatically invalidate /everything/ they say on that topic regardless if some of it is correct and some of it is not?



Thu Jun 10 11:09:08 2010: 8690   TonyLawrence

gravatar


I have let these last two comments pass because you think you aee being censored for your opinions. You misunderstand: I am cutting you off because you say is not adding anything of value.

You blather on that I have not made my case. I have - in the original post and in many comments that follow. I'm not going to waste more time and space repeating myself again. You don't understand, and I do not care.

You have shown yourself to be deficient in both reading comprehension and in programming knowledge, I have no more patience to humor you.

So, please: hang it up. Not because I fear your nonsense or cannot retort it, but simply because it is nit adding anything of value. Whatever you think you have to offer has already been said. Enough!



Thu Jun 10 16:07:35 2010: 8692   anonymous

gravatar


Post this, or don't. I really don't care. My last comment to you on this issue and then I'll move on (it's Google's fault - I hit your page when looking for something about Firefox otherwise I would never have read and commented on your postings).

I read through all your postings (unless you also posted under another name). You mentioned external indexing at one point - that was the only thing I missed the first time I scanned through them. But the crux of your argument is that people don't need their history in a database. You want a vanilla browser with everything an add-on beyond that. Well good for you. But the browser was not written for you. So where is your needs assessment or survey to validate that people don't need the advantages offered by having history in a database? Why do your specific needs trump all the other users that do use it in a way that they benefit from the current model?

It all comes back to the fact that you have a particular preference for what you want out of a browser. Based purely on your preference and how you use the browser you've made the argument that they should not have used database files but rather plain text files. Well Mr. programmer, Firefox is open source. So have at it. Strip out what you don't want, redesign it to your liking by going to text files with support for add-ons for those that want more than what you want, call it APLawrence browser and everybody is happy.

What is the cut off before it is more efficient to go to sqlite vs flat text file? 10,000 records, 5,000 records, 100,000 records? What about other features that leverages the database files that would not be as easily implemented in a flat text file? Where is the cut off before it makes sense to use sqlite files or some other database file?

If you are going to use a flat text file, I would expect you'll assign it a data structure. And if you use an external indexing system, you are in essence using a database model. Firefox developers met with the creator of sqlite and designed the databases (tables, relations between each) in consulation with that person. They didn't just throw it all in a database for the sake of doing so with no consideration given to how the data could be structured in such a manner as to support the design goals of the project.

Granted your agument being that with a plain text file you can access your information without difficulty. Well there are add-ons such as sqlite manager to easily access the database files. Or you can write your own XUL Runner application, or even use VB, PHP, or any other high level language to run sql querries against the database files.

I've seen users that want to extend the history beyond 99 days (the current limit imposed by Firefox) because they really appreciate that feature. I've seen users that have a real need to have cache retained because they are still on dial-up where they live.

Firefox is very customizable. You can tweak it to your likings between preferences and add-ons. The end user usability will not suffer because of sqlite files even for the very light users. But the reverse is quite likely not the case. Moderate to heavy users and power users will be negatively impacted by going to a plain text file model.

Now your dilema - if I don't post and reply to this, I won't be able to respond to this guy's inaccuracies. If I do post, then I'm giving this person more air time that I'd rather not give.

Do as you please (of course that goes without saying really, it's your site). Either way life will go on for me as it did before I stumbled across your site.



Wed Jul 21 14:10:33 2010: 8840   anonymous

gravatar


SQLITE is without technical merit...just another data base utility that supports ONLY FF and its "friends." It clogs your PC with useless info, and is one of the most egregious utility programs out there. SQLITE are just a bunch of useless hackers who once again steal your information for their profit...and you let these Cretans get away with it. For what ever good it may do, I have a small command file that deletes all the sqlite files (except the dumb sqlite.dll which is needed to make dumb FF run...) at least the info is not saved with the rest of the asinine SQLITE crap. SQLITE crappy files are easy to search...then get rid of them every time you stop FF. When Windows tries to write to these loads of junk, it nearly stops all actions until this sophomoric function is stopped...I use FF portable on a thumb drive and when I suspect Windows "write" is loading up SQLIT I just pull the plug (remove the thumb drive...without the jerky stop function thing about stopping a device first...it does no harm and stops the childish use of MY PC hard drive...) I should just stop using FF I suppose...it has become a pile of dog-doo.



Wed Jul 21 16:37:12 2010: 8841   skepticalhippo

gravatar


Good points, Anonymous. Interestingly, the same people responsible for SQLite have been suspected of stocking store shelves with fake tinfoil. However by avoiding the use of giant tinfoil balls you can dodge that bullet too.



Fri Jul 23 13:32:01 2010: 8851   anonymous

gravatar


I recently commented on how ridiculous the use of SQLIT is by FireFox (Portable)...it is NOT necessary, and is used ONLY in order to pad the pockets of the producers of FF and SQLIT at your expense (time and use of your PC and it's HD or where you may have FF (Portable). SQLIT, regardless of what any of you computers geniuses say is a total waste...it makes money for SQLITE and FF and who knows who by sellling YOUR information. Their excuse that it is a useful data base logging tool is a thunderous LIE! All of us sheep out here in internet land might just as well pull down our pants and show the world how stupid we are to let these creeps make money off of our personal "data." But hey...we all think the intenet (which is mostly a sewer) is a wonder agent. Facebook, Myspace, Twitter also fall in the same category of selling your personal data to make money and laugh at you dummies all the way to the bank. Want to see how SQLIT gums up your PC and the extent to which it's used...find "journal.sqlit" on your computer (some where in FireFox/data/profiles or search...you'll find it.) Then right click its properties and make the file "read only," and uncheck "archive." Now run FF and see how long it takes trying to write your crap to THEIR dumb file...HA! SQLITE is pure CRAP...so is the company and its programers...they are the quitessential of internet pimps. You may want to uncheck the read only and check archive in the journal.sqlite if you insist on the mostly only FF Portable version... SQLITE STINKS and they use YOU to make $$$$$. Dummies! P.S. Text files are bad enough, but far better than data bases like SQLITE in MOST internet uses...especially email.



Wed Aug 18 02:04:03 2010: 8911   Tim

gravatar


Ugh, I feel for your frustration with these trolls. Trolls always love to take one aspect and fixate on that one point without addressing anything else. Half of them haven't even bothered to leave their name.

I agree with you that Sqlite is more pain than its worth, particularly because Firefox doesn't actually use it for a proper unified database. There's, what, 4 or 5 different sqlite files in the profile? If they're going to leave all their data in separate files they might as well have kept them in a more easily-accessible file format like JSON or XML. From what I can see all they've done is started using Sqlite as a spiffy file format instead of a real database.

It's a bit ridiculous for people to argue that Sqlite is somehow automatically better because it's a database. Technically, text files are also a type of database. Sqlite is just a more complex one, and that complexity has also made it a pain in the ass. I've had to VACUUM my places.sqlite more than a couple of times, which is a huge oversight if I have to keep doing this manually myself. It's a square peg in a round hole.

Personally I prefer many files vs one. For example, Macs store all their settings in many hundreds of human-readable Property List files, so you can drill down to the settings you want to trash/copy/fix without touching any other settings. Compare to the venerable Windows Registry.



Wed Aug 18 02:26:50 2010: 8912   anonymous

gravatar


Geez, anyone remember FF 1.5 when life was simplier for Gecko?

Firefox, imo, has become kludge central. In the rush to have the latest and greatest, Mozilla has begun to march to the beat of its own internal drummer, and the result for those of us who liked the old minimalist FireFox? Lots of kludges to turn off what some see as impressive new features and others as bloatware.

Cases in point: Old Location Bar (to mitigate the collage that is the Awesome Bar); turning off Geolocation (never liked mailing home to Mother Google); Switch Themes (should be the default if you too like the old themes, not the new "lite" Persona stuff which may be a malware magnet... the jury is still out); Tab Candy, can't turn it off, but you can drag it off the Nav bar (geez... how many tabs do you need to play with? And on old Pentium M the answer is too many and too resource intensive); Plug In Container, great a new way to enhance memory leaks (truly astounding on my Pentium M with 768 Megs RAM) so yet another toggle switch turned to false.

And what is this up close and personal thing that Mozilla has with Google? Is Google's white/black lists better than the security software people? Or are they worse? As in if it's a google-based site, it most definitely is whitelisted....(don't you trust Mother Google to be objective?). And those Gigabyte sqlite files... gross bloat imo, not to mention the churn on my laptop's hard drive and consequent drain on the battery when I'm on the road (which is often) and yet another case of mailing home to Mother Google.










Thu Sep 9 10:21:18 2010: 8965   Alex

gravatar


I would say, the primary problem with SQLite by FF is that the relational DB is used for full text search. This is main misconcept, I think. They need to use something like Lucene or similar to search bookmarks and history...

Regards,
Alex



Thu Sep 9 14:37:42 2010: 8966   Pete

gravatar


When I use FF "portable" with it's egregious and poor use of SQL, I set all SQL files as read only...that seems to P-O some FF protocol by putting "dumb" warning messages up about security, etc. not being able to write to such and such file...but I've always felt, regardless of the arguments as to how useful it is, that software that insists on writing and logging their garbage to my HD is of no technical merit...or use to me.



Sat Sep 25 11:54:57 2010: 9006   anonymous

gravatar


Random note. My places.sqlite is about 40MB and when I type a new URL to the address bar and press enter, then Firefox freezes for 3-4 seconds. It's completely irresponsive during this time. And then starts loading the page. VACUUMing/REINDEXing the db is no help.



Sat Sep 25 12:05:00 2010: 9007   TonyLawrence

gravatar


I've given up on Firefox. I use Chrome now. There are pages that just don't work; for those I switch to Safari. But for 99% of my day, it's Chrome - no bloat, no freezing. The Firefox developers have screwed this pooch solid.



Mon Nov 1 01:51:03 2010: 9082   anonymous

gravatar


clearing history does not clear places.sqlite of same history that you just cleared FFv3.






Mon Nov 1 11:42:14 2010: 9083   anonymous

gravatar


I no longer have ANY problems with Firefox and its "egregious" use of SQLITE...I got rid of Firefox from all my computers and will never use it again!



Tue Nov 2 00:40:39 2010: 9085   anonymous

gravatar


Although I dislike SQLite in FireFox, I've not totally given up on the beast. I have, however, constrained it with everything suggested here. The largest sqlite file is places which weighs in at 1,500 kb and my FF doesn't mail home to Google ever. I also took the precaution of only using FireFox Portable.

For everyday browsing, I use K-Meleon, which has truncated almost all of SQLite and replaced elements with tried and true components, like Wallet and gotten rid of XUL/XPI by using Windows APIs instead in a robust macro script.

Hope FF sees the light somewhere down the road. But am not holding my breath.






Tue Feb 8 05:18:15 2011: 9285   Lynn

gravatar


I personally use opera for most my internet ventures but i have bookmarks and history totally disabled in all browsers. I use a small text file to store my bookmarks and history in as this gives me control over what gets stored and who see's it. Its 1000 x's faster than any browswer on the market using grep, and the author is absolutely right there is no need for a a database server for bookmarks and history. 99% the users i heard complain about the old style bookmarks in flat txt files had far worse trouble elsewhere denying them fast access to their bookmarks/history.



Sat Apr 30 12:04:09 2011: 9474   anonymous

gravatar


I have used Firefox for several years. Today it is lost & gone because I don't have an up to date version of sqlite, which I have never heard of nor even realized that I had an obsolete version of!






Sat Jul 9 11:45:01 2011: 9611   TonyLawrence

gravatar


http://www.theinquirer.net/inquirer/news/2092583/mozilla-stuffs-ram-usage-optimiser-firefox-aurora-beta

The discovery that increasing garbage collection on its Firefox Javascript engine resulted in an 80 per cent drop in memory usage raises questions about the quality of Mozilla's code testing.

It's not just their testing. It's the whole clunky ball of code. I switched to Chrome and have not looked back.







Sun Aug 14 16:30:16 2011: 9701   anonymous

gravatar


The advantage of using a transactional database instead of text files is easy from an implementers perspective. Off the shelf tested components almost always beat "roll your own" solutions.
1. The database takes care of all the search algorithms.
2. The database will recover atomically if the process is killed.
3. The database will take care of upgrades. Extending and upgrading database structures is far easier than recovering a custom file structure.

Also, Chrome (Google) uses sqlite as well.






Sun Aug 14 16:40:18 2011: 9702   TonyLawrence

gravatar


Sure, whatever you say. You are wrong, but go on believing that.



Sun Aug 14 16:42:51 2011: 9703   TonyLawrence

gravatar


By the way, a text file is not a "custom file structure" :-)



Sun Aug 14 20:20:12 2011: 9706   BigDumbDinosaur
http://bcstechnology.net
gravatar


The advantage of using a transactional database instead of text files is easy from an implementers perspective. Off the shelf tested components almost always beat "roll your own" solutions.

1. The database takes care of all the search algorithms.


You mean the database engine does that. The database itself is just a giant blob of bytes organized in some possibly strange way that the user wouldn't understand.

2. The database will recover atomically if the process is killed.

Having more than five minutes of programming experience with databases, I'd have to disagree with that statement. A kill -kill to the database engine during a read-modify-write cycle almost always causes some damage to the index. Separate tools are usually required to repair the index after such a calamity.

3. The database will take care of upgrades. Extending and upgrading database structures is far easier than recovering a custom file structure.

Again, that's only true if appropriate tools are used in a timely fashion.

BTW, calling a flat file a "custom file structure" is like call a salad with Italian dressing a "custom appetizer."

I think you are completely missing Tony's point. Just as an admiral wouldn't deploy a battleship to sink a rowboat, any IT jock worth his/her salt would not use a complex database structure to maintain information that can be handled in a sequential format. There's little in a web browser that requires the type of crossindexing typically provided by a relational database engine. Lacking that need, why deploy the battleship? Or to use another analogy, would you attempt to crack peanuts with a brick maul? I didn't think so.



Sun Aug 14 20:52:41 2011: 9707   TonyLawrence

gravatar


I see that particular battleship used in far too many places it isn't needed.



Mon Aug 15 05:04:40 2011: 9709   AnIhMull

gravatar


Yes, Tony's argument is basically that browsers should never have evolved past the state they were in in about 1998. I think you can get that level of functionality with Dillo these days. All of the "modern" browsers have moved on to provide indexing that makes it possible to dig up information out of our ever-expanding network of visited sites in increasingly intelligent ways.



Mon Aug 15 11:42:39 2011: 9710   TonyLawrence

gravatar


The comment above proves that none of these idiots are capable of reading.







Thu Dec 29 04:19:33 2011: 10413   differentanonymous

gravatar


I found this note about SQLite and thought I would share it. I do not remember where I found it...but I found it doing a search.

Any comments?

"For several years of free AVG and now the Pro version, I have had no problem
with Mozilla Firefox until today, February 10, 2009.

There is a potentially dangerous tracking cookie residing in:
C:\Documents and Settings\Administrator\Application
Data\Mozilla\Firefox\Profiles\lnim1snr.default\cookies.sqlite

cookies.sqlite is a protected file that requires a special program to open it.
Inside, you will find that your Mozilla Firefox browser is now sending information about you to an anonymous 3rd party.

You will constantly need to check for this file and delete it after using your browser.
To delete the "unwanted” cookies.sqlite,
do a Windows Explorer search for "cookies.sqlite"
and when you find it residing at
C:\Documents and Settings\Administrator\Application Data\Mozilla\Firefox\Profiles\lnim1snr.default\cookies.sqlite

Delete the file and then empty the trash...

Best regards."






Fri Jan 27 15:12:31 2012: 10523   Rick

gravatar


I'm a developer for about 10 years now. I personally don't have a problem with using sqlite files for this. I've found that sometimes it's not always about speed, extendability, maintenance, organaizability (I think I just made that up :) ) play a big part in software design. It's always a trade off between a number of things. I respect that you would have done it differently but I'm confused as to why you would care so much. Great thing about choices I guess is you can choose not to use it now if you like. I like the choice because I can open the files up and easily see what's going on. I can see schemas and how things are organized and it's nice and neat. Often in text files you don't get that meta information without really analyzing the data.



Fri Jan 27 15:15:45 2012: 10524   TonyLawrence

gravatar


I cared because Firefox became a clumsy, crash prone bear. I don't care any more: I switched to Chrome.



Fri Jan 27 15:40:26 2012: 10526   Rick

gravatar


And you attribute that purely to sqlite? I'm sure you don't, but it's sort of what that comment sounds like. I've read all these comments and your vision of what a browser should be doesn't match with the majority of users. The majority of browser users are technologically stupid. They are 30-60 year old people who hardly ever used computers growing up. Maybe for a couple papers they had to write in high school or college. They don't know anything about browser history or much less how to clear it. Which lines up with a good number of the complaints FF was getting as related to one of the posts in this topic.

In short using sqlite didn't destroy FF.



Fri Jan 27 15:54:19 2012: 10527   TonyLawrence

gravatar


Absolutely correct. IMNSHO, however, it is one indication of the bad choices the developers have made.



Thu Mar 1 12:41:46 2012: 10677   SensibleComputerUser

gravatar


Firefox is such a pos. I despise it. Using sql to keep track of bookmarks is typical of the retarded decisions made by that group of imbeciles. They are so stupid I'm surprised they can remember to breathe.

When can we get a decent browser ? Please, God ? Please ?



Fri Mar 9 23:31:53 2012: 10721   anonymous

gravatar


Agree completely with this article. Furthermore, the close 'alliance' between Firefox and GOOGLE gives one pause as to why the browser needs to database our cookies and places visited. Sounds suspiciously like SNOOPING--for what?? We need to ask why these things are being done in the background without our knowledge or permission.



Fri Mar 9 23:54:02 2012: 10722   anonymous

gravatar


How Do We TURN OFF Squirrel-lite

This squirrel-lite stuff is as annoying as real-live squirrels are on my roof in the morning when I want to sleep. How do we turn it off? It seems to be a security risk. Have found long files of every site visited, with intimate details, not encrypted, including income tax information AND SOCIAL SEC NUMBERS, in places.sqlite, etc. WHY IS FIREFOX DOING THIS?

And another thing: THESE FILES ARE CREATED EVEN WHEN YOU ARE NOT USING FIREFOX!? That is mind-boggling! We often use another Mozilla browser and we find these sqlite files created constantly, even though we may not open Firefox for MONTHS OR EVEN YEARS ON END. We keep it as back-up for when our favored browser doesn't seem able to load a webpage.

Something fishy seems to be going on...



Sat Mar 10 14:46:18 2012: 10724   TonyLawrence

gravatar


I don't know that Firefox is the only offender..



Fri Oct 19 07:56:18 2012: 11388   anonymous

gravatar


According to Mozilla itself, http://support.mozilla.org/en-US/questions/731546, it says do not use FF. You are advised to use other browsers. So, isn't it obvious?

Perhaps, I think, FF has been bought out. So some espionage going on. In case you are not aware, all the major browsers, systems has some elements of stealth. Many of the malware are actually from them - either for testing the system or for competitive info. Some are funded by spurious governms.

They know the problem, they can easily solve it if the want to.
Just for your info, the moment I switch to their new homepage that searches using google, everything is a breeze. Even the Tools-options etc info all changed. Ya, I did deleted some sqlite stuff but don't think that's the cause for the improved service.
You may recall the recent call to use chrome, which unfortunately, is too large for my small drive. So their this new way of running it saves me the trouble, after much annoying wait-times and agony.
Espionage going on all the time else how you think MS made all the wrong moves whilst android made all the right moves?



Fri Oct 19 14:51:50 2012: 11389   TonyLawrence

gravatar


No, that's not "Mozilla itself". That's just a disgruntled user commenting.






Tue Nov 27 09:09:52 2012: 11442   anonymous

gravatar


I'm curious how you feel about this article in hindsight? The awesome bar is basically a demonstrably superior experience for 90%+ users at this point. Chrome and Firefox both use sqlite to support this feature. The dataset of input history required to support it is extremely large.

Clearly the Firefox devs knew what they were doing by dumping flat files.



Tue Nov 27 11:30:56 2012: 11443   TonyLawrence

gravatar


In hindsight, I feel that dumping Firefox was one of the best decisions I ever made.

I continue to run into people complaining of this, that or the other problem because they have stayed with this bloated carcass of a web browser.

Clearly the Firefox developers have screwed it up even more in the ensuing years.



Tue Nov 27 13:12:06 2012: 11444   anonymous

gravatar


Yet the thing you dumped it for uses sqlite as well.

Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments


If you want a picture to show with your comment, go get a Gravatar

Kerio Samepage


Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more. We appreciate comments and article submissions.

Publishing your articles here

Jump to Comments



Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

I am a Kerio reseller. Articles here related to Kerio products reflect my honest opinion, but I do have an obvious interest in selling those products also.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.

pavatar.jpg

This post tagged:

       - Programming
       - SQL
       - Web/HTML



















My Troubleshooting E-Book will show you how to solve tough problems on Linux and Unix systems!


book graphic unix and linux troubleshooting guide



Buy Kerio from a dealer
who knows tech:
I sell and support

Kerio Connect Mail server, Control, Workspace and Operator licenses and subscription renewals



Click and enter your name and phone number to call me about Kerio® products right now (Flash required)