Fossil Forum
Post: Fossil works on LumoSQL v0.8, if you are brave
Hello Fossilers,
LumoSQL v0.8 is out, and you can compile Fossil and libfossil against it. LumoSQL is a modification (not a fork) of SQLite, combining it with multiple backend databases using some shim code. The backends that make sense for Fossil are LMDB and LMDBv1.0. This early preview release is for Fossil users because Fossil and SQLite have a very special relationship -- besides, people here are often curious and break things, which is what we need.
The instructions for building Fossil and libfossil on LMDB-backed LumoSQL are fairly well-tested. The recipe is quite manual and will take about 30 minutes depending how fast your machine is. You will be stepped through creating an amalgamation just like SQLite. In that same Fossil directory is a test script and a stress tester. Pretty much, if something doesn't work just like it does in SQLite then that is a bug. Also be aware that this pre-release version of LumoSQL will eat your data, so don't rely on it.
Why would someone want to use LMDB-backed Fossil? Reasons I can think of:
- it's fun, nobody has ever tried to do this before
- LMDB-backed LumoSQL (not Fossil) tends to be about 1.7x faster than SQLite on average, being a bit slower for some workloads and faster for others. This does not appear to hold for Fossil, where . Right now it works and that is a really good start.
- LumoSQL can use LMDBv1.0 as a backend, which includes encryption. Therefore Fossil can now be encrypted and there is a basic example included in this release. v1.0 has other features like incremental backup that can be used without Fossil even knowing about it.
- LMDB is more crash-proof under many (but not all) circumstances than SQLite due to it using the copy-on-write facilities of the underlying operating system.
We have some musings on importing massive trees, because that is the sort of thing some people here seem to enjoy doing.
LumoSQL stays as close as possible to the SQLite way of software development, recognising that as the only trillion-scale software in existence, there is little room to make changes without creating a fork. Hopefully some of the ideas in LumoSQL will be useful food for thought. If there is one consistent lesson I have learned in LumoSQL so far it is that there isn't any other software like it, and LumoSQL should continue to make itself the skinniest possible intervention while connecting two very solid codebases. As an illustration of maturity, SQLite ships with about quarter of a million tests, and LumoSQL has maybe 50.
So there we are - do please consider giving LumoSQL a try.
--
Dan Shearer
[email protected]
Hello Fossilers,
LumoSQL v0.8 is out, and you can compile Fossil and libfossil against it. LumoSQL is a modification (not a fork) of SQLite, combining it with multiple backend databases using some shim code. The backends that make sense for Fossil are LMDB and LMDBv1.0. This early preview release is for Fossil users because Fossil and SQLite have a very special relationship -- besides, people here are often curious and break things, which is what we need.
The instructions for building Fossil and libfossil on LMDB-backed LumoSQL are fairly well-tested. The recipe is quite manual and will take about 30 minutes depending how fast your machine is. You will be stepped through creating an amalgamation just like SQLite. In that same Fossil directory is a test script and a stress tester. Pretty much, if something doesn't work just like it does in SQLite then that is a bug. Also be aware that this pre-release version of LumoSQL will eat your data, so don't rely on it.
Why would someone want to use LMDB-backed Fossil? Reasons I can think of:
- it's fun, nobody has ever tried to do this before
- LMDB-backed LumoSQL (not Fossil) tends to be about 1.7x faster than SQLite on average, being a bit slower for some workloads and faster for others. This does not appear to hold for Fossil to the extent of my very simple tests, where LMDB seems about 8% faster for sequential writes (importing or re-creating a repo) and 10% slower for random writes. What results do you get?
- LumoSQL can use LMDBv1.0 as a backend, which includes encryption. Therefore Fossil can now be encrypted and there is a basic example included in this release. v1.0 has other features like incremental backup that can be used without Fossil even knowing about it, but isn't quite released yet.
- LMDB is more crash-proof under many (but not all) circumstances than SQLite due to it using the copy-on-write facilities of the underlying operating system.
We have some musings on importing massive trees, because that is the sort of thing some people here seem to enjoy doing.
LumoSQL stays as close as possible to the SQLite way of software development, recognising that as the only trillion-scale software in existence, there is little room to make changes without creating a fork. Hopefully some of the ideas in LumoSQL will be useful food for thought. If there is one consistent lesson I have learned in LumoSQL so far it is that there isn't any other software like it, and LumoSQL should continue to make itself the skinniest possible intervention while connecting two very solid codebases. As an illustration of maturity, SQLite ships with about quarter of a million tests, and LumoSQL has maybe 50.
So there we are - do please consider giving LumoSQL a try.
--
Dan Shearer
[email protected]
Reasons I can think of:
Thank you very much for this :).
recognising that as the only trillion-scale software in existence...
Devil's Advocate here: there's some mumbling that curl is more widely-deployed than sqlite. That seems inflated to me but a search engine is telling me:
is curl the most widely-deployed software in the world?
Yes. With an estimated 20 to 30 billion active installations worldwide, curl (and its underlying library, libcurl) is widely considered the most pervasive and widely-deployed software component on earth.
And:
is sqlite the most widely-deployed software in the world?
Yes, SQLite is widely considered the most widely deployed and used database engine in the world, and it is generally ranked among the top five most deployed software modules of any description.
🤷♂️
(@sean: no, still no plans to add an emoji selection widget!) Z 40238f
LumoSQL v0.81 is out, and the encryption seems to be solid for all operations the tests in fossil/test/ use. This includes downloading SQLite into an encrypted fossil and then doing fossil rebuild. This is still very much pre-release software.
Next up quite possibly might be:
- Turning on LMDB sync options to match the SQLite sync modes. Right now LumoSQL uses the most conservative mode which makes it much slower than it could be
- siphash in Secure Message Authentication mode, for fast and strong per-row change-and-corruption detection. (This applies to traditional sqlite3 Fossil databases, not just LMDB.)
--
Dan Shearer
[email protected]
Curl is almost certainly not deployed more than SQLite. But I guess it depends on how you count… Further, I think it tracks that any application using curl is likely to use sqlite, but the reverse may not be true? Z
Thanks for all your work! This seems far easier to spin up than the SEE Fossil repositories. I hope I am able to give it a try soon. Having encrypted fossil repos as an ordinary option in the distant future without having to jump through hoops would be incredible.
Next up quite possibly might be: Turning on LMDB sync options to match the SQLite sync modes. Right now LumoSQL uses the most conservative mode which makes it much slower than it could be
This is now in trunk and, for Fossil, it makes a consistent 30% speed increase on sequential writes, being a bit faster or a bit slower in other circumstances but never more than a percentage point or two slower native Btree Fossil in the worst case I have seen. This is a compromise, being less safe because you might lose the last transaction, the same compromise SQLite makes with sync=normal (the default for Fossil as far as I know.)
Dan
Thanks. I look forward to the bug reports. Since we get everyone else's well-tested code to do the work (SQLite, Fossil, LMDB, libsodium) hopefully it can mature fairly quickly.