But why just every 8th artifact?

Funnily enough... i had the same question when adding the content cache to libfossil (it was, for years, just a useless placeholder saying "fossil does this, do we need to?").

After much experimentation i got best results, overall, with that every-8th-artifact approach. Why, i have no idea, but it works well in practice. As proof i offer only this comment from libfossil's counterpart of content_get():

        /*
           2021-03-24: in a debug build, running:

           f-parseparty -t c -c -q

           (i.e.: parse and crosslink all checkin artifacts)

           on the libfossil repo with 2003 checkins takes:

           10.5s without this cache
           5.2s with this cache

           We shave another 0.5s if we always cache instead of using
           this mysterious (mx-n)%8 heuristic.

           Later testing with f-rebuild gives much different results:
           the (mx-n)%8 heuristic provides the best results of the
           variations tested, including always caching.
        */

Shrug.

After much experimentation i got best results, overall, with that every-8th-artifact approach.

Did you experiment with repos having different average chain lengths? The Fossil and SQLite repos have deep chains, and GCC almost certainly even deeper, but many do not. By "chain" I mean how many deltas are applied to a given base artifact. I suspect this might be where numbers other than 8 could be optimal.

--
Dan Shearer
[email protected] / https://shearer.org

Did you experiment with repos having ...

No - only that one repo in its then-current state. At the time i don't recall having been conscious of how deep delta chains got but have since seen chains 500+ long. (Which, frankly, surprised me - my expectation was to see a full "snapshot" copy retained every Nth levels (20? 50?) if for no other reason than mpg does that with "reference frames".)

Turns out the artifact cache fix also helps fossil git export by a large amount. Unsurprising, but it only just occurred to me to test.

Considering I have a real-world project that is comparable to the size of Fossil's and SQLite's repository, if you were to ignore the unversioned space which stores for the most part CI/CD artifacts. Mostly was running this to ensure that the repository would not break on these changes, or cause some major issues. From running these, have found the gains to be insignificantly better at best or insignificantly worse at the worst. Project wise, everything is encoded as text and yes it is a Java Virtual Machine with its own runtime that has been getting worked on for the past almost 11 years using exclusively Fossil.

Repository wise, looking at this, using the Fossil version from Debian:

Repository Size:    900,063,232 bytes
Number Of Artifacts:    123,528 (24,017 fulltext and 99,511 deltas) Details
Uncompressed Artifact Size: 96,511 bytes average, 9,385,540 bytes max, 11,921,902,840 total
Compression Ratio:  13:1
Unversioned Files:  3549 files, 742.9MB compressed, 83% of total repository space
Number Of Check-ins:    20,923
Number Of Files:    33,042
Number Of Wiki Pages:   0
Number Of Chat Messages:    0 (0 still alive, 0 bytes in size)
Project Age:    10 years, 7 months, 1 days
Project ID: 3cd3b073348ef46a9516345172ed6ab3e3f021d4 SquirrelJME
Fossil Version: 2025-05-16 13:52:48 [611144ddd9] (2.26) (details)
SQLite Version: 2025-04-30 14:37:00 [20abf1ec10] (3.50.0) (details)
OpenSSL Version:    OpenSSL 3.5.7 9 Jun 2026 (0x030500070)
Web Server: fossil version 2.26 [611144ddd9] 2025-05-16 13:52:48
Schema Version: 2015-01-24
Repository Rebuilt: 2026-02-08 03:05:15 By Fossil 2.26 [611144ddd9] 2025-05-16 13:52:48 UTC
Database Stats: 109,871 pages, 8192 bytes/page, 37 free pages, UTF-8, delete mode
Backoffice: Last run: 39.4 minutes ago

It does get 26% of the way there to your 8000 commits per year metric.

As for the results, using the version from the branch:

$ ls -al squirreljme.fossil
-rw-r--r-- 1 stephanie stephanie 900063232 Sep 28 03:20 ../squirreljme.fossil

$ ./export.sh 
Place your finger on the fingerprint reader
cache_n=500

real    0m31.687s
user    0m25.680s
sys     0m5.974s
cache_n=2000

real    0m31.839s
user    0m25.703s
sys     0m5.994s
cache_n=5000

real    0m31.668s
user    0m25.991s
sys     0m5.600s

$ ./rebuild.sh 
sqlcache=0
  100.0% complete...

real    1m54.155s
user    1m49.522s
sys     0m4.233s
sqlcache=-32000
  100.0% complete...

real    1m51.298s
user    1m49.614s
sys     0m1.292s
sqlcache=-256000
  100.0% complete...

real    1m51.991s
user    1m50.361s
sys     0m1.035s

$ ./test-integrity.sh 
cache_n=500

real    1m34.954s
user    1m33.281s
sys     0m1.282s
cache_n=2000

real    1m30.549s
user    1m29.507s
sys     0m0.961s
cache_n=5000

real    1m31.305s
user    1m30.040s
sys     0m0.828s

Considering this is not a freshly rebuilt repository that was literally just converted/imported/created and is my current working repository as it stands on the disk... would not really consider this a win but more of a loss as more memory is used for almost no benefit.

As for whether my input is valuable or not, there is https://squirreljme.cc/, SquirrelJME is probably in the group of the largest and longest living naturally Fossil projects in terms of sheer size and longevity so it might be useful for testing Fossil... at least with the stuff I do normally.

Thanks Stephanie, this is exactly the kind of feedback I was looking for, it really helps and bears on the chat with Stephan about chain depth earlier. It seems to me like shorter chains and lower compression ratios have less gain, and I replicated your results that the cache isn't worth changing for SquirrelJME. This does rather suggest that the cache size could be adaptive, because I presume/hope fetching the compression ratio is not expensive.

I tweaked the throwaway test scripts and ran them overnight on SquirrelJME, the current Fossil repo, and LumoSQL, using Fossil [62e4b3a879]. This time I used shuffled cache settings (no more drop cache commands needing sudo in the scripts) and ran each script five times.

I got similar results as previously reported for SQLite, and the current Fossil repository confirms the large gains:

command          baseline   best       change
export             76.05s    36.93s     51.4%
rebuild            29.08s    22.95s     21.1%
test-integrity    114.07s    48.63s     57.4%
git export        531.94s   463.73s     12.8%

But SquirrelJME is just like you say:

command          baseline   best       change
export             21.57s    21.39s      0.9%
rebuild            75.55s    72.76s      3.7%
test-integrity     75.72s    72.68s      4.0%

LumoSQL, has a tiny wee 793 check-ins and 4,263 artifacts and didn't gain on any test. It looks like the result depends strongly on repository structure. Fossil has 59,244 deltas among 67,727 artifacts and an 83:1 compression ratio. SquirrelJME has a 13:1 ratio, while LumoSQL has a 7:1 ratio. So far it looks like the larger cache pays off when repeatedly reconstructing and reusing deep delta chains.

A content cache around 2000 entries is the sweet spot for the Fossil tree. And a 32 MB SQLite page cache captures most of the rebuild gain. Larger settings produce small further reductions but increase memory use and so probably aren't worth it.

I realised that I am under-reporting the gains from the SQLite page cache, because the scripts test on /tmp and that uses tmpfs. As you'd expect, testing on a real disk improves the gain still more in those cases where it has any effect.

Thanks Stephanie. I don't see that there is any caching win at all for SquirrelJME, although I am curious to try turning it off altogether and set very low values and see what happens for your repo (that's just tweaking the numbers in the scripts.)

--
Dan Shearer
[email protected] / https://shearer.org

Yeah, your further testing just shows that the adjustments have to be adaptive and it cannot be static.

As for memory and /tmp being on tmpfs, which is the case for me as well, but you also have to consider that SSDs are in use today which are very fast and I do have 64GiB of RAM on this system so the entire repository easily fits in memory without any swapping. You may want to consider testing this on systems with much less memory, slower disks, and are CPU bound as there could still be improvements for those systems.

Thinking about it, hypothetically, you might be running into an inverted performance optimization as this is memory and throughput related. If you were to say to run this on a Pentium 3 or a PowerMac G4, with far less and slower memory, with SquirrelJME it would likely have a more substantial gain compared to say Fossil or SQLite. That is, our current systems that we use daily for a project such as mine are already at a throughput barrier. For Fossil and SQLite however, this change removes overhead that is not caused by memory but more of throughput, so it has gains to be made. I suppose the main point in this case is that SquirrelJME is memory-bound whereas Fossil and SQLite are throughput-bound, and depending on which one is at a premium this can benefit either potentially. Maybe if memory-bound (our systems), anything >= 25:1 gets this turned on? And if throughput-bound, anything <= 25:1 gets this turned on? This is just a hypothetical thought though.

I think you're right Stephanie, and I'm beginning to have a theory for how it works in the code, noting these two caches interact.

This is a good example of a clear research question, and it can be answered by a one-off exhaustive testing run in a controlled environment. If someone has exclusive access to a large Linux computer for a few days then that's what is needed here: matrix testing with parameters of: memory constraints, disk speed constraints, repo chain depth, repo size, content cache size and SQL cache size. Then we do some analysis on the results and that gives us the heuristics to encode. It's worth this much bother because of the very large potential savings.

--
Dan Shearer
[email protected] / https://shearer.org

Stephanie,

Some more results, and I think I have an answer for you:

• I was wrong about the compression ratio of your SquirrelJME repo, because I believed fossil dbstat but then I noticed that includes unversioned files. Total bytes from dbstat minus SELECT sum(length(content)) FROM unversioned gives the actual compression ratio, SquirrelJME is 190:1 and Fossil is 129:1. So its not likely to be about compression ratios. And I eventually noticed that neither rebuild nor test-integrity touches unversioned artifacts so its about the structure of that relatively small bit of versioned SquirrelJME.
• SquirrelJME has a very different structure to fossil, with very short delta chains of around 2 deltas each, except for one massive one, the manifest which is 5000+ deep. Fossil has about 40 deltas for every file on average over 20 years, many of them edited regularly. The Fossil chains are all in use at once so a bigger cache really helps. The squirrel chains are mostly used one at a time so a cache doesn't help. Fossil has 1500 files, SquirrelJME has 10,000. Basically if Fossil has too small a cache, fossil test-integrity thrashes. SquirrelJME needs room for about 500 entries, and more doesn't help.
• No delta manifests! I think this is the cause Stephanie. Each SquirrelJME manifest lists every file in its check-in, 10,508 at the tip of trunk, about 111 bytes per F card, thus of the uncompressed, versioned content, 11.1 GB out of your 11.9 GB is manifests. I hacked up a way of regenerating a repo with delta manifests, and this went down dramatically (the hashes get recomputed so it isn't a very useful hack except for measuring structure.) I used perf record -g to measure what happens with test-integrity, plus ran the original cache tests, and you're back in fossil territory of cache speedups.

One thing you can do is make sure you have delta manifests from now on.

So on the whole maybe we don't need much of a heuristic? I have got some degree of answer about why cache every 8th, and the scratchings of a formula to calculate the cache, but if it turns out the main bad case for caching is lack of delta manifests then perhaps that is the main thing to detect. There isn't a way to fix them without breaking the DAG, like I did in my test.

--
Dan Shearer
[email protected] / https://shearer.org

That is very interesting!

I have actually been considering setting forbid-delta-manifests to on (and in .fossil-settings/ although that might not actually do anything) since I rather agree with the Manifests as Proof of Code Integrity point for such a large project. Considering I do target out-of-date systems, the project can still be checked out with Fossil 1.28 (from 2014) without any SSL support.

Thinking on your results, considering how important the manifests are and how effectively the entire repository would need to be rebuilt to take advantage of them... it sounds like for repositories like mine that there would need to be some kind of special manifest compression without the hashes needing to be recalculated. For the most part, situations where almost every file gets touched or moved around is quite rare but does happen from time to time. On some ideas off the top of my head, for an optional vacuum setting perhaps, maybe some kind of checkpoint system where every n manifests in a chain (or when a merge occurs) a full one is recorded, then each one in between internally uses delta compression although on the outside it looks as if nothing changed at all. Of course this would basically mean no backwards compatibility for local .fossil repositories with this enabled, although that would probably not be an issue for remote servers considering they generally send just the artifacts.

Stephanie,

In the short term (ie now) I propose making the content cache 2000 entries and the SQLite page cache 32 MB. That will speed up SQLite and Fossil repos and any others even remotely like them in structure, at a memory cost to some (hopefully quite rare?) repos that are like yours. You previously said this isn't worth the cost to you, but I'm asking a different question now: would you find it overly annoying?

As another step another day I have a formula to introduce that seems to relate the parameters involved but that needs more testing.

Best,

--
Dan Shearer
[email protected] / https://shearer.org/