Fossil Forum
Post: pathologically bad performance of fossil sync
I noticed that when I have a big repository with lots of updates that haven't been synced in a very long time, the performance of fossil sync is pathologically bad compared to just deleting the local repository and cloning it again. I recently tried running fossil sync on an old computer that's a few years of commits behind the main repository, and I had to eventually kill the process after a few hours because it wasn't making progress. By comparison, a fresh fossil clone of the same upstream repository finishes in a minute or two, even though fossil sync theoretically has to transfer less data.
Is something that can/should/will be fixed, or is this just the way it is?
I'm running fossil 2.23 on the server and fossil 2.28 on the client.
Please try updating to the latest Fossil on your server. I seem to remember fixing the problem you describe.
Good to know that this is fixed. Unfortunately I can't test the fix because I foolishly replaced the old client-side repository with a new clone without keeping a backup, so I'll have to take your word for it.
You ought not take my word for it, because I'm just guessing. I need you to verify.
I'm not asking you to downgrade the client. I want you to upgrade the server. Put both the client and the server on the latest version of Fossil and see if you can still reproduce the problem. If you can, then we will investigate further. But if not, then and only then will we assume the problem has been resolved.
Richard I think the problem is he doesn't have the original repository that was two years behind on updates. Sounds like he doesn't have a old backup he can recover it from either.
Is there some way for him to create a clone that is two years out of date? Then he could try updating with the older fossil in his server (to verify the bug with the new out of date clone) and with the newer fossil on the server to verify the fix.
It seems that you can use fossil purge checkins 2024-01-01 to get somewhere close to this effect (testing it on a clone of Fossil itself, it leaves behind non-checkin artifacts). You're definitely not supposed to use this command so casually, but it might still be useful to see if the original behavior can both be replicated and tested if upgrading the server fixes it. I'd advise only using it on a clone specifically made to test the original issue, not a real working copy.
If you really want to see the difference, just create an empty repository and update the project-code to be the same as the target project and then run "fossil sync":
$ mkdir /tmp/empty && fossil reconstruct /tmp/project.fossil /tmp/empty Reading files from directory "/tmp/empty"...
Building the Fossil repository... 100.0% complete... project-id: 1e46dd45e1a2abad96363e0caf0a20ae75b21019 server-id: 670c0a781220f8a5f6a320ddf63c5ca56338f2f5 admin-user: amb (initial password is "pY6hmxuUiH") $ fossil remote -R /tmp/project.fossil https://fossil-scm.org/home $ echo "UPDATE config SET value = 'CE59BB9F186226D80E49D1FA2DB29F935CCA0333' WHERE name = 'project-code';" | fossil sql -R /tmp/project.fossil $ fossil pull -R /tmp/project.fossil
This will effectively "clone" the entire repository using the sync protocol, instead of cloning and yes, expect it to take a long time.
There is a huge difference between "fossil clone" which is optimized for speed, and "fossil sync" which is optimized for incremental updates and deltas. Cloning effectively just does a raw copy of all the artifacts in the remote repository and sends them using the "cfile" card. Sync, on the other hand, uses the "file" card and they are typically deltas.
Furthermore, there's no guarantee that the deltas and the content that they update will arrive in any particular order, so it all has to be reconstructed as it arrives.
I'm not sure it's a good idea to let a repository go for years without an occasional pull, however, Fossil will easily sync the data, it might just take longer. I think that's a good argument for getting into the habit of running "fossil all pull" on a more regular basis.
Is this really something that needs a "fix"?
Is something that can/should/will be fixed, or is this just the way it is?
Hard to say. I'm curious what would happen if you had upgraded your server. I'm not sure what bug was fixed in this regard, but if I get some time later maybe I'll setup a 2.23 server and try to compare sync times against a newer server.
I synced the entire Fossil repository using the sync protocol in 37 minutes. That's every single artifact and deltas over the wire:
$ time nice fossil pull -R project.fossil
Pull from ssh://[email protected]//fossil
Round-trips: 46 Artifacts sent: 0 received: 66579
Pull done, wire bytes sent: 7200795 received: 97619972 remote: fossil.bradfords.org
37m00.90s real 8m17.99s user 1m40.06s system
That's definitely longer than cloning, but I think this is expected given the differences in the protocols.
It's size is 598,282,240 which is larger than I would expect, but again, since I used "sync" and not "clone" there was no final rebuilding of the entire repository.
Here is a clone under the same conditions:
$ time nice fossil clone ssh://[email protected]//fossil clone.fossil Round-trips: 19 Artifacts sent: 0 received: 66698 Clone done, wire bytes sent: 5680 received: 100501254 remote: fossil.bradfords.org Rebuilding repository meta-data... 100.1% complete... Extra delta compression... 121 deltas save 1,967,975 bytes Vacuuming the database... project-id: CE59BB9F186226D80E49D1FA2DB29F935CCA0333 server-id: 199d72b8250a5a58e8bd288364e0ac5eb4c1c7a2 admin-user: anonfsl (password is "i56Noqnet2") 6m05.54s real 2m02.69s user 0m57.26s system
Definitely faster than sync, and the file size is much smaller at 73,416,704 bytes.
Let's see what happens if I rebuild the "sync" clone:
$ time nice fossil rebuild --compress project.fossil 101.3% complete... Extra delta compression... 5433 new deltas save 165,849,738 bytes Vacuuming the database... done 10m50.96s real 4m29.63s user 1m45.21s system
After rebuild the "sync" clone is 82,026,496
Anyway, maybe later when I have more time I'll setup an older Fossil server and "sync" from it to see if there is a difference in time (and perhaps narrow it down to a bug fix if there is a difference).
I believe that this is the fix that you're remembering:
https://fossil-scm.org/home/info/637fcef83023bad3
In my testing, while this significantly reduced the number of round-trips that occur (because it increases the amount of work completed each round trip), it didn't really make the sync protocol synchronize the entire set of artifacts any faster. Here is what I got before that fix:
Round-trips: 192 Artifacts sent: 0 received: 66580 Time: 38m43.51s real 5m31.85s user 1m58.17s system
And here is after the fix:
Round-trips: 46 Artifacts sent: 0 received: 66580 Time: 36m51.45s real 7m52.38s user 1m42.44s system
Where this might improve things, I suppose, is if the cost of connecting is expensive. My tests were all done on a local LAN so each round-trip had low latency. But where latency is a factor, this fix will reduce the number of connections and so latency will impact less.
I'm curious about the original problem that started this thread... the claim that no progress was being made at all because that's not what I observed with my testing. How big is the repository? What kind of artifacts are being synchronized? How many artifacts are there and many changes were there in that 2 year gap?
In my case, I effectively transfered all 66,580 artifacts and 20,493 check-ins that spanned 18+ years of development in 38 minutes. That's slow, but not unbearable, and, I might add, atypical.
I believe that this is the fix that you're remembering:
https://fossil-scm.org/home/info/637fcef83023bad3
In my testing, while this significantly reduced the number of round-trips that occur (because it increases the amount of work completed each round trip), it didn't really make the sync protocol synchronize the entire set of artifacts any faster. Here is what I got before that fix:
Round-trips: 192 Artifacts sent: 0 received: 66580 Time: 38m43.51s real 5m31.85s user 1m58.17s system
And here is after the fix:
Round-trips: 46 Artifacts sent: 0 received: 66580 Time: 36m51.45s real 7m52.38s user 1m42.44s system
Where this might improve things, I suppose, is if the cost of connecting is expensive. My tests were all done on a local LAN so each round-trip had low latency. But where latency is a factor, this fix will reduce the number of connections and so latency will impact less.
I'm curious about the original problem that started this thread... the claim that no progress was being made at all because that's not what I observed with my testing. How big is the repository? What kind of artifacts are being synchronized? How many artifacts are there and many changes were there in that few years gap?
In my case, I effectively transfered all 66,580 artifacts and 20,493 check-ins that spanned 18+ years of development in 38 minutes. That's slow, but not unbearable, and, I might add, atypical.