Fossil Forum
Post: Local cloned repo and backup have binary differences
Hi, I'm testing out migrating from git to fossil for some private projects and I'm experimenting with my backup workflows right now. Everything is self-hosted on my local network, with off-site backups.
As these repos are just single file databases (I love this so much, by the way), I had intended to use rsync for off-site backups.
After seeing a fossil backup command, I wanted to experiment with that too to see if there was any difference.
What I found is that cloning the fossil-scm repo, and then running fossil backup on that gave me a backup which was the same size on disk, but they had different SHA256's. With some closer inspection, I saw a small number of diffs in the binary.
So, I guess my question is, what is the most canonically "correct" backup approach? I will be running the backups from my "central" host, which has any/all admin permissions - but I also take a separate SHA256 of the copied files to keep an eye on file corruption.
The backups occur at 4am, so they're not in use here, no server running, etc...
Thanks!
The two repos should be logically equivalent, even if they are not byte-for-byte identical. Don't worry that the checksums don't match.
You can use sqlite3_rsync or rsync or "fossil backup" to make backups, if you want. But what I normally do is just host the repositories on multiple, geographically dispersed VPSes and set a cron job to run "fossil sync" on each of the backups once per hour (or whatever other interval seems appropriate for your situation). That is how the Fossil self-hosting repositories are backed up, and also the SQLite repositories.
There are three clones of https://fossil-scm.org/
All three stay synchronized using crons. They are in three different States of the US (GA, NJ, and CA, respectively) on servers leased from two different ISPs. And, of course, we have are local clones as well.
Another thing you can do is to register multiple remotes for your local repos, (using the fossil remote add command), perhaps including some that are actually off-site and reachable only by ssh: instead of https:. Then run:
~~~ fossil set autosync all ~~~
Then whenever you do a check-in, all content is automatically pushed out to all of your remotes, including all of your backups. That makes check-ins slightly slower, since now Fossil has to go out and talk to a bunch of remote systems, but check-ins are not that common, so this isn't really a big negative. I use this latter approach for backing up the financial records for the business that manages Fossil and SQLite.
Thanks for the reply Richard!
Great to know that each of the mechanisms is logically the same, so I don't need to think about it. I do have one "live" off-site replica on a cron, but in the instance I asked about above, those were intended to be immutable snapshots that get backed up with the rest of my data periodically (in the event of data corruption or malware, I can just rollback to one of these immutable backups from a few days/weeks prior).
Thanks again for the quick reply!
Clones are probably always slightly different. If you want a perfect binary matching, you'll need to use a sync tool that enforces this, like rsync.
- Fossil records where check-ins come from per data receive event. Artifacts (the immutable bits of information that make up the low-level data, which is assembled into high-level forum threads like this one or check-ins with message and file diff) are order-independent, but have a local received id. So depending on the order they are received, they are stored in different orders in the database, resulting in different hashes
- Additionally, clones made with
fossilwill not carry over user accounts, and will have a different password salt - Some other settings are also not transferred by a
clone