Fossil SCM

fossil-scm / www / backup.md
Source Blame History 304 lines
5c2ef2a… wyoung 1 # Backing Up a Remote Fossil Repository
5c2ef2a… wyoung 2
c14a1f6… drh 3 One of the great benefits of Fossil and other [distributed version control systems][dvcs]
fc2c124… wyoung 4 is that cloning a repository makes a backup. If you are running a project with multiple
c14a1f6… drh 5 developers who share their work using a [central server][server] and the server hardware
8a539f1… wyoung 6 catches fire, the clones of the repository on each developer
fc2c124… wyoung 7 workstation *may* serve as a suitable backup.
c14a1f6… drh 8
c14a1f6… drh 9 [dvcs]: wikipedia:/wiki/Distributed_version_control
c14a1f6… drh 10 [server]: ./server/whyuseaserver.wiki
c14a1f6… drh 11
fc2c124… wyoung 12 We say “may” because
fc2c124… wyoung 13 it turns out not everything in a Fossil repository is copied when cloning. You
fc2c124… wyoung 14 don’t even always get copies of all historical file artifacts. More than
fc2c124… wyoung 15 that, a Fossil repository typically contains
fc2c124… wyoung 16 other useful information that is not always shared as part of a clone, which might need
c14a1f6… drh 17 to be backed up separately. To wit:
5c2ef2a… wyoung 18
fc2c124… wyoung 19
8a539f1… wyoung 20 ## <a id="pii"></a> Sensitive Information
5c2ef2a… wyoung 21
5c2ef2a… wyoung 22 Fossil purposefully does not clone certain sensitive information unless
8a539f1… wyoung 23 you’re logged in as a user with [Setup] capability. As an example, a local clone
fc2c124… wyoung 24 may have a different `user` table than the remote, because only a
5c2ef2a… wyoung 25 Setup user is allowed to see the full version for privacy and security
5c2ef2a… wyoung 26 reasons.
5c2ef2a… wyoung 27
e3e4bec… wyoung 28
8a539f1… wyoung 29 ## <a id="config"></a> Configuration Drift
8a539f1… wyoung 30
2b1c5bc… wyoung 31 Fossil allows the local configuration to differ in several areas from
2b1c5bc… wyoung 32 that of the remote. You get a copy
2b1c5bc… wyoung 33 of *some* of these configuration areas on initial clone — not all! — but after that,
2b1c5bc… wyoung 34 remote configuration changes mostly do not sync down automatically.
2b1c5bc… wyoung 35
2b1c5bc… wyoung 36
2b1c5bc… wyoung 37 #### <a id="skin"></a> Skin
2b1c5bc… wyoung 38
2b1c5bc… wyoung 39 Changes to the remote’s skin don’t sync down, on purpose, since you may
2b1c5bc… wyoung 40 want to have a different skin on the local clone than on the remote. You
2b1c5bc… wyoung 41 can ask for updates with [`fossil config pull skin`][cfg], but that does
2b1c5bc… wyoung 42 not happen automatically during the course of normal development.
2b1c5bc… wyoung 43
2b1c5bc… wyoung 44
2b1c5bc… wyoung 45 #### <a id="alerts"></a> Email Alerts
2b1c5bc… wyoung 46
2b1c5bc… wyoung 47 The Admin → Notification settings do not get copied on clone or sync,
2b1c5bc… wyoung 48 and it is not possible to push such settings from one repository to
2b1c5bc… wyoung 49 another. We did this on purpose because you may have a network of peer
2b1c5bc… wyoung 50 repositories, and you only want one repository sending email alerts. If
2b1c5bc… wyoung 51 Fossil were to automatically replicate the email alert settings to a
2b1c5bc… wyoung 52 separate repository, subscribers would get multiple alerts for each
2b1c5bc… wyoung 53 event, which would be *bad.*
2b1c5bc… wyoung 54
2b1c5bc… wyoung 55 The only element of the email alert configuration that can be pulled
2b1c5bc… wyoung 56 over the sync protocol on demand is the subscriber list, via
2b1c5bc… wyoung 57 [`fossil config pull subscriber`][cfg].
2b1c5bc… wyoung 58
2b1c5bc… wyoung 59
2b1c5bc… wyoung 60 #### <a id="project"></a> Project Configuration
2b1c5bc… wyoung 61
2b1c5bc… wyoung 62 This is normally generated once during `fossil init` and never changed,
2b1c5bc… wyoung 63 so Fossil doesn’t pull this information without being forced, on
2b1c5bc… wyoung 64 purpose. You could accidentally merge two separate Fossil repos by
2b1c5bc… wyoung 65 pushing one repo’s project config up to another, for example.
2b1c5bc… wyoung 66
2b1c5bc… wyoung 67
2b1c5bc… wyoung 68 #### <a id="other-cfg"></a> Others
2b1c5bc… wyoung 69
2b1c5bc… wyoung 70 A repo’s URL aliases, [interwiki configuration](./interwiki.md), and
2b1c5bc… wyoung 71 [ticket customizations](./custom_tcket.wiki) also do not normally sync.
2b1c5bc… wyoung 72
c64f28d… drh 73 [cfg]: /help/configuration
2b1c5bc… wyoung 74
e3e4bec… wyoung 75
e3e4bec… wyoung 76
8a539f1… wyoung 77 ## <a id="private"></a> Private Branches
78c0c55… wyoung 78
78c0c55… wyoung 79 The very nature of Fossil’s [private branch feature][pbr] ensures that
78c0c55… wyoung 80 remote clones don’t get a copy of those branches. Normally this is
78c0c55… wyoung 81 exactly what you want, but in the case of making backups, you probably
fc2c124… wyoung 82 want to back up these branches as well. One of the two backup methods below
78c0c55… wyoung 83 provides this.
78c0c55… wyoung 84
78c0c55… wyoung 85
8a539f1… wyoung 86 ## <a id="shun"></a> Shunned Artifacts
5c2ef2a… wyoung 87
5c2ef2a… wyoung 88 Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want
5c2ef2a… wyoung 89 your local clone to be a precise match to the remote, it needs to track
5c2ef2a… wyoung 90 changes to the shun table as well.
5c2ef2a… wyoung 91
5c2ef2a… wyoung 92
8a539f1… wyoung 93 ## <a id="uv"></a> Unversioned Artifacts
5c2ef2a… wyoung 94
5c2ef2a… wyoung 95 Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by
5c2ef2a… wyoung 96 default unless you specifically ask for it. Like local configuration
5c2ef2a… wyoung 97 data, it doesn’t get pulled as part of a normal `fossil sync`, but
5c2ef2a… wyoung 98 *unlike* the config data, you don’t get unversioned files as part of the
5c2ef2a… wyoung 99 initial clone unless you ask for it by passing the `--unversioned/-u`
5c2ef2a… wyoung 100 flag.
5c2ef2a… wyoung 101
5c2ef2a… wyoung 102
8a539f1… wyoung 103 ## <a id="ait"></a>Autosync Is Intransitive
1299d67… wyoung 104
1299d67… wyoung 105 If you’re using Fossil in a truly distributed mode, rather than the
1299d67… wyoung 106 simple central-and-clones model that is more common, there may be no
1299d67… wyoung 107 single source of truth in the network because Fossil’s autosync feature
1299d67… wyoung 108 isn’t transitive.
1299d67… wyoung 109
1299d67… wyoung 110 That is, if you cloned from server A, and then you stand that up on a
fc2c124… wyoung 111 server B, then if I clone from your server as my repository C, your changes to B
1299d67… wyoung 112 autosync up to A, but not down to me on C until I do something locally
1299d67… wyoung 113 that triggers autosync. The inverse is also true: if I commit something
1299d67… wyoung 114 on C, it will autosync up to B, but A won’t get a copy until someone on
fc2c124… wyoung 115 B does something to trigger a sync there.
1299d67… wyoung 116
1299d67… wyoung 117 An easy way to run into this problem is to set up failover servers
1299d67… wyoung 118 `svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync
1299d67… wyoung 119 with the first. If all of the users normally clone from `svr1`, their
1299d67… wyoung 120 commits don’t get to `svr2` and `svr3` until something on one of the
1299d67… wyoung 121 servers pushes or pulls the changes down to the next server in the sync
1299d67… wyoung 122 chain.
1299d67… wyoung 123
1299d67… wyoung 124 Likewise, if `svr1` falls over and all of the users re-point their local
1299d67… wyoung 125 clones at `svr2`, then `svr1` later reappears, `svr1` is likely to
1299d67… wyoung 126 remain a stale copy of the old version of the repository until someone
1299d67… wyoung 127 causes it to sync with `svr2` or `svr3` to catch up again. And then if
1299d67… wyoung 128 you originally designed the sync scheme to treat `svr1` as the primary
1299d67… wyoung 129 source of truth, those users still syncing with `svr2` won’t have their
1299d67… wyoung 130 commits pushed up to `svr1` unless you’ve set up bidirectional sync,
1299d67… wyoung 131 rather than have the two backup servers do `pull` only.
1299d67… wyoung 132
1299d67… wyoung 133
2b1c5bc… wyoung 134 # <a id="sync-solution"></a> Solution 1: Explicit Pulls
78c0c55… wyoung 135
1299d67… wyoung 136 The following script solves most of the above problems for the use case
78c0c55… wyoung 137 where you want a *nearly-complete* clone of the remote repository using nothing
fc2c124… wyoung 138 but the normal Fossil sync protocol. It only does so if you are logged into
fc2c124… wyoung 139 the remote as a user with Setup capability, however.
5c2ef2a… wyoung 140
5c2ef2a… wyoung 141 ``` shell
5c2ef2a… wyoung 142 #!/bin/sh
5c2ef2a… wyoung 143 fossil sync --unversioned
5c2ef2a… wyoung 144 fossil configuration pull all
5c2ef2a… wyoung 145 fossil rebuild
5c2ef2a… wyoung 146 ```
5c2ef2a… wyoung 147
5c2ef2a… wyoung 148 The last step is needed to ensure that shunned artifacts on the remote
5c2ef2a… wyoung 149 are removed from the local clone. The second step includes
fc2c124… wyoung 150 `fossil conf pull shun`, but until those artifacts are actually rebuilt
fc2c124… wyoung 151 out of existence, your backup will be “more than complete” in the sense
fc2c124… wyoung 152 that it will continue to have information that the remote says should
fc2c124… wyoung 153 not exist any more. That would be not so much a “backup” as an
fc2c124… wyoung 154 “archive,” which might not be what you want.
fc2c124… wyoung 155
8a539f1… wyoung 156
2b1c5bc… wyoung 157 # <a id="sql-solution"></a> Solution 2: SQL-Level Backup
8a539f1… wyoung 158
8a539f1… wyoung 159 The first method doesn’t get you a copy of the remote’s
78c0c55… wyoung 160 [private branches][pbr], on purpose. It may also miss other info on the
78c0c55… wyoung 161 remote, such as SQL-level customizations that the sync protocol can’t
fc2c124… wyoung 162 see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can
78c0c55… wyoung 163 solve such problems if you have access to the remote server, which
ad47a44… wyoung 164 allows you to get a SQL-level backup by delegating handling of locking
ad47a44… wyoung 165 and transaction isolation to
ad47a44… wyoung 166 [the `backup` command][bu], allowing the user to safely back up an in-use
ea57a2f… wyoung 167 repository.
78c0c55… wyoung 168
ea57a2f… wyoung 169 If you have SSH access to the remote server, something like this will work:
5c2ef2a… wyoung 170
5c2ef2a… wyoung 171 ``` shell
5c2ef2a… wyoung 172 #!/bin/bash
5c2ef2a… wyoung 173 bf=repo-$(date +%Y-%m-%d).fossil
5c2ef2a… wyoung 174 ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" &&
e3e4bec… wyoung 175 scp example.com:museum/backups/$bf ~/museum/backups
5c2ef2a… wyoung 176 ```
5c2ef2a… wyoung 177
8a539f1… wyoung 178 Beware that this method does not solve [the intransitive sync
8a539f1… wyoung 179 problem](#ait), in and of itself: if you do a SQL-level backup of a
8a539f1… wyoung 180 stale repo DB, you have a *stale backup!* You should therefore run this
8a539f1… wyoung 181 on every node that may need to serve as a backup so that at least *one*
8a539f1… wyoung 182 of the backups is also up-to-date.
8a539f1… wyoung 183
5c2ef2a… wyoung 184
8a539f1… wyoung 185 # <a id="enc"></a> Encrypted Off-Site Backups
5c2ef2a… wyoung 186
5c2ef2a… wyoung 187 A useful refinement that you can apply to both methods above is
5c2ef2a… wyoung 188 encrypted off-site backups. You may wish to store backups of your
ea57a2f… wyoung 189 repositories off-site on a service such as Dropbox, Google Drive, iCloud,
5c2ef2a… wyoung 190 or Microsoft OneDrive, where you don’t fully trust the service not to
5c2ef2a… wyoung 191 leak your information. This addition to the prior scripts will encrypt
5c2ef2a… wyoung 192 the resulting backup in such a way that the cloud copy is a useless blob
5c2ef2a… wyoung 193 of noise to anyone without the key:
5c2ef2a… wyoung 194
5c2ef2a… wyoung 195 ```shell
6a3d6fa… wyoung 196 iter=152830
5c2ef2a… wyoung 197 pass="h8TixP6Mt6edJ3d6COaexiiFlvAM54auF2AjT7ZYYn"
5c2ef2a… wyoung 198 gd="$HOME/Google Drive/Fossil Backups/$bf.xz.enc"
5c2ef2a… wyoung 199 fossil sql -R ~/museum/backups/"$bf" .dump | xz -9 |
fc300d5… wyoung 200 openssl enc -e -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -out "$gd"
5c2ef2a… wyoung 201 ```
5c2ef2a… wyoung 202
5c2ef2a… wyoung 203 If you’re adding this to the first script above, remove the
5c2ef2a… wyoung 204 “`-R repo-name`” bit so you get a dump of the repository backing the
5c2ef2a… wyoung 205 current working directory.
5c2ef2a… wyoung 206
5c2ef2a… wyoung 207 Change the `pass` value to some other long random string, and change the
6a3d6fa… wyoung 208 `iter` value to something in the hundreds of thousands range. A good source for
5c2ef2a… wyoung 209 the first is [here][grcp], and for the second, [here][rint].
6a3d6fa… wyoung 210
6a3d6fa… wyoung 211 You may find posts online written by people recommending millions of
6a3d6fa… wyoung 212 iterations for PBKDF2, but they’re generally talking about this in the
6a3d6fa… wyoung 213 context of memorizable passwords, where adding even one more character
6a3d6fa… wyoung 214 to the password is a significant burden. Given our script’s purely
6a3d6fa… wyoung 215 random maximum-length passphrase, there isn’t much more that increasing
6a3d6fa… wyoung 216 the key derivation iteration count can do for us.
6a3d6fa… wyoung 217
6a3d6fa… wyoung 218 Conversely, if you were to reduce the passphrase to 41 characters, that
6a3d6fa… wyoung 219 would drop the key strength by roughly 2⁶, being the entropy value per
6a3d6fa… wyoung 220 character for using most of printable ASCII in our passphrase. To make
6a3d6fa… wyoung 221 that lost strength up on the PBKDF2 end, you’d have to multiply your
6a3d6fa… wyoung 222 iterations by 2⁶ = 64 times. It’s easier to use a max-length passphrase
6a3d6fa… wyoung 223 in this situation than get crazy with key derivation iteration counts.
6a3d6fa… wyoung 224
6a3d6fa… wyoung 225 (This, by the way, is why the example passphrase above is 42 characters:
6a3d6fa… wyoung 226 with 6 bits of entropy per character, that gives you a key size of 252,
6a3d6fa… wyoung 227 as close as we can get to our chosen encryption algorithm’s 256-bit key
6a3d6fa… wyoung 228 size without going over. If it pleases you to give it 43 random
6a3d6fa… wyoung 229 characters for a passphrase in order to pick up those last four bits of
6a3d6fa… wyoung 230 security, you’re welcome to do so.)
5c2ef2a… wyoung 231
5c2ef2a… wyoung 232 Compressing the data before encrypting it removes redundancies that can
5c2ef2a… wyoung 233 make decryption easier, and it results in a smaller backup than you get
5c2ef2a… wyoung 234 with the previous script alone, at the expense of a lot of CPU time
ea57a2f… wyoung 235 during the backup. You may wish to switch to a less space-efficient
ea57a2f… wyoung 236 compression algorithm that takes less CPU power, such as [`lz4`][lz4].
ea57a2f… wyoung 237 Changing up the compression algorithm also provides some
ea57a2f… wyoung 238 security-thru-obscurity, which is useless on its own, but it *is* a
ea57a2f… wyoung 239 useful adjunct to strong encryption.
ea57a2f… wyoung 240
8a0ea9b… wyoung 241 This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you
8a0ea9b… wyoung 242 won’t have the `-pbkdf2` and `-iter` options, and you may have to choose
8a0ea9b… wyoung 243 a different cipher algorithm; both changes are likely to weaken the
8a0ea9b… wyoung 244 encryption significantly, so you should install a newer version rather
0e1cc78… wyoung 245 than work around the lack of these features.
0e1cc78… wyoung 246
a55042a… wyoung 247 Beware that macOS ships a fork of OpenSSL called [LibreSSL][lssl] that
a55042a… wyoung 248 lacked this capability until Ventura (13.0). If you’re on Monterey (12)
a55042a… wyoung 249 or older, we recommend use of the [Homebrew][hb] OpenSSL package rather
a55042a… wyoung 250 than give up on the security afforded by use of configurable-iteration
a55042a… wyoung 251 PBKDF2. To avoid a conflict with the platform’s `openssl` binary,
a55042a… wyoung 252 Homebrew’s installation is [unlinked][hbul] by default, so you have to
a55042a… wyoung 253 give an explicit path to it, one of:
a55042a… wyoung 254
8a1ba49… wyoung 255 /usr/local/opt/openssl/bin/openssl ... # Intel x86 Macs
8a1ba49… wyoung 256 /opt/homebrew/opt/openssl/bin/openssl ... # ARM Macs (“Apple silicon”)
8a0ea9b… wyoung 257
0e1cc78… wyoung 258 [lssl]: https://www.libressl.org/
8a0ea9b… wyoung 259
e0a38eb… wyoung 260
8a539f1… wyoung 261 ## <a id="rest"></a> Restoring From An Encrypted Backup
e0a38eb… wyoung 262
e0a38eb… wyoung 263 The “restore” script for the above fragment is basically an inverse of
e0a38eb… wyoung 264 it, but it’s worth showing it because there are some subtleties to take
e0a38eb… wyoung 265 care of. If all variables defined in earlier scripts are available, then
e0a38eb… wyoung 266 restoration is:
e0a38eb… wyoung 267
e0a38eb… wyoung 268 ```
fc300d5… wyoung 269 openssl enc -d -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -in "$gd" |
bcff80c… wyoung 270 xz -d | fossil sql --no-repository ~/museum/restored-repo.fossil
e0a38eb… wyoung 271 ```
e0a38eb… wyoung 272
e0a38eb… wyoung 273 We changed the `-e` to `-d` on the `openssl` command to get decryption,
e0a38eb… wyoung 274 and we changed the `-out` to `-in` so it reads from the encrypted backup
e0a38eb… wyoung 275 file and writes the result to stdout.
e0a38eb… wyoung 276
e0a38eb… wyoung 277 The decompression step is trivial.
e0a38eb… wyoung 278
e0a38eb… wyoung 279 The last change is tricky: we used `fossil sql` above to ensure that
e0a38eb… wyoung 280 we’re using the same version of SQLite to write the encrypted backup DB
ff7cedf… wyoung 281 as was used to maintain the repository. We must also do that on
ff7cedf… wyoung 282 restoration:
ff7cedf… wyoung 283 Fossil serves as a dogfooding project for SQLite,
ff7cedf… wyoung 284 often making use of the latest features, so it is quite likely that a given
ff7cedf… wyoung 285 random `sqlite3` binary in your `PATH` will be unable to understand the
ff7cedf… wyoung 286 file created by “`fossil sql .dump`”! The tricky bit is, you can’t just
bcff80c… wyoung 287 pipe the decrypted SQL dump into `fossil sql`, because on startup, Fossil
ff7cedf… wyoung 288 normally goes looking for tables created by `fossil init`, and it won’t
ff7cedf… wyoung 289 find them in a newly-created repo DB. We get around this by passing
ff7cedf… wyoung 290 the `--no-repository` flag, which suppresses this behavior. Doing it
ff7cedf… wyoung 291 this way saves you from needing to go and build a matching version of
ff7cedf… wyoung 292 `sqlite3` just to restore the backup.
ff7cedf… wyoung 293
c64f28d… drh 294 [bu]: /help/backup
5c2ef2a… wyoung 295 [grcp]: https://www.grc.com/passwords.htm
5c2ef2a… wyoung 296 [hb]: https://brew.sh
5c2ef2a… wyoung 297 [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean
ea57a2f… wyoung 298 [lz4]: https://lz4.github.io/lz4/
78c0c55… wyoung 299 [pbr]: ./private.wiki
6a3d6fa… wyoung 300 [rint]: https://www.random.org/integers/?num=1&min=100000&max=1000000&col=5&base=10&format=html&rnd=new
8a539f1… wyoung 301 [Setup]: ./caps/admin-v-setup.md#apsu
5c2ef2a… wyoung 302 [shun]: ./shunning.wiki
78c0c55… wyoung 303 [tkt]: ./tickets.wiki
5c2ef2a… wyoung 304 [uv]: ./unvers.wiki

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button