Fossil SCM
Several refinements to the backup.md doc, the most substantive of which is that it now warns that even the second method doesn't inherently solve the stale-repo problem in distributed cloning networks.
Commit
8a539f14da59a477526e6d151263ef026517b6c04dc03037193fe1eddf9b1646
Parent
ff7cedfeec93a05…
1 file changed
+24
-13
+24
-13
| --- www/backup.md | ||
| +++ www/backup.md | ||
| @@ -1,11 +1,11 @@ | ||
| 1 | 1 | # Backing Up a Remote Fossil Repository |
| 2 | 2 | |
| 3 | 3 | One of the great benefits of Fossil and other [distributed version control systems][dvcs] |
| 4 | 4 | is that cloning a repository makes a backup. If you are running a project with multiple |
| 5 | 5 | developers who share their work using a [central server][server] and the server hardware |
| 6 | -catches fire or otherwise becomes unavailable, the clones of the repository on each developer | |
| 6 | +catches fire, the clones of the repository on each developer | |
| 7 | 7 | workstation *may* serve as a suitable backup. |
| 8 | 8 | |
| 9 | 9 | [dvcs]: wikipedia:/wiki/Distributed_version_control |
| 10 | 10 | [server]: ./server/whyuseaserver.wiki |
| 11 | 11 | |
| @@ -15,20 +15,20 @@ | ||
| 15 | 15 | that, a Fossil repository typically contains |
| 16 | 16 | other useful information that is not always shared as part of a clone, which might need |
| 17 | 17 | to be backed up separately. To wit: |
| 18 | 18 | |
| 19 | 19 | |
| 20 | -## Sensitive Information | |
| 20 | +## <a id="pii"></a> Sensitive Information | |
| 21 | 21 | |
| 22 | 22 | Fossil purposefully does not clone certain sensitive information unless |
| 23 | -you’re logged in as a user with [setup] capability. As an example, a local clone | |
| 23 | +you’re logged in as a user with [Setup] capability. As an example, a local clone | |
| 24 | 24 | may have a different `user` table than the remote, because only a |
| 25 | 25 | Setup user is allowed to see the full version for privacy and security |
| 26 | 26 | reasons. |
| 27 | 27 | |
| 28 | 28 | |
| 29 | -## Configuration Drift | |
| 29 | +## <a id="config"></a> Configuration Drift | |
| 30 | 30 | |
| 31 | 31 | Fossil allows the local configuration in certain areas to differ from |
| 32 | 32 | that of the remote. With the exception of the prior item, you get a copy |
| 33 | 33 | of these configuration areas on initial clone, but after that, some |
| 34 | 34 | remote configuration changes don’t sync down automatically, such as the |
| @@ -35,37 +35,37 @@ | ||
| 35 | 35 | remote’s skin. You can ask for updates by running the |
| 36 | 36 | [`fossil config pull skin`](./help?cmd=config) command, but that |
| 37 | 37 | does not happen automatically during the course of normal development. |
| 38 | 38 | |
| 39 | 39 | |
| 40 | -## Private Branches | |
| 40 | +## <a id="private"></a> Private Branches | |
| 41 | 41 | |
| 42 | 42 | The very nature of Fossil’s [private branch feature][pbr] ensures that |
| 43 | 43 | remote clones don’t get a copy of those branches. Normally this is |
| 44 | 44 | exactly what you want, but in the case of making backups, you probably |
| 45 | 45 | want to back up these branches as well. One of the two backup methods below |
| 46 | 46 | provides this. |
| 47 | 47 | |
| 48 | 48 | |
| 49 | -## Shunned Artifacts | |
| 49 | +## <a id="shun"></a> Shunned Artifacts | |
| 50 | 50 | |
| 51 | 51 | Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want |
| 52 | 52 | your local clone to be a precise match to the remote, it needs to track |
| 53 | 53 | changes to the shun table as well. |
| 54 | 54 | |
| 55 | 55 | |
| 56 | -## Unversioned Artifacts | |
| 56 | +## <a id="uv"></a> Unversioned Artifacts | |
| 57 | 57 | |
| 58 | 58 | Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by |
| 59 | 59 | default unless you specifically ask for it. Like local configuration |
| 60 | 60 | data, it doesn’t get pulled as part of a normal `fossil sync`, but |
| 61 | 61 | *unlike* the config data, you don’t get unversioned files as part of the |
| 62 | 62 | initial clone unless you ask for it by passing the `--unversioned/-u` |
| 63 | 63 | flag. |
| 64 | 64 | |
| 65 | 65 | |
| 66 | -## Autosync Is Intransitive | |
| 66 | +## <a id="ait"></a>Autosync Is Intransitive | |
| 67 | 67 | |
| 68 | 68 | If you’re using Fossil in a truly distributed mode, rather than the |
| 69 | 69 | simple central-and-clones model that is more common, there may be no |
| 70 | 70 | single source of truth in the network because Fossil’s autosync feature |
| 71 | 71 | isn’t transitive. |
| @@ -92,11 +92,11 @@ | ||
| 92 | 92 | source of truth, those users still syncing with `svr2` won’t have their |
| 93 | 93 | commits pushed up to `svr1` unless you’ve set up bidirectional sync, |
| 94 | 94 | rather than have the two backup servers do `pull` only. |
| 95 | 95 | |
| 96 | 96 | |
| 97 | -# Solutions | |
| 97 | +# Solution 1: Explicit Pulls | |
| 98 | 98 | |
| 99 | 99 | The following script solves most of the above problems for the use case |
| 100 | 100 | where you want a *nearly-complete* clone of the remote repository using nothing |
| 101 | 101 | but the normal Fossil sync protocol. It only does so if you are logged into |
| 102 | 102 | the remote as a user with Setup capability, however. |
| @@ -118,11 +118,14 @@ | ||
| 118 | 118 | out of existence, your backup will be “more than complete” in the sense |
| 119 | 119 | that it will continue to have information that the remote says should |
| 120 | 120 | not exist any more. That would be not so much a “backup” as an |
| 121 | 121 | “archive,” which might not be what you want. |
| 122 | 122 | |
| 123 | -This method doesn’t get you a copy of the remote’s | |
| 123 | + | |
| 124 | +# Solution 2: SQL-Level Backup | |
| 125 | + | |
| 126 | +The first method doesn’t get you a copy of the remote’s | |
| 124 | 127 | [private branches][pbr], on purpose. It may also miss other info on the |
| 125 | 128 | remote, such as SQL-level customizations that the sync protocol can’t |
| 126 | 129 | see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can |
| 127 | 130 | solve such problems if you have access to the remote server, which |
| 128 | 131 | allows you to get a SQL-level backup. This requires Fossil 2.12 or |
| @@ -139,12 +142,20 @@ | ||
| 139 | 142 | bf=repo-$(date +%Y-%m-%d).fossil |
| 140 | 143 | ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" && |
| 141 | 144 | scp example.com:museum/backups/$bf ~/museum/backups |
| 142 | 145 | ``` |
| 143 | 146 | |
| 147 | +---- | |
| 148 | + | |
| 149 | +Beware that this method does not solve [the intransitive sync | |
| 150 | +problem](#ait), in and of itself: if you do a SQL-level backup of a | |
| 151 | +stale repo DB, you have a *stale backup!* You should therefore run this | |
| 152 | +on every node that may need to serve as a backup so that at least *one* | |
| 153 | +of the backups is also up-to-date. | |
| 154 | + | |
| 144 | 155 | |
| 145 | -## Encrypted Off-Site Backups | |
| 156 | +# <a id="enc"></a> Encrypted Off-Site Backups | |
| 146 | 157 | |
| 147 | 158 | A useful refinement that you can apply to both methods above is |
| 148 | 159 | encrypted off-site backups. You may wish to store backups of your |
| 149 | 160 | repositories off-site on a service such as Dropbox, Google Drive, iCloud, |
| 150 | 161 | or Microsoft OneDrive, where you don’t fully trust the service not to |
| @@ -193,11 +204,11 @@ | ||
| 193 | 204 | Changing up the compression algorithm also provides some |
| 194 | 205 | security-thru-obscurity, which is useless on its own, but it *is* a |
| 195 | 206 | useful adjunct to strong encryption. |
| 196 | 207 | |
| 197 | 208 | |
| 198 | -## Restoring From An Encrypted Backup | |
| 209 | +## <a id="rest"></a> Restoring From An Encrypted Backup | |
| 199 | 210 | |
| 200 | 211 | The “restore” script for the above fragment is basically an inverse of |
| 201 | 212 | it, but it’s worth showing it because there are some subtleties to take |
| 202 | 213 | care of. If all variables defined in earlier scripts are available, then |
| 203 | 214 | restoration is: |
| @@ -233,9 +244,9 @@ | ||
| 233 | 244 | [hb]: https://brew.sh |
| 234 | 245 | [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean |
| 235 | 246 | [lz4]: https://lz4.github.io/lz4/ |
| 236 | 247 | [pbr]: ./private.wiki |
| 237 | 248 | [rint]: https://www.random.org/integers/?num=1&min=10000&max=100000&col=5&base=10&format=html&rnd=new |
| 238 | -[setup]: ./caps/admin-v-setup.md#apsu | |
| 249 | +[Setup]: ./caps/admin-v-setup.md#apsu | |
| 239 | 250 | [shun]: ./shunning.wiki |
| 240 | 251 | [tkt]: ./tickets.wiki |
| 241 | 252 | [uv]: ./unvers.wiki |
| 242 | 253 |
| --- www/backup.md | |
| +++ www/backup.md | |
| @@ -1,11 +1,11 @@ | |
| 1 | # Backing Up a Remote Fossil Repository |
| 2 | |
| 3 | One of the great benefits of Fossil and other [distributed version control systems][dvcs] |
| 4 | is that cloning a repository makes a backup. If you are running a project with multiple |
| 5 | developers who share their work using a [central server][server] and the server hardware |
| 6 | catches fire or otherwise becomes unavailable, the clones of the repository on each developer |
| 7 | workstation *may* serve as a suitable backup. |
| 8 | |
| 9 | [dvcs]: wikipedia:/wiki/Distributed_version_control |
| 10 | [server]: ./server/whyuseaserver.wiki |
| 11 | |
| @@ -15,20 +15,20 @@ | |
| 15 | that, a Fossil repository typically contains |
| 16 | other useful information that is not always shared as part of a clone, which might need |
| 17 | to be backed up separately. To wit: |
| 18 | |
| 19 | |
| 20 | ## Sensitive Information |
| 21 | |
| 22 | Fossil purposefully does not clone certain sensitive information unless |
| 23 | you’re logged in as a user with [setup] capability. As an example, a local clone |
| 24 | may have a different `user` table than the remote, because only a |
| 25 | Setup user is allowed to see the full version for privacy and security |
| 26 | reasons. |
| 27 | |
| 28 | |
| 29 | ## Configuration Drift |
| 30 | |
| 31 | Fossil allows the local configuration in certain areas to differ from |
| 32 | that of the remote. With the exception of the prior item, you get a copy |
| 33 | of these configuration areas on initial clone, but after that, some |
| 34 | remote configuration changes don’t sync down automatically, such as the |
| @@ -35,37 +35,37 @@ | |
| 35 | remote’s skin. You can ask for updates by running the |
| 36 | [`fossil config pull skin`](./help?cmd=config) command, but that |
| 37 | does not happen automatically during the course of normal development. |
| 38 | |
| 39 | |
| 40 | ## Private Branches |
| 41 | |
| 42 | The very nature of Fossil’s [private branch feature][pbr] ensures that |
| 43 | remote clones don’t get a copy of those branches. Normally this is |
| 44 | exactly what you want, but in the case of making backups, you probably |
| 45 | want to back up these branches as well. One of the two backup methods below |
| 46 | provides this. |
| 47 | |
| 48 | |
| 49 | ## Shunned Artifacts |
| 50 | |
| 51 | Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want |
| 52 | your local clone to be a precise match to the remote, it needs to track |
| 53 | changes to the shun table as well. |
| 54 | |
| 55 | |
| 56 | ## Unversioned Artifacts |
| 57 | |
| 58 | Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by |
| 59 | default unless you specifically ask for it. Like local configuration |
| 60 | data, it doesn’t get pulled as part of a normal `fossil sync`, but |
| 61 | *unlike* the config data, you don’t get unversioned files as part of the |
| 62 | initial clone unless you ask for it by passing the `--unversioned/-u` |
| 63 | flag. |
| 64 | |
| 65 | |
| 66 | ## Autosync Is Intransitive |
| 67 | |
| 68 | If you’re using Fossil in a truly distributed mode, rather than the |
| 69 | simple central-and-clones model that is more common, there may be no |
| 70 | single source of truth in the network because Fossil’s autosync feature |
| 71 | isn’t transitive. |
| @@ -92,11 +92,11 @@ | |
| 92 | source of truth, those users still syncing with `svr2` won’t have their |
| 93 | commits pushed up to `svr1` unless you’ve set up bidirectional sync, |
| 94 | rather than have the two backup servers do `pull` only. |
| 95 | |
| 96 | |
| 97 | # Solutions |
| 98 | |
| 99 | The following script solves most of the above problems for the use case |
| 100 | where you want a *nearly-complete* clone of the remote repository using nothing |
| 101 | but the normal Fossil sync protocol. It only does so if you are logged into |
| 102 | the remote as a user with Setup capability, however. |
| @@ -118,11 +118,14 @@ | |
| 118 | out of existence, your backup will be “more than complete” in the sense |
| 119 | that it will continue to have information that the remote says should |
| 120 | not exist any more. That would be not so much a “backup” as an |
| 121 | “archive,” which might not be what you want. |
| 122 | |
| 123 | This method doesn’t get you a copy of the remote’s |
| 124 | [private branches][pbr], on purpose. It may also miss other info on the |
| 125 | remote, such as SQL-level customizations that the sync protocol can’t |
| 126 | see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can |
| 127 | solve such problems if you have access to the remote server, which |
| 128 | allows you to get a SQL-level backup. This requires Fossil 2.12 or |
| @@ -139,12 +142,20 @@ | |
| 139 | bf=repo-$(date +%Y-%m-%d).fossil |
| 140 | ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" && |
| 141 | scp example.com:museum/backups/$bf ~/museum/backups |
| 142 | ``` |
| 143 | |
| 144 | |
| 145 | ## Encrypted Off-Site Backups |
| 146 | |
| 147 | A useful refinement that you can apply to both methods above is |
| 148 | encrypted off-site backups. You may wish to store backups of your |
| 149 | repositories off-site on a service such as Dropbox, Google Drive, iCloud, |
| 150 | or Microsoft OneDrive, where you don’t fully trust the service not to |
| @@ -193,11 +204,11 @@ | |
| 193 | Changing up the compression algorithm also provides some |
| 194 | security-thru-obscurity, which is useless on its own, but it *is* a |
| 195 | useful adjunct to strong encryption. |
| 196 | |
| 197 | |
| 198 | ## Restoring From An Encrypted Backup |
| 199 | |
| 200 | The “restore” script for the above fragment is basically an inverse of |
| 201 | it, but it’s worth showing it because there are some subtleties to take |
| 202 | care of. If all variables defined in earlier scripts are available, then |
| 203 | restoration is: |
| @@ -233,9 +244,9 @@ | |
| 233 | [hb]: https://brew.sh |
| 234 | [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean |
| 235 | [lz4]: https://lz4.github.io/lz4/ |
| 236 | [pbr]: ./private.wiki |
| 237 | [rint]: https://www.random.org/integers/?num=1&min=10000&max=100000&col=5&base=10&format=html&rnd=new |
| 238 | [setup]: ./caps/admin-v-setup.md#apsu |
| 239 | [shun]: ./shunning.wiki |
| 240 | [tkt]: ./tickets.wiki |
| 241 | [uv]: ./unvers.wiki |
| 242 |
| --- www/backup.md | |
| +++ www/backup.md | |
| @@ -1,11 +1,11 @@ | |
| 1 | # Backing Up a Remote Fossil Repository |
| 2 | |
| 3 | One of the great benefits of Fossil and other [distributed version control systems][dvcs] |
| 4 | is that cloning a repository makes a backup. If you are running a project with multiple |
| 5 | developers who share their work using a [central server][server] and the server hardware |
| 6 | catches fire, the clones of the repository on each developer |
| 7 | workstation *may* serve as a suitable backup. |
| 8 | |
| 9 | [dvcs]: wikipedia:/wiki/Distributed_version_control |
| 10 | [server]: ./server/whyuseaserver.wiki |
| 11 | |
| @@ -15,20 +15,20 @@ | |
| 15 | that, a Fossil repository typically contains |
| 16 | other useful information that is not always shared as part of a clone, which might need |
| 17 | to be backed up separately. To wit: |
| 18 | |
| 19 | |
| 20 | ## <a id="pii"></a> Sensitive Information |
| 21 | |
| 22 | Fossil purposefully does not clone certain sensitive information unless |
| 23 | you’re logged in as a user with [Setup] capability. As an example, a local clone |
| 24 | may have a different `user` table than the remote, because only a |
| 25 | Setup user is allowed to see the full version for privacy and security |
| 26 | reasons. |
| 27 | |
| 28 | |
| 29 | ## <a id="config"></a> Configuration Drift |
| 30 | |
| 31 | Fossil allows the local configuration in certain areas to differ from |
| 32 | that of the remote. With the exception of the prior item, you get a copy |
| 33 | of these configuration areas on initial clone, but after that, some |
| 34 | remote configuration changes don’t sync down automatically, such as the |
| @@ -35,37 +35,37 @@ | |
| 35 | remote’s skin. You can ask for updates by running the |
| 36 | [`fossil config pull skin`](./help?cmd=config) command, but that |
| 37 | does not happen automatically during the course of normal development. |
| 38 | |
| 39 | |
| 40 | ## <a id="private"></a> Private Branches |
| 41 | |
| 42 | The very nature of Fossil’s [private branch feature][pbr] ensures that |
| 43 | remote clones don’t get a copy of those branches. Normally this is |
| 44 | exactly what you want, but in the case of making backups, you probably |
| 45 | want to back up these branches as well. One of the two backup methods below |
| 46 | provides this. |
| 47 | |
| 48 | |
| 49 | ## <a id="shun"></a> Shunned Artifacts |
| 50 | |
| 51 | Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want |
| 52 | your local clone to be a precise match to the remote, it needs to track |
| 53 | changes to the shun table as well. |
| 54 | |
| 55 | |
| 56 | ## <a id="uv"></a> Unversioned Artifacts |
| 57 | |
| 58 | Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by |
| 59 | default unless you specifically ask for it. Like local configuration |
| 60 | data, it doesn’t get pulled as part of a normal `fossil sync`, but |
| 61 | *unlike* the config data, you don’t get unversioned files as part of the |
| 62 | initial clone unless you ask for it by passing the `--unversioned/-u` |
| 63 | flag. |
| 64 | |
| 65 | |
| 66 | ## <a id="ait"></a>Autosync Is Intransitive |
| 67 | |
| 68 | If you’re using Fossil in a truly distributed mode, rather than the |
| 69 | simple central-and-clones model that is more common, there may be no |
| 70 | single source of truth in the network because Fossil’s autosync feature |
| 71 | isn’t transitive. |
| @@ -92,11 +92,11 @@ | |
| 92 | source of truth, those users still syncing with `svr2` won’t have their |
| 93 | commits pushed up to `svr1` unless you’ve set up bidirectional sync, |
| 94 | rather than have the two backup servers do `pull` only. |
| 95 | |
| 96 | |
| 97 | # Solution 1: Explicit Pulls |
| 98 | |
| 99 | The following script solves most of the above problems for the use case |
| 100 | where you want a *nearly-complete* clone of the remote repository using nothing |
| 101 | but the normal Fossil sync protocol. It only does so if you are logged into |
| 102 | the remote as a user with Setup capability, however. |
| @@ -118,11 +118,14 @@ | |
| 118 | out of existence, your backup will be “more than complete” in the sense |
| 119 | that it will continue to have information that the remote says should |
| 120 | not exist any more. That would be not so much a “backup” as an |
| 121 | “archive,” which might not be what you want. |
| 122 | |
| 123 | |
| 124 | # Solution 2: SQL-Level Backup |
| 125 | |
| 126 | The first method doesn’t get you a copy of the remote’s |
| 127 | [private branches][pbr], on purpose. It may also miss other info on the |
| 128 | remote, such as SQL-level customizations that the sync protocol can’t |
| 129 | see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can |
| 130 | solve such problems if you have access to the remote server, which |
| 131 | allows you to get a SQL-level backup. This requires Fossil 2.12 or |
| @@ -139,12 +142,20 @@ | |
| 142 | bf=repo-$(date +%Y-%m-%d).fossil |
| 143 | ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" && |
| 144 | scp example.com:museum/backups/$bf ~/museum/backups |
| 145 | ``` |
| 146 | |
| 147 | ---- |
| 148 | |
| 149 | Beware that this method does not solve [the intransitive sync |
| 150 | problem](#ait), in and of itself: if you do a SQL-level backup of a |
| 151 | stale repo DB, you have a *stale backup!* You should therefore run this |
| 152 | on every node that may need to serve as a backup so that at least *one* |
| 153 | of the backups is also up-to-date. |
| 154 | |
| 155 | |
| 156 | # <a id="enc"></a> Encrypted Off-Site Backups |
| 157 | |
| 158 | A useful refinement that you can apply to both methods above is |
| 159 | encrypted off-site backups. You may wish to store backups of your |
| 160 | repositories off-site on a service such as Dropbox, Google Drive, iCloud, |
| 161 | or Microsoft OneDrive, where you don’t fully trust the service not to |
| @@ -193,11 +204,11 @@ | |
| 204 | Changing up the compression algorithm also provides some |
| 205 | security-thru-obscurity, which is useless on its own, but it *is* a |
| 206 | useful adjunct to strong encryption. |
| 207 | |
| 208 | |
| 209 | ## <a id="rest"></a> Restoring From An Encrypted Backup |
| 210 | |
| 211 | The “restore” script for the above fragment is basically an inverse of |
| 212 | it, but it’s worth showing it because there are some subtleties to take |
| 213 | care of. If all variables defined in earlier scripts are available, then |
| 214 | restoration is: |
| @@ -233,9 +244,9 @@ | |
| 244 | [hb]: https://brew.sh |
| 245 | [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean |
| 246 | [lz4]: https://lz4.github.io/lz4/ |
| 247 | [pbr]: ./private.wiki |
| 248 | [rint]: https://www.random.org/integers/?num=1&min=10000&max=100000&col=5&base=10&format=html&rnd=new |
| 249 | [Setup]: ./caps/admin-v-setup.md#apsu |
| 250 | [shun]: ./shunning.wiki |
| 251 | [tkt]: ./tickets.wiki |
| 252 | [uv]: ./unvers.wiki |
| 253 |