Fossil SCM

Assorted minor tweaks to the new backup.md doc

wyoung 2020-10-13 08:32 trunk
Commit fc2c12442ca27953cbf1af106835d3ac7a7c60911321a18e94f42a6249c9a58b
1 file changed +31 -25
+31 -25
--- www/backup.md
+++ www/backup.md
@@ -1,28 +1,29 @@
11
# Backing Up a Remote Fossil Repository
22
33
One of the great benefits of Fossil and other [distributed version control systems][dvcs]
4
-is that cloning a repository makes a backup. So if you are running a project with multiple
4
+is that cloning a repository makes a backup. If you are running a project with multiple
55
developers who share their work using a [central server][server] and the server hardware
66
catches fire or otherwise becomes unavailable, the clones of the repository on each developer
7
-workstation serve as a suitable backup.
7
+workstation *may* serve as a suitable backup.
88
99
[dvcs]: wikipedia:/wiki/Distributed_version_control
1010
[server]: ./server/whyuseaserver.wiki
1111
12
-Mostly.
13
-
14
-It turns out not everything in a Fossil repository is copied via clone. A clone
15
-protects all of the checked in code. But a Fossil repository typically contains
16
-other useful information that is not always shared as part of a clone, and might need
12
+We say “may” because
13
+it turns out not everything in a Fossil repository is copied when cloning. You
14
+don’t even always get copies of all historical file artifacts. More than
15
+that, a Fossil repository typically contains
16
+other useful information that is not always shared as part of a clone, which might need
1717
to be backed up separately. To wit:
18
+
1819
1920
## Sensitive Information
2021
2122
Fossil purposefully does not clone certain sensitive information unless
22
-you’re logged in with [setup] capability. As an example, a local clone
23
-may have a different `user` table than the remote, because only the
23
+you’re logged in as a user with [setup] capability. As an example, a local clone
24
+may have a different `user` table than the remote, because only a
2425
Setup user is allowed to see the full version for privacy and security
2526
reasons.
2627
2728
2829
## Configuration Drift
@@ -30,29 +31,28 @@
3031
Fossil allows the local configuration in certain areas to differ from
3132
that of the remote. With the exception of the prior item, you get a copy
3233
of these configuration areas on initial clone, but after that, some
3334
remote configuration changes don’t sync down automatically, such as the
3435
remote’s skin. You can ask for updates by running the
35
-[`fossil config pull skiin`](./help?cmd=config) if you want, but that
36
+[`fossil config pull skiin`](./help?cmd=config) command, but that
3637
does not happen automatically during the course of normal development.
3738
3839
3940
## Private Branches
4041
4142
The very nature of Fossil’s [private branch feature][pbr] ensures that
4243
remote clones don’t get a copy of those branches. Normally this is
4344
exactly what you want, but in the case of making backups, you probably
44
-want these branches as well. One of the two backup methods below
45
+want to back up these branches as well. One of the two backup methods below
4546
provides this.
4647
4748
4849
## Shunned Artifacts
4950
5051
Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want
5152
your local clone to be a precise match to the remote, it needs to track
5253
changes to the shun table as well.
53
-
5454
5555
5656
## Unversioned Artifacts
5757
5858
Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by
@@ -69,15 +69,15 @@
6969
simple central-and-clones model that is more common, there may be no
7070
single source of truth in the network because Fossil’s autosync feature
7171
isn’t transitive.
7272
7373
That is, if you cloned from server A, and then you stand that up on a
74
-server B, then if I clone from you as repository C, changes to B
74
+server B, then if I clone from your server as my repository C, your changes to B
7575
autosync up to A, but not down to me on C until I do something locally
7676
that triggers autosync. The inverse is also true: if I commit something
7777
on C, it will autosync up to B, but A won’t get a copy until someone on
78
-B does something to trigger autosync there.
78
+B does something to trigger a sync there.
7979
8080
An easy way to run into this problem is to set up failover servers
8181
`svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync
8282
with the first. If all of the users normally clone from `svr1`, their
8383
commits don’t get to `svr2` and `svr3` until something on one of the
@@ -96,12 +96,12 @@
9696
9797
# Solutions
9898
9999
The following script solves most of the above problems for the use case
100100
where you want a *nearly-complete* clone of the remote repository using nothing
101
-but the normal Fossil sync protocol. It requires that you be logged into
102
-the remote as a user with Setup capability.
101
+but the normal Fossil sync protocol. It only does so if you are logged into
102
+the remote as a user with Setup capability, however.
103103
104104
----
105105
106106
``` shell
107107
#!/bin/sh
@@ -112,22 +112,24 @@
112112
113113
----
114114
115115
The last step is needed to ensure that shunned artifacts on the remote
116116
are removed from the local clone. The second step includes
117
-`fossil conf pull shun`, so your repo won’t offer the shunned artifacts
118
-to others cloning from it, but the backup can’t be said to be “complete”
119
-if it contains information that the remote now lacks.
117
+`fossil conf pull shun`, but until those artifacts are actually rebuilt
118
+out of existence, your backup will be “more than complete” in the sense
119
+that it will continue to have information that the remote says should
120
+not exist any more. That would be not so much a “backup” as an
121
+“archive,” which might not be what you want.
120122
121123
This method doesn’t get you a copy of the remote’s
122124
[private branches][pbr], on purpose. It may also miss other info on the
123125
remote, such as SQL-level customizations that the sync protocol can’t
124
-see. (Some [ticket system customization][tkt] schemes do this.) You can
126
+see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can
125127
solve such problems if you have access to the remote server, which
126128
allows you to get a SQL-level backup. This requires Fossil 2.12 or
127
-newer, which added [the `backup` command][bu], which takes care of
128
-locking and transaction isolation to allow backing up an in-use
129
+newer, which added [the `backup` command][bu] to take care of
130
+locking and transaction isolation, allowing the user to safely back up an in-use
129131
repository.
130132
131133
If you have SSH access to the remote server, something like this will work:
132134
133135
----
@@ -167,11 +169,13 @@
167169
168170
This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you
169171
won’t have the `-pbkdf2` and `-iter` options, and you may have to choose
170172
a different cipher algorithm; both changes are likely to weaken the
171173
encryption significantly, so you should install a newer version rather
172
-than work around the lack of these features. If you’re on macOS, which
174
+than work around the lack of these features.
175
+
176
+If you’re on macOS, which
173177
still ships 1.0 as of the time of this writing, [Homebrew][hb] offers
174178
the current version of OpenSSL, but to avoid a conflict with the platform
175179
version it’s [unlinked][hbul] by default, so you have to give an explicit
176180
path to its “cellar” directory:
177181
@@ -211,19 +215,21 @@
211215
212216
The last change is tricky: we used `fossil sql` above to ensure that
213217
we’re using the same version of SQLite to write the encrypted backup DB
214218
as was used to maintain the repository, but unfortunately, we can’t get
215219
the built-in SQLite shell to write a backup into an empty database.
220
+(As soon as it starts up, it goes looking for tables created by
221
+`fossil init` and fails with an error.)
216222
Therefore, we have to either run the restoration against a
217223
possibly-different version of SQLite and hope there are no
218224
incompatibilities, or we have to go out of our way to build a matching
219225
version of `sqlite3` before we can safely do the restoration.
220226
221227
Keep in mind that Fossil often acts as a dogfooding project for SQLite,
222228
making use of the latest features, so it is quite likely that a given
223
-random `sqlite3` binary in your `PATH` may be unable to understand the
224
-dump file created by the backup script!
229
+random `sqlite3` binary in your `PATH` will be unable to understand the
230
+file created by “`fossil sql .dump`”!
225231
226232
[bu]: /help?cmd=backup
227233
[grcp]: https://www.grc.com/passwords.htm
228234
[hb]: https://brew.sh
229235
[hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean
230236
--- www/backup.md
+++ www/backup.md
@@ -1,28 +1,29 @@
1 # Backing Up a Remote Fossil Repository
2
3 One of the great benefits of Fossil and other [distributed version control systems][dvcs]
4 is that cloning a repository makes a backup. So if you are running a project with multiple
5 developers who share their work using a [central server][server] and the server hardware
6 catches fire or otherwise becomes unavailable, the clones of the repository on each developer
7 workstation serve as a suitable backup.
8
9 [dvcs]: wikipedia:/wiki/Distributed_version_control
10 [server]: ./server/whyuseaserver.wiki
11
12 Mostly.
13
14 It turns out not everything in a Fossil repository is copied via clone. A clone
15 protects all of the checked in code. But a Fossil repository typically contains
16 other useful information that is not always shared as part of a clone, and might need
17 to be backed up separately. To wit:
 
18
19 ## Sensitive Information
20
21 Fossil purposefully does not clone certain sensitive information unless
22 you’re logged in with [setup] capability. As an example, a local clone
23 may have a different `user` table than the remote, because only the
24 Setup user is allowed to see the full version for privacy and security
25 reasons.
26
27
28 ## Configuration Drift
@@ -30,29 +31,28 @@
30 Fossil allows the local configuration in certain areas to differ from
31 that of the remote. With the exception of the prior item, you get a copy
32 of these configuration areas on initial clone, but after that, some
33 remote configuration changes don’t sync down automatically, such as the
34 remote’s skin. You can ask for updates by running the
35 [`fossil config pull skiin`](./help?cmd=config) if you want, but that
36 does not happen automatically during the course of normal development.
37
38
39 ## Private Branches
40
41 The very nature of Fossil’s [private branch feature][pbr] ensures that
42 remote clones don’t get a copy of those branches. Normally this is
43 exactly what you want, but in the case of making backups, you probably
44 want these branches as well. One of the two backup methods below
45 provides this.
46
47
48 ## Shunned Artifacts
49
50 Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want
51 your local clone to be a precise match to the remote, it needs to track
52 changes to the shun table as well.
53
54
55
56 ## Unversioned Artifacts
57
58 Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by
@@ -69,15 +69,15 @@
69 simple central-and-clones model that is more common, there may be no
70 single source of truth in the network because Fossil’s autosync feature
71 isn’t transitive.
72
73 That is, if you cloned from server A, and then you stand that up on a
74 server B, then if I clone from you as repository C, changes to B
75 autosync up to A, but not down to me on C until I do something locally
76 that triggers autosync. The inverse is also true: if I commit something
77 on C, it will autosync up to B, but A won’t get a copy until someone on
78 B does something to trigger autosync there.
79
80 An easy way to run into this problem is to set up failover servers
81 `svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync
82 with the first. If all of the users normally clone from `svr1`, their
83 commits don’t get to `svr2` and `svr3` until something on one of the
@@ -96,12 +96,12 @@
96
97 # Solutions
98
99 The following script solves most of the above problems for the use case
100 where you want a *nearly-complete* clone of the remote repository using nothing
101 but the normal Fossil sync protocol. It requires that you be logged into
102 the remote as a user with Setup capability.
103
104 ----
105
106 ``` shell
107 #!/bin/sh
@@ -112,22 +112,24 @@
112
113 ----
114
115 The last step is needed to ensure that shunned artifacts on the remote
116 are removed from the local clone. The second step includes
117 `fossil conf pull shun`, so your repo won’t offer the shunned artifacts
118 to others cloning from it, but the backup can’t be said to be “complete”
119 if it contains information that the remote now lacks.
 
 
120
121 This method doesn’t get you a copy of the remote’s
122 [private branches][pbr], on purpose. It may also miss other info on the
123 remote, such as SQL-level customizations that the sync protocol can’t
124 see. (Some [ticket system customization][tkt] schemes do this.) You can
125 solve such problems if you have access to the remote server, which
126 allows you to get a SQL-level backup. This requires Fossil 2.12 or
127 newer, which added [the `backup` command][bu], which takes care of
128 locking and transaction isolation to allow backing up an in-use
129 repository.
130
131 If you have SSH access to the remote server, something like this will work:
132
133 ----
@@ -167,11 +169,13 @@
167
168 This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you
169 won’t have the `-pbkdf2` and `-iter` options, and you may have to choose
170 a different cipher algorithm; both changes are likely to weaken the
171 encryption significantly, so you should install a newer version rather
172 than work around the lack of these features. If you’re on macOS, which
 
 
173 still ships 1.0 as of the time of this writing, [Homebrew][hb] offers
174 the current version of OpenSSL, but to avoid a conflict with the platform
175 version it’s [unlinked][hbul] by default, so you have to give an explicit
176 path to its “cellar” directory:
177
@@ -211,19 +215,21 @@
211
212 The last change is tricky: we used `fossil sql` above to ensure that
213 we’re using the same version of SQLite to write the encrypted backup DB
214 as was used to maintain the repository, but unfortunately, we can’t get
215 the built-in SQLite shell to write a backup into an empty database.
 
 
216 Therefore, we have to either run the restoration against a
217 possibly-different version of SQLite and hope there are no
218 incompatibilities, or we have to go out of our way to build a matching
219 version of `sqlite3` before we can safely do the restoration.
220
221 Keep in mind that Fossil often acts as a dogfooding project for SQLite,
222 making use of the latest features, so it is quite likely that a given
223 random `sqlite3` binary in your `PATH` may be unable to understand the
224 dump file created by the backup script!
225
226 [bu]: /help?cmd=backup
227 [grcp]: https://www.grc.com/passwords.htm
228 [hb]: https://brew.sh
229 [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean
230
--- www/backup.md
+++ www/backup.md
@@ -1,28 +1,29 @@
1 # Backing Up a Remote Fossil Repository
2
3 One of the great benefits of Fossil and other [distributed version control systems][dvcs]
4 is that cloning a repository makes a backup. If you are running a project with multiple
5 developers who share their work using a [central server][server] and the server hardware
6 catches fire or otherwise becomes unavailable, the clones of the repository on each developer
7 workstation *may* serve as a suitable backup.
8
9 [dvcs]: wikipedia:/wiki/Distributed_version_control
10 [server]: ./server/whyuseaserver.wiki
11
12 We say “may” because
13 it turns out not everything in a Fossil repository is copied when cloning. You
14 don’t even always get copies of all historical file artifacts. More than
15 that, a Fossil repository typically contains
16 other useful information that is not always shared as part of a clone, which might need
17 to be backed up separately. To wit:
18
19
20 ## Sensitive Information
21
22 Fossil purposefully does not clone certain sensitive information unless
23 you’re logged in as a user with [setup] capability. As an example, a local clone
24 may have a different `user` table than the remote, because only a
25 Setup user is allowed to see the full version for privacy and security
26 reasons.
27
28
29 ## Configuration Drift
@@ -30,29 +31,28 @@
31 Fossil allows the local configuration in certain areas to differ from
32 that of the remote. With the exception of the prior item, you get a copy
33 of these configuration areas on initial clone, but after that, some
34 remote configuration changes don’t sync down automatically, such as the
35 remote’s skin. You can ask for updates by running the
36 [`fossil config pull skiin`](./help?cmd=config) command, but that
37 does not happen automatically during the course of normal development.
38
39
40 ## Private Branches
41
42 The very nature of Fossil’s [private branch feature][pbr] ensures that
43 remote clones don’t get a copy of those branches. Normally this is
44 exactly what you want, but in the case of making backups, you probably
45 want to back up these branches as well. One of the two backup methods below
46 provides this.
47
48
49 ## Shunned Artifacts
50
51 Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want
52 your local clone to be a precise match to the remote, it needs to track
53 changes to the shun table as well.
 
54
55
56 ## Unversioned Artifacts
57
58 Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by
@@ -69,15 +69,15 @@
69 simple central-and-clones model that is more common, there may be no
70 single source of truth in the network because Fossil’s autosync feature
71 isn’t transitive.
72
73 That is, if you cloned from server A, and then you stand that up on a
74 server B, then if I clone from your server as my repository C, your changes to B
75 autosync up to A, but not down to me on C until I do something locally
76 that triggers autosync. The inverse is also true: if I commit something
77 on C, it will autosync up to B, but A won’t get a copy until someone on
78 B does something to trigger a sync there.
79
80 An easy way to run into this problem is to set up failover servers
81 `svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync
82 with the first. If all of the users normally clone from `svr1`, their
83 commits don’t get to `svr2` and `svr3` until something on one of the
@@ -96,12 +96,12 @@
96
97 # Solutions
98
99 The following script solves most of the above problems for the use case
100 where you want a *nearly-complete* clone of the remote repository using nothing
101 but the normal Fossil sync protocol. It only does so if you are logged into
102 the remote as a user with Setup capability, however.
103
104 ----
105
106 ``` shell
107 #!/bin/sh
@@ -112,22 +112,24 @@
112
113 ----
114
115 The last step is needed to ensure that shunned artifacts on the remote
116 are removed from the local clone. The second step includes
117 `fossil conf pull shun`, but until those artifacts are actually rebuilt
118 out of existence, your backup will be “more than complete” in the sense
119 that it will continue to have information that the remote says should
120 not exist any more. That would be not so much a “backup” as an
121 “archive,” which might not be what you want.
122
123 This method doesn’t get you a copy of the remote’s
124 [private branches][pbr], on purpose. It may also miss other info on the
125 remote, such as SQL-level customizations that the sync protocol can’t
126 see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can
127 solve such problems if you have access to the remote server, which
128 allows you to get a SQL-level backup. This requires Fossil 2.12 or
129 newer, which added [the `backup` command][bu] to take care of
130 locking and transaction isolation, allowing the user to safely back up an in-use
131 repository.
132
133 If you have SSH access to the remote server, something like this will work:
134
135 ----
@@ -167,11 +169,13 @@
169
170 This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you
171 won’t have the `-pbkdf2` and `-iter` options, and you may have to choose
172 a different cipher algorithm; both changes are likely to weaken the
173 encryption significantly, so you should install a newer version rather
174 than work around the lack of these features.
175
176 If you’re on macOS, which
177 still ships 1.0 as of the time of this writing, [Homebrew][hb] offers
178 the current version of OpenSSL, but to avoid a conflict with the platform
179 version it’s [unlinked][hbul] by default, so you have to give an explicit
180 path to its “cellar” directory:
181
@@ -211,19 +215,21 @@
215
216 The last change is tricky: we used `fossil sql` above to ensure that
217 we’re using the same version of SQLite to write the encrypted backup DB
218 as was used to maintain the repository, but unfortunately, we can’t get
219 the built-in SQLite shell to write a backup into an empty database.
220 (As soon as it starts up, it goes looking for tables created by
221 `fossil init` and fails with an error.)
222 Therefore, we have to either run the restoration against a
223 possibly-different version of SQLite and hope there are no
224 incompatibilities, or we have to go out of our way to build a matching
225 version of `sqlite3` before we can safely do the restoration.
226
227 Keep in mind that Fossil often acts as a dogfooding project for SQLite,
228 making use of the latest features, so it is quite likely that a given
229 random `sqlite3` binary in your `PATH` will be unable to understand the
230 file created by “`fossil sql .dump`”!
231
232 [bu]: /help?cmd=backup
233 [grcp]: https://www.grc.com/passwords.htm
234 [hb]: https://brew.sh
235 [hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean
236

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button