|
1
|
<title>The Fossil Sync Protocol</title> |
|
2
|
|
|
3
|
This document describes the wire protocol used to synchronize |
|
4
|
content between two Fossil repositories. |
|
5
|
|
|
6
|
<h2>1.0 Overview</h2> |
|
7
|
|
|
8
|
The global state of a fossil repository consists of an unordered |
|
9
|
[./fileformat.wiki|collection of artifacts]. Each artifact is |
|
10
|
identified by a cryptographic |
|
11
|
hash of its content, expressed as a lower-case hexadecimal string. |
|
12
|
Synchronization is the process of sharing artifacts between |
|
13
|
repositories so that all repositories have copies of all artifacts. Because |
|
14
|
artifacts are unordered, the order in which artifacts are received |
|
15
|
is unimportant. It is assumed that the hash names |
|
16
|
of artifacts are unique - that every artifact has a different hash. |
|
17
|
To a first approximation, synchronization proceeds by sharing lists |
|
18
|
of hashes for available artifacts, then sharing the content of artifacts |
|
19
|
whose names are missing from one side or the other of the connection. |
|
20
|
In practice, a repository might contain millions of artifacts. The list of |
|
21
|
hash names for this many artifacts can be large. So optimizations are |
|
22
|
employed that usually reduce the number of hashes that need to be |
|
23
|
shared to a few dozen. |
|
24
|
|
|
25
|
Each repository also has local state. The local state determines |
|
26
|
the web-page formatting preferences, authorized users, ticket formats, |
|
27
|
and similar information that varies from one repository to another. |
|
28
|
The local state is not usually transferred during a sync. Except, |
|
29
|
some local state is transferred during a [/help/clone|clone] |
|
30
|
in order to initialize the local state of the new repository. Also, |
|
31
|
an administrator can sync local state using |
|
32
|
the [/help/configuration|config push] and |
|
33
|
[/help/configuration|config pull] |
|
34
|
commands. |
|
35
|
|
|
36
|
<h3 id="crdt">1.1 Conflict-Free Replicated Datatypes</h3> |
|
37
|
|
|
38
|
The "bag of artifacts" data model used by Fossil is apparently an |
|
39
|
implementation of a particular |
|
40
|
[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type|Conflict-Free |
|
41
|
Replicated Datatype (CRDT)] called a "G-Set" or "Grow-only Set". The |
|
42
|
academic literature on CRDTs only began to appear in about 2011, and |
|
43
|
Fossil predates that research by at least 4 years. But it is nice to |
|
44
|
know that theorists have now proven that the underlying data model of |
|
45
|
Fossil can provide strongly-consistent replicas using only |
|
46
|
peer-to-peer communication and without any kind of central |
|
47
|
authority. |
|
48
|
|
|
49
|
If you are already familiar with CRDTs and were wondering if Fossil |
|
50
|
used them, the answer is "yes". We just don't call them by that name. |
|
51
|
|
|
52
|
|
|
53
|
<h2 id="transport">2.0 Transport</h2> |
|
54
|
|
|
55
|
All communication between client and server is via HTTP requests. |
|
56
|
The server is listening for incoming HTTP requests. The client |
|
57
|
issues one or more HTTP requests and receives replies for each |
|
58
|
request. |
|
59
|
|
|
60
|
The server might be running as an independent server |
|
61
|
using the [/help/server|"fossil server" command], or it |
|
62
|
might be launched from inetd or xinetd using the |
|
63
|
[/help/http|"fossil http" command]. Or the server might |
|
64
|
be [./server/any/cgi.md|launched from CGI] or from |
|
65
|
[./server/any/scgi.md|SCGI]. |
|
66
|
(See "[./server/|How To Configure A Fossil Server]" for details.) |
|
67
|
The specifics of how the server listens |
|
68
|
for incoming HTTP requests is immaterial to this protocol. |
|
69
|
The important point is that the server is listening for requests and |
|
70
|
the client is the issuer of the requests. |
|
71
|
|
|
72
|
A single [/help/push|push], |
|
73
|
[/help/pull|pull], or [/help?cmd=sync|sync] |
|
74
|
might involve multiple HTTP requests. |
|
75
|
The client maintains state between all requests. But on the server |
|
76
|
side, each request is independent. The server does not preserve |
|
77
|
any information about the client from one request to the next. |
|
78
|
|
|
79
|
Note: Throughout this article, we use the terms "server" and "client" |
|
80
|
to represent the listener and initiator of the interaction, respectively. |
|
81
|
Nothing in this protocol requires that the server actually be a back-room |
|
82
|
processor housed in a datacenter, nor does the client need to be a desktop |
|
83
|
or handheld device. For the purposes of this article "client" simply means |
|
84
|
the repository that initiates the conversation and "server" is the repository |
|
85
|
that responds. Nothing more. |
|
86
|
|
|
87
|
<h4 id="https">2.0.1 HTTPS Transport</h4> |
|
88
|
|
|
89
|
HTTPS differs from HTTP only in that the HTTPS protocol is |
|
90
|
encrypted as it travels over the wire. The underlying protocol |
|
91
|
is the same. This document describes only the underlying, unencrypted |
|
92
|
messages that go client to server and back to client. |
|
93
|
Whether or not those messages are encrypted does not come into play |
|
94
|
in this document. |
|
95
|
|
|
96
|
Fossil includes built-in |
|
97
|
[./ssl-server.md|support for HTTPS encryption] in both client and server. |
|
98
|
|
|
99
|
<h4 id="ssh">2.0.2 SSH Transport</h4> |
|
100
|
|
|
101
|
When doing a sync using an "<code>ssh:…</code>" URL, the same HTTP transport protocol |
|
102
|
is used. Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh] |
|
103
|
to start an instance of the [/help?cmd=test-http|fossil test-http] command |
|
104
|
running on the remote machine. It then sends HTTP requests and gets back HTTP |
|
105
|
replies over the SSH connection, rather than sending and receiving over an |
|
106
|
internet socket. To see the specific "ssh" command that the Fossil client |
|
107
|
runs in order to set up a connection, add either of the the "--httptrace" or |
|
108
|
"--sshtrace" options to the "fossil sync" command line. |
|
109
|
|
|
110
|
This method is dependent on the remote <var>PATH</var> set by the SSH |
|
111
|
daemon, which may not be the same as your interactive shell's |
|
112
|
<var>PATH</var> on that same server. It is common to find |
|
113
|
<var>$HOME/bin</var> in the latter but not the former, for instance, |
|
114
|
leading to failures to sync over <code>ssh:…</code> URLs when you |
|
115
|
install the <code>fossil</code> binary in a nonstandard location, as |
|
116
|
with |
|
117
|
|
|
118
|
<verbatim>./configure --prefix=$HOME && make install</verbatim> |
|
119
|
|
|
120
|
The simpler of the two solutions to this problem is to install Fossil |
|
121
|
where sshd expects to find it, but when that isn't an option, you can |
|
122
|
instead give a URL like this: |
|
123
|
|
|
124
|
<verbatim>fossil clone ssh://myserver.example.com/path/to/repo.fossil?fossil=/home/me/bin/fossil</verbatim> |
|
125
|
|
|
126
|
That gives the local Fossil instance the absolute path to the binary on |
|
127
|
the remote machine for use when calling that Fossil instance through the |
|
128
|
SSH tunnel. |
|
129
|
|
|
130
|
<h4 id="file">2.0.3 FILE Transport</h4> |
|
131
|
|
|
132
|
When doing a sync using a "file:..." URL, the same HTTP protocol is |
|
133
|
still used. But instead of sending each HTTP request over a socket or |
|
134
|
via SSH, the HTTP request is written into a temporary file. The client |
|
135
|
then invokes the [/help?cmd=http|fossil http] command in a subprocess |
|
136
|
to process the request and and generate a reply. The client then reads |
|
137
|
the HTTP reply out of a temporary file on disk, and deletes the two |
|
138
|
temporary files. To see the specific "fossil http" command that is run |
|
139
|
in order to implement the "file:" transport, add the "--httptrace" |
|
140
|
option to the "fossil sync" command. |
|
141
|
|
|
142
|
<h3 id="srv-id">2.1 Server Identification</h3> |
|
143
|
|
|
144
|
The server is identified by a URL argument that accompanies the |
|
145
|
push, pull, or sync command on the client. (As a convenience to |
|
146
|
users, the URL can be omitted on the client command and the same URL |
|
147
|
from the most recent push, pull, or sync will be reused. This saves |
|
148
|
typing in the common case where the client does multiple syncs to |
|
149
|
the same server.) |
|
150
|
|
|
151
|
The client modifies the URL by appending the method name "<b>/xfer</b>" |
|
152
|
to the end. For example, if the URL specified on the client command |
|
153
|
line is |
|
154
|
|
|
155
|
<pre>https://fossil-scm.org/home</pre> |
|
156
|
|
|
157
|
Then the URL that is really used to do the synchronization will |
|
158
|
be: |
|
159
|
|
|
160
|
<pre>https://fossil-scm.org/home/xfer</pre> |
|
161
|
|
|
162
|
<h3 id="req-format">2.2 HTTP Request Format</h3> |
|
163
|
|
|
164
|
The client always sends a POST request to the server. The |
|
165
|
general format of the POST request is as follows: |
|
166
|
|
|
167
|
<pre> |
|
168
|
POST /home/xfer HTTP/1.0 |
|
169
|
Host: fossil-scm.hwaci.com:80 |
|
170
|
Content-Type: application/x-fossil |
|
171
|
Content-Length: 4216 |
|
172
|
</pre> |
|
173
|
|
|
174
|
<i><pre>content...</pre></i> |
|
175
|
|
|
176
|
In the example above, the pathname given after the POST keyword |
|
177
|
on the first line is a copy of the URL pathname. The Host: parameter |
|
178
|
is also taken from the URL. The content type is always either |
|
179
|
"application/x-fossil" or "application/x-fossil-debug". The "x-fossil" |
|
180
|
content type is the default. The only difference is that "x-fossil" |
|
181
|
content is compressed using zlib whereas "x-fossil-debug" is sent |
|
182
|
uncompressed. |
|
183
|
|
|
184
|
A typical reply from the server might look something like this: |
|
185
|
|
|
186
|
<pre> |
|
187
|
HTTP/1.0 200 OK |
|
188
|
Date: Mon, 10 Sep 2007 12:21:01 GMT |
|
189
|
Connection: close |
|
190
|
Cache-control: private |
|
191
|
Content-Type: application/x-fossil; charset=US-ASCII |
|
192
|
Content-Length: 265 |
|
193
|
</pre> |
|
194
|
|
|
195
|
<i><pre>content...</pre></i> |
|
196
|
|
|
197
|
The content type of the reply is always the same as the content type |
|
198
|
of the request. |
|
199
|
|
|
200
|
<h2 id="content">3.0 Fossil Synchronization Content</h2> |
|
201
|
|
|
202
|
A synchronization request between a client and server consists of |
|
203
|
one or more HTTP requests as described in the previous section. This |
|
204
|
section details the "x-fossil" content type. |
|
205
|
|
|
206
|
<h3 id="lines">3.1 Line-oriented Format</h3> |
|
207
|
|
|
208
|
The x-fossil content type consists of zero or more "cards". Cards |
|
209
|
are separated by the newline character ("\n"). Leading and trailing |
|
210
|
whitespace on a card is ignored. Blank cards are ignored. |
|
211
|
|
|
212
|
Each card is divided into zero or more space separated tokens. |
|
213
|
The first token on each card is the operator. Subsequent tokens |
|
214
|
are arguments. The set of operators understood by servers is slightly |
|
215
|
different from the operators understood by clients, though the two |
|
216
|
are very similar. |
|
217
|
|
|
218
|
<h3 id="login">3.2 Login Cards</h3> |
|
219
|
|
|
220
|
Every message from client to server begins with one or more login |
|
221
|
cards. Each login card has the following format: |
|
222
|
|
|
223
|
<pre><b>login</b> <i>userid nonce signature</i></pre> |
|
224
|
|
|
225
|
The userid is the name of the user that is requesting service from the |
|
226
|
server, encoded in "fossilized" form (exactly as described for <a |
|
227
|
href="#error">the error card</a>). The nonce is the SHA1 hash of the |
|
228
|
remainder of the message - all text that follows the newline character |
|
229
|
that terminates the login card. The signature is the SHA1 hash of the |
|
230
|
concatenation of the nonce and the users password. |
|
231
|
|
|
232
|
When receving a login card, the server looks up the user and verifies |
|
233
|
that the nonce matches the SHA1 hash of the remainder of the message. |
|
234
|
It then checks the signature hash to make sure the signature matches. |
|
235
|
If everything checks out, then the client is granted all privileges of |
|
236
|
the specified user. |
|
237
|
|
|
238
|
Only one login card is permitted. A second login card will trigger |
|
239
|
a sync error. (Prior to 2025-07-21, the protocol permitted multiple |
|
240
|
logins, treating the login as the union of all privileges from all |
|
241
|
login cards. That capability was never used and has been removed.) |
|
242
|
|
|
243
|
As of version 2.27, Fossil supports transfering of the login card |
|
244
|
externally to the request payload via a Cookie HTTP header: |
|
245
|
|
|
246
|
<verbatim> |
|
247
|
Cookie: x-f-x-l=... |
|
248
|
</verbatim> |
|
249
|
|
|
250
|
Where "..." is the URL-encoded login cookie. <code>x-f-x-l</code> is |
|
251
|
short for X-Fossil-Xfer-Login. |
|
252
|
|
|
253
|
|
|
254
|
<h3 id="file">3.3 File Cards</h3> |
|
255
|
|
|
256
|
Artifacts are transferred using either "file" cards, or "cfile" |
|
257
|
or "uvfile" cards. |
|
258
|
The name "file" card comes from the fact that most artifacts correspond to |
|
259
|
files that are under version control. |
|
260
|
The "cfile" name is an abbreviation for "compressed file". |
|
261
|
The "uvfile" name is an abbreviation for "unversioned file". |
|
262
|
|
|
263
|
<h4 id="ordinary-fc">3.3.1 Ordinary File Cards</h4> |
|
264
|
|
|
265
|
For sync protocols, artifacts are transferred using "file" |
|
266
|
cards. File cards come in two different formats depending |
|
267
|
on whether the artifact is sent directly or as a |
|
268
|
[./delta_format.wiki|delta] from some |
|
269
|
other artifact. |
|
270
|
|
|
271
|
<pre> |
|
272
|
<b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> |
|
273
|
<b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> |
|
274
|
</pre> |
|
275
|
|
|
276
|
File cards are followed by in-line "payload" data. |
|
277
|
The content of the artifact |
|
278
|
or the artifact delta is the first <i>size</i> bytes of the |
|
279
|
x-fossil content that immediately follow the newline that |
|
280
|
terminates the file card. |
|
281
|
|
|
282
|
|
|
283
|
The first argument of a file card is the ID of the artifact that |
|
284
|
is being transferred. The artifact ID is the lower-case hexadecimal |
|
285
|
representation of the name hash for the artifact. |
|
286
|
The last argument of the file card is the number of bytes of |
|
287
|
payload that immediately follow the file card. If the file |
|
288
|
card has only two arguments, that means the payload is the |
|
289
|
complete content of the artifact. If the file card has three |
|
290
|
arguments, then the payload is a |
|
291
|
[./delta_format.wiki|delta] and the second argument is |
|
292
|
the ID of another artifact that is the source of the delta. |
|
293
|
|
|
294
|
File cards are sent in both directions: client to server and |
|
295
|
server to client. A delta might be sent before the source of |
|
296
|
the delta, so both client and server should remember deltas |
|
297
|
and be able to apply them when their source arrives. |
|
298
|
|
|
299
|
<h4 id="compressed-fc">3.3.2 Compressed File Cards</h4> |
|
300
|
|
|
301
|
A client that sends a clone protocol version "3" or greater will |
|
302
|
receive artifacts as "cfile" cards while cloning. This card was |
|
303
|
introduced to improve the speed of the transfer of content by sending |
|
304
|
the compressed artifact directly from the server database to the |
|
305
|
client. In this case, the containing response body is <em>not</em> |
|
306
|
compressed separately because the vast majority of the response is |
|
307
|
already compressed in cfile cards. In practice, version "3" is |
|
308
|
significantly faster than version "2". |
|
309
|
|
|
310
|
Compressed File cards are similar to File cards, sharing the same |
|
311
|
in-line "payload" data characteristics and also the same treatment of |
|
312
|
direct content or delta content. Cfile cards come in two different formats |
|
313
|
depending on whether the artifact is sent directly or as a delta from |
|
314
|
some other artifact. |
|
315
|
|
|
316
|
<pre> |
|
317
|
<b>cfile</b> <i>artifact-id usize csize</i> <b>\n</b> <i>content</i> |
|
318
|
<b>cfile</b> <i>artifact-id delta-artifact-id usize csize</i> <b>\n</b> <i>content</i> |
|
319
|
</pre> |
|
320
|
|
|
321
|
The first argument of the cfile card is the ID of the artifact that |
|
322
|
is being transferred. The artifact ID is the lower-case hexadecimal |
|
323
|
representation of the name hash for the artifact. The second argument of |
|
324
|
the cfile card is the original size in bytes of the artifact. The last |
|
325
|
argument of the cfile card is the number of compressed bytes of payload |
|
326
|
that immediately follow the cfile card. If the cfile card has only |
|
327
|
three arguments, that means the payload is the complete content of the |
|
328
|
artifact. If the cfile card has four arguments, then the payload is a |
|
329
|
delta and the second argument is the ID of another artifact that is the |
|
330
|
source of the delta and the third argument is the original size of the |
|
331
|
delta artifact. |
|
332
|
|
|
333
|
Unlike file cards, cfile cards are only sent in one direction during a |
|
334
|
clone from server to client for clone protocol version "3" or greater. |
|
335
|
|
|
336
|
<h4 id="private">3.3.3 Private artifacts</h4> |
|
337
|
|
|
338
|
"Private" content consist of artifacts that are not normally synced. |
|
339
|
However, private content will be synced when the |
|
340
|
the [/help?cmd=sync|fossil sync] command includes the "--private" option. |
|
341
|
|
|
342
|
Private content is marked by a "private" card: |
|
343
|
|
|
344
|
<pre><b>private</b></pre> |
|
345
|
|
|
346
|
The private card has no arguments and must directly precede a |
|
347
|
file card that contains the private content. |
|
348
|
|
|
349
|
<h4 id="uv-fc">3.3.4 Unversioned File Cards</h4> |
|
350
|
|
|
351
|
Unversioned content is sent in both directions (client to server and |
|
352
|
server to client) using "uvfile" cards in the following format: |
|
353
|
|
|
354
|
<pre><b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i></pre> |
|
355
|
|
|
356
|
The <i>name</i> field is the name of the unversioned file. The |
|
357
|
<i>mtime</i> is the last modification time of the file in seconds |
|
358
|
since 1970. The <i>hash</i> field is the hash of the content |
|
359
|
for the unversioned file, or "<b>-</b>" for deleted content. |
|
360
|
The <i>size</i> field is the (uncompressed) size of the content |
|
361
|
in bytes. The <i>flags</i> field is an integer which is interpreted |
|
362
|
as an array of bits. The 0x0004 bit of <i>flags</i> indicates that |
|
363
|
the <i>content</i> is to be omitted. The content might be omitted if |
|
364
|
it is too large to transmit, or if the sender merely wants to update the |
|
365
|
modification time of the file without changing the files content. |
|
366
|
The <i>content</i> is the (uncompressed) content of the file. |
|
367
|
|
|
368
|
The receiver should only accept the uvfile card if the hash and |
|
369
|
size match the content and if the mtime is newer than any existing |
|
370
|
instance of the same file held by the receiver. The sender will not |
|
371
|
normally transmit a uvfile card unless all these constraints are true, |
|
372
|
but the receiver should double-check. |
|
373
|
|
|
374
|
A server will only accept uvfile cards if the login user has |
|
375
|
the "y" write-unversioned permission. |
|
376
|
|
|
377
|
Servers send uvfile cards in response to uvgimme cards received from |
|
378
|
the client. Clients send uvfile cards when they determine that the server |
|
379
|
needs the content based on uvigot cards previously received from the server. |
|
380
|
|
|
381
|
<h3 id="push" name="pull">3.4 Push and Pull Cards</h3> |
|
382
|
|
|
383
|
Among the first cards in a client-to-server message are |
|
384
|
the push and pull cards. The push card tells the server that |
|
385
|
the client is pushing content. The pull card tells the server |
|
386
|
that the client wants to pull content. In the event of a sync, |
|
387
|
both cards are sent. The format is as follows: |
|
388
|
|
|
389
|
<pre> |
|
390
|
<b>push</b> <i>servercode projectcode</i> |
|
391
|
<b>pull</b> <i>servercode projectcode</i> |
|
392
|
</pre> |
|
393
|
|
|
394
|
The <i>servercode</i> argument is the repository ID for the |
|
395
|
client. The <i>projectcode</i> is the identifier |
|
396
|
of the software project that the client repository contains. |
|
397
|
The projectcode for the client and server must match in order |
|
398
|
for the transaction to proceed. |
|
399
|
|
|
400
|
The server will also send a push card back to the client |
|
401
|
during a clone. This is how the client determines what project |
|
402
|
code to put in the new repository it is constructing. |
|
403
|
|
|
404
|
The <i>servercode</i> argument is currently unused. |
|
405
|
|
|
406
|
<h3 id="clones">3.5 Clone Cards</h3> |
|
407
|
|
|
408
|
A clone card works like a pull card in that it is sent from |
|
409
|
client to server in order to tell the server that the client |
|
410
|
wants to pull content. The clone card comes in two formats. Older |
|
411
|
clients use the no-argument format and newer clients use the |
|
412
|
two-argument format. |
|
413
|
|
|
414
|
<pre> |
|
415
|
<b>clone</b> |
|
416
|
<b>clone</b> <i>protocol-version sequence-number</i> |
|
417
|
</pre> |
|
418
|
|
|
419
|
<h4>3.5.1 Protocol 3</h4> |
|
420
|
|
|
421
|
The latest clients send a two-argument clone message with a |
|
422
|
protocol version of "3". (Future versions of Fossil might use larger |
|
423
|
protocol version numbers.) Version "3" of the protocol enhanced version |
|
424
|
"2" by introducing the "cfile" card which is intended to speed up clone |
|
425
|
operations. Instead of sending "file" cards, the server will send "cfile" |
|
426
|
cards |
|
427
|
|
|
428
|
<h4>3.5.2 Protocol 2</h4> |
|
429
|
|
|
430
|
The sequence-number sent is the number |
|
431
|
of artifacts received so far. For the first clone message, the |
|
432
|
sequence number is 1. The server will respond by sending file |
|
433
|
cards for some number of artifacts up to the maximum message size. |
|
434
|
|
|
435
|
The server will also send a single "clone_seqno" card to the client |
|
436
|
so that the client can know where the server left off. |
|
437
|
|
|
438
|
<pre> |
|
439
|
<b>clone_seqno</b> <i>sequence-number</i> |
|
440
|
</pre> |
|
441
|
|
|
442
|
The clone message in subsequent HTTP requests for the same clone |
|
443
|
operation will use the sequence-number from the |
|
444
|
clone_seqno of the previous reply. |
|
445
|
|
|
446
|
In response to an initial clone message, the server also sends the client |
|
447
|
a push message so that the client can discover the projectcode for |
|
448
|
this project. |
|
449
|
|
|
450
|
<h4>3.5.3 Legacy Protocol</h4> |
|
451
|
|
|
452
|
Older clients send a clone card with no argument. The server responds |
|
453
|
to a blank clone card by sending an "igot" card for every artifact in the |
|
454
|
repository. The client will then issue "gimme" cards to pull down all the |
|
455
|
content it needs. |
|
456
|
|
|
457
|
The legacy protocol works well for smaller repositories (50MB with 50,000 |
|
458
|
artifacts) but is too slow and unwieldy for larger repositories. |
|
459
|
The version 2 protocol is an effort to improve performance. Further |
|
460
|
performance improvements with higher-numbered clone protocols are |
|
461
|
possible in future versions of Fossil. |
|
462
|
|
|
463
|
<h3 id="igot">3.6 Igot Cards</h3> |
|
464
|
|
|
465
|
An igot card can be sent from either client to server or from |
|
466
|
server to client in order to indicate that the sender holds a copy |
|
467
|
of a particular artifact. The format is: |
|
468
|
|
|
469
|
<pre> |
|
470
|
<b>igot</b> <i>artifact-id</i> ?<i>flag</i>? |
|
471
|
</pre> |
|
472
|
|
|
473
|
The first argument of the igot card is the ID of the artifact that |
|
474
|
the sender possesses. |
|
475
|
The receiver of an igot card will typically check to see if |
|
476
|
it also holds the same artifact and if not it will request the artifact |
|
477
|
using a gimme card in either the reply or in the next message. |
|
478
|
|
|
479
|
If the second argument exists and is "1", then the artifact |
|
480
|
identified by the first argument is private on the sender and should |
|
481
|
be ignored unless a "--private" [/help?cmd=sync|sync] is occurring. |
|
482
|
|
|
483
|
The name "igot" comes from the English slang expression "I got" meaning |
|
484
|
"I have". |
|
485
|
|
|
486
|
<h4>3.6.1 Unversioned Igot Cards</h4> |
|
487
|
|
|
488
|
Zero or more "uvigot" cards are sent from server to client when |
|
489
|
synchronizing unversioned content. The format of a uvigot card is |
|
490
|
as follows: |
|
491
|
|
|
492
|
<pre> |
|
493
|
<b>uvigot</b> <i>name mtime hash size</i> |
|
494
|
</pre> |
|
495
|
|
|
496
|
The <i>name</i> argument is the name of an unversioned file. |
|
497
|
The <i>mtime</i> is the last modification time of the unversioned file |
|
498
|
in seconds since 1970. |
|
499
|
The <i>hash</i> is the SHA1 or SHA3-256 hash of the unversioned file |
|
500
|
content, or "<b>-</b>" if the file has been deleted. |
|
501
|
The <i>size</i> is the uncompressed size of the file in bytes. |
|
502
|
|
|
503
|
When the server sees a "pragma uv-hash" card for which the hash |
|
504
|
does not match, it sends uvigot cards for every unversioned file that it |
|
505
|
holds. The client will use this information to figure out which |
|
506
|
unversioned files need to be synchronized. |
|
507
|
The server might also send a uvigot card when it receives a uvgimme card |
|
508
|
but its reply message size is already oversized and hence unable to hold |
|
509
|
the usual uvfile reply. |
|
510
|
|
|
511
|
When a client receives a "uvigot" card, it checks to see if the |
|
512
|
file needs to be transferred from client to server or from server to client. |
|
513
|
If a client-to-server transmission is needed, the client schedules that |
|
514
|
transfer to occur on a subsequent HTTP request. If a server-to-client |
|
515
|
transfer is needed, then the client sends a "uvgimme" card back to the |
|
516
|
server to request the file content. |
|
517
|
|
|
518
|
<h3 id="gimme">3.7 Gimme Cards</h3> |
|
519
|
|
|
520
|
A gimme card is sent from either client to server or from server |
|
521
|
to client. The gimme card asks the receiver to send a particular |
|
522
|
artifact back to the sender. The format of a gimme card is this: |
|
523
|
|
|
524
|
<pre> |
|
525
|
<b>gimme</b> <i>artifact-id</i> |
|
526
|
</pre> |
|
527
|
|
|
528
|
The argument to the gimme card is the ID of the artifact that |
|
529
|
the sender wants. The receiver will typically respond to a |
|
530
|
gimme card by sending a file card in its reply or in the next |
|
531
|
message. |
|
532
|
|
|
533
|
The "gimme" name means "give me". The imperative "give me" is |
|
534
|
pronounced as if it were a single word "gimme" in some dialects of |
|
535
|
English (including the dialect spoken by the original author of Fossil). |
|
536
|
|
|
537
|
<h4>3.7.1 Unversioned Gimme Cards</h4> |
|
538
|
|
|
539
|
When synchronizing unversioned content, the client may send "uvgimme" |
|
540
|
cards to the server. A uvgimme card requests that the server send |
|
541
|
unversioned content to the client. The format of a uvgimme card is |
|
542
|
as follows: |
|
543
|
|
|
544
|
<pre> |
|
545
|
<b>uvgimme</b> <i>name</i> |
|
546
|
</pre> |
|
547
|
|
|
548
|
The <i>name</i> is the name of the unversioned file found on the |
|
549
|
server that the client would like to have. When a server sees a |
|
550
|
uvgimme card, it normally responses with a uvfile card, though it might |
|
551
|
also send another uvigot card if the HTTP reply is already oversized. |
|
552
|
|
|
553
|
<h3 id="cookie">3.8 Cookie Cards</h3> |
|
554
|
|
|
555
|
A cookie card can be used by a server to record a small amount |
|
556
|
of state information on a client. The server sends a cookie to the |
|
557
|
client. The client sends the same cookie back to the server on |
|
558
|
its next request. The cookie card has a single argument which |
|
559
|
is its payload. |
|
560
|
|
|
561
|
<pre> |
|
562
|
<b>cookie</b> <i>payload</i> |
|
563
|
</pre> |
|
564
|
|
|
565
|
The client is not required to return the cookie to the server on |
|
566
|
its next request. Or the client might send a cookie from a different |
|
567
|
server on the next request. So the server must not depend on the |
|
568
|
cookie and the server must structure the cookie payload in such |
|
569
|
a way that it can tell if the cookie it sees is its own cookie or |
|
570
|
a cookie from another server. (Typically the server will embed |
|
571
|
its servercode as part of the cookie.) |
|
572
|
|
|
573
|
<h3 id="reqconfig">3.9 Request-Configuration Cards</h3> |
|
574
|
|
|
575
|
A request-configuration or "reqconfig" card is sent from client to |
|
576
|
server in order to request that the server send back "configuration" |
|
577
|
data. "Configuration" data is information about users or website |
|
578
|
appearance or other administrative details which are not part of the |
|
579
|
persistent and versioned state of the project. For example, the "name" |
|
580
|
of the project, the default Cascading Style Sheet (CSS) for the web-interface, |
|
581
|
and the project logo displayed on the web-interface are all configuration |
|
582
|
data elements. |
|
583
|
|
|
584
|
The reqconfig card is normally sent in response to the |
|
585
|
"fossil configuration pull" command. The format is as follows: |
|
586
|
|
|
587
|
<pre> |
|
588
|
<b>reqconfig</b> <i>configuration-name</i> |
|
589
|
</pre> |
|
590
|
|
|
591
|
As of 2024-10-22, the configuration-name must be one of the |
|
592
|
following values: |
|
593
|
|
|
594
|
<table border=0 align="center"> |
|
595
|
<tr><td valign="top"> |
|
596
|
<ul> |
|
597
|
<li> css |
|
598
|
<li> header |
|
599
|
<li> mainmenu |
|
600
|
<li> footer |
|
601
|
<li> details |
|
602
|
<li> js |
|
603
|
<li> default-skin |
|
604
|
<li> logo-mimetype |
|
605
|
<li> logo-image |
|
606
|
<li> background-mimetype |
|
607
|
<li> background-image |
|
608
|
<li> icon-mimetype |
|
609
|
<li> icon-image |
|
610
|
<li> timeline-block-markup |
|
611
|
<li> timeline-date-format |
|
612
|
<li> timeline-default-style |
|
613
|
<ul></td><td valign="top"><ul> |
|
614
|
<li> timeline-dwelltime |
|
615
|
<li> timeline-closetime |
|
616
|
<li> timeline-hard-newlines |
|
617
|
<li> timeline-max-comment |
|
618
|
<li> timeline-plaintext |
|
619
|
<li> timeline-truncate-at-blank |
|
620
|
<li> timeline-tslink-info |
|
621
|
<li> timeline-utc |
|
622
|
<li> adunit |
|
623
|
<li> adunit-omit-if-admin |
|
624
|
<li> adunit-omit-if-user |
|
625
|
<li> default-csp |
|
626
|
<li> sitemap-extra |
|
627
|
<li> safe-html |
|
628
|
<li> th1-hooks |
|
629
|
<li> th1-uri-regexp |
|
630
|
<li> project-name |
|
631
|
<ul></td><td valign="top"><ul> |
|
632
|
<li> short-project-name |
|
633
|
<li> project-description |
|
634
|
<li> index-page |
|
635
|
<li> manifest |
|
636
|
<li> binary-glob |
|
637
|
<li> clean-glob |
|
638
|
<li> ignore-glob |
|
639
|
<li> keep-glob |
|
640
|
<li> crlf-glob |
|
641
|
<li> crnl-glob |
|
642
|
<li> encoding-glob |
|
643
|
<li> empty-dirs |
|
644
|
<li> <s title="removed 2020-08, version 2.12.1">allow-symlinks</s> |
|
645
|
<li> dotfiles |
|
646
|
<li> parent-project-code |
|
647
|
<li> parent-project-name |
|
648
|
<ul></td><td valign="top"><ul> |
|
649
|
<li> hash-policy |
|
650
|
<li> comment-format |
|
651
|
<li> mimetypes |
|
652
|
<li> forbid-delta-manifests |
|
653
|
<li> mv-rm-files |
|
654
|
<li> ticket-table |
|
655
|
<li> ticket-common |
|
656
|
<li> ticket-change |
|
657
|
<li> ticket-newpage |
|
658
|
<li> ticket-viewpage |
|
659
|
<li> ticket-editpage |
|
660
|
<li> ticket-reportlist |
|
661
|
<li> ticket-report-template |
|
662
|
<li> ticket-key-template |
|
663
|
<li> ticket-title-expr |
|
664
|
<li> ticket-closed-expr |
|
665
|
<ul></td><td valign="top"><ul> |
|
666
|
<li> user-color-map |
|
667
|
<li> xfer-common-script |
|
668
|
<li> xfer-push-script |
|
669
|
<li> xfer-commit-script |
|
670
|
<li> xfer-ticket-script |
|
671
|
<li> @reportfmt |
|
672
|
<li> @user |
|
673
|
<li> @concealed |
|
674
|
<li> @shun |
|
675
|
<li> @alias |
|
676
|
<li> @subscriber |
|
677
|
<li> @interwiki |
|
678
|
</ul></td></tr> |
|
679
|
</table> |
|
680
|
|
|
681
|
New configuration-names are likely to be added in future releases of |
|
682
|
Fossil. If the server receives a configuration-name that it does not |
|
683
|
understand, the entire reqconfig card is silently ignored. The reqconfig |
|
684
|
card might also be ignored if the user lacks sufficient privilege to |
|
685
|
access the requested information. |
|
686
|
|
|
687
|
The configuration-names that begin with an alphabetic character refer |
|
688
|
to values in the "config" table of the server database. For example, |
|
689
|
the "logo-image" configuration item refers to the project logo image |
|
690
|
that is configured on the Admin page of the [./webui.wiki | web-interface]. |
|
691
|
The value of the configuration item is returned to the client using a |
|
692
|
"config" card. |
|
693
|
|
|
694
|
If the configuration-name begins with "@", that refers to a class of |
|
695
|
values instead of a single value. The content of these configuration items |
|
696
|
is returned in a "config" card that contains pure SQL text that is |
|
697
|
intended to be evaluated by the client. |
|
698
|
|
|
699
|
The @user and @concealed configuration items contain sensitive information |
|
700
|
and are ignored for clients without sufficient privilege. |
|
701
|
|
|
702
|
<h3 id="config">3.10 Configuration Cards</h3> |
|
703
|
|
|
704
|
A "config" card is used to send configuration information from client |
|
705
|
to server (in response to a "fossil configuration push" command) or |
|
706
|
from server to client (in response to a "fossil configuration pull" or |
|
707
|
"fossil clone" command). The format is as follows: |
|
708
|
|
|
709
|
<pre> |
|
710
|
<b>config</b> <i>configuration-name size</i> <b>\n</b> <i>content</i> |
|
711
|
</pre> |
|
712
|
|
|
713
|
The server will only accept a config card if the user has |
|
714
|
"Admin" privilege. A client will only accept a config card if |
|
715
|
it had sent a corresponding reqconfig card in its request. |
|
716
|
|
|
717
|
The content of the configuration item is used to overwrite the |
|
718
|
corresponding configuration data in the receiver. |
|
719
|
|
|
720
|
<h3 id="pragma">3.11 Pragma Cards</h3> |
|
721
|
|
|
722
|
The client may try to influence the behavior of the server by |
|
723
|
issuing a pragma card: |
|
724
|
|
|
725
|
<pre> |
|
726
|
<b>pragma</i> <i>name value...</i> |
|
727
|
</pre> |
|
728
|
|
|
729
|
The "pragma" card has at least one argument which is the pragma name. |
|
730
|
The pragma name defines what the pragma does. |
|
731
|
A pragma might have zero or more "value" arguments |
|
732
|
depending on the pragma name. |
|
733
|
|
|
734
|
New pragma names may be added to the protocol from time to time |
|
735
|
in order to enhance the capabilities of Fossil. |
|
736
|
Unknown pragmas are silently ignored, for backwards compatibility. |
|
737
|
|
|
738
|
The following are the known pragma names as of 2019-06-30: |
|
739
|
|
|
740
|
<ol> |
|
741
|
<li><b>send-private</b> The send-private pragma instructs the server to send all of its |
|
742
|
private artifacts to the client. The server will only obey this |
|
743
|
request if the user has the "x" or "Private" privilege. |
|
744
|
|
|
745
|
<li><b>send-catalog</b> The send-catalog pragma instructs the server to transmit igot |
|
746
|
cards for every known artifact. This can help the client and server |
|
747
|
to get back in synchronization after a prior protocol error. The |
|
748
|
"--verily" option to the [/help?cmd=sync|fossil sync] command causes |
|
749
|
the send-catalog pragma to be transmitted. |
|
750
|
|
|
751
|
<li><b>uv-hash</b> <i>HASH</i> The uv-hash pragma is sent from client to server to provoke a |
|
752
|
synchronization of unversioned content. The <i>HASH</i> is a SHA1 |
|
753
|
hash of the names, modification times, and individual hashes of all |
|
754
|
unversioned files on the client. If the unversioned content hash |
|
755
|
from the client does not match the unversioned content hash on the |
|
756
|
server, then the server will reply with either a "pragma uv-push-ok" |
|
757
|
or "pragma uv-pull-only" card followed by one "uvigot" card for |
|
758
|
each unversioned file currently held on the server. The collection |
|
759
|
of "uvigot" cards sent in response to a "uv-hash" pragma is called |
|
760
|
the "unversioned catalog". The client will used the unversioned |
|
761
|
catalog to figure out which files (if any) need to be synchronized |
|
762
|
between client and server and send appropriate "uvfile" or "uvgimme" |
|
763
|
cards on the next HTTP request. |
|
764
|
|
|
765
|
If a client sends a uv-hash pragma and does not receive back |
|
766
|
either a uv-pull-only or uv-push-ok pragma, that means that the |
|
767
|
content on the server exactly matches the content on the client and |
|
768
|
no further synchronization is required. |
|
769
|
|
|
770
|
<li><b>uv-pull-only</b></i> A server sends the uv-pull-only pragma to the client in response |
|
771
|
to a uv-hash pragma with a mismatched content hash argument. This |
|
772
|
pragma indicates that there are differences in unversioned content |
|
773
|
between the client and server but that content can only be transferred |
|
774
|
from server to client. The server is unwilling to accept content from |
|
775
|
the client because the client login lacks the "write-unversioned" |
|
776
|
permission. |
|
777
|
|
|
778
|
<li><b>uv-push-ok</b></i> A server sends the uv-push-ok pragma to the client in response |
|
779
|
to a uv-hash pragma with a mismatched content hash argument. This |
|
780
|
pragma indicates that there are differences in unversioned content |
|
781
|
between the client and server and that content can be transferred |
|
782
|
in either direction. The server is willing to accept content from |
|
783
|
the client because the client login has the "write-unversioned" |
|
784
|
permission. |
|
785
|
|
|
786
|
<li><b>ci-lock</b> <i>CHECKIN-HASH CLIENT-ID</i> A client sends the "ci-lock" pragma to the server to indicate |
|
787
|
that it is about to add a new check-in as a child of the |
|
788
|
CHECKIN-HASH check-in and on the same branch as CHECKIN-HASH. |
|
789
|
If some other client has already indicated that it was also |
|
790
|
trying to commit against CHECKIN-HASH, that indicates that a |
|
791
|
fork is about to occur, and the server will reply with |
|
792
|
a "ci-lock-fail" pragma (see below). Check-in locks |
|
793
|
automatically expire when the check-in actually occurs, or |
|
794
|
after a timeout (currently one minute but subject to change). |
|
795
|
|
|
796
|
<li><b>ci-lock-fail</b> <i>LOGIN MTIME</i> When a server receives two or more "ci-lock" pragma messages |
|
797
|
for the same check-in but from different clients, the second a |
|
798
|
subsequent ci-lock will provoke a ci-lock-fail pragma in the |
|
799
|
reply to let the client know that it if continues with the |
|
800
|
check-in it will likely generate a fork. The LOGIN and MTIME |
|
801
|
arguments are intended to provide information to the client to |
|
802
|
help it generate a more useful error message. |
|
803
|
|
|
804
|
<li><b>ci-unlock</b> <i>CLIENT-ID</i> A client sends the "ci-unlock" pragma to the server after |
|
805
|
a successful commit. This instructs the server to release |
|
806
|
any lock on any check-in previously held by that client. |
|
807
|
The ci-unlock pragma helps to avoid false-positive lock warnings |
|
808
|
that might arise if a check-in is aborted and then restarted |
|
809
|
on a branch. |
|
810
|
|
|
811
|
<li><b>req-clusters</b> A client sends the "req-clusters" pragma |
|
812
|
to the server to ask the server to reply with "igot" cards for |
|
813
|
every [./fileformat.wiki#cluster|cluster artifact] that it holds. The |
|
814
|
client typically does this when it thinks that it might be attempting |
|
815
|
to pull a long chain of cluster artifacts. Sending the artifacts |
|
816
|
all at once can dramatically reduce the number of round trip |
|
817
|
messages needed to complete the synchronization. |
|
818
|
</ol> |
|
819
|
|
|
820
|
<h3 id="comment">3.12 Comment Cards</h3> |
|
821
|
|
|
822
|
Any card that begins with "#" (ASCII 0x23) is a comment card and |
|
823
|
is silently ignored. |
|
824
|
|
|
825
|
<h3 id="error">3.13 Message and Error Cards</h3> |
|
826
|
|
|
827
|
If the server discovers anything wrong with a request, it generates |
|
828
|
an error card in its reply. When the client sees the error card, |
|
829
|
it displays an error message to the user and aborts the sync |
|
830
|
operation. An error card looks like this: |
|
831
|
|
|
832
|
<pre> |
|
833
|
<b>error</b> <i>error-message</i> |
|
834
|
</pre> |
|
835
|
|
|
836
|
The error message is English text that is encoded in order to |
|
837
|
be a single token. |
|
838
|
A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73). A |
|
839
|
newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E). A backslash |
|
840
|
(ASCII 0x5C) is represented as two backslashes "\\". Apart from |
|
841
|
space and newline, no other whitespace characters nor any |
|
842
|
unprintable characters are allowed in |
|
843
|
the error message. |
|
844
|
|
|
845
|
The server can also send a message card that also prints a |
|
846
|
message on the client console, but which is not an error: |
|
847
|
|
|
848
|
<pre> |
|
849
|
<b>message</b> <i>message-text</i> |
|
850
|
</pre> |
|
851
|
|
|
852
|
The message-text uses the same format as an error message. |
|
853
|
|
|
854
|
<h3 id="unknown">3.14 Unknown Cards</h3> |
|
855
|
|
|
856
|
If either the client or the server sees a card that is not |
|
857
|
described above, then it generates an error and aborts. |
|
858
|
|
|
859
|
<h2 id="phantoms" name="clusters">4.0 Phantoms And Clusters</h2> |
|
860
|
|
|
861
|
When a repository knows that an artifact exists and knows the ID of |
|
862
|
that artifact, but it does not know the artifact content, then it stores that |
|
863
|
artifact as a "phantom". A repository will typically create a phantom when |
|
864
|
it receives an igot card for an artifact that it does not hold or when it |
|
865
|
receives a file card that references a delta source that it does not |
|
866
|
hold. When a server is generating its reply or when a client is |
|
867
|
generating a new request, it will usually send gimme cards for every |
|
868
|
phantom that it holds. |
|
869
|
|
|
870
|
A cluster is a special artifact that tells of the existence of other |
|
871
|
artifacts. Any artifact in the repository that follows the syntactic rules |
|
872
|
of a cluster is considered a cluster. |
|
873
|
|
|
874
|
A cluster is line oriented. Each line of a cluster |
|
875
|
is a card. The cards are separated by the newline ("\n") character. |
|
876
|
Each card consists of a single character card type, a space, and a |
|
877
|
single argument. No extra whitespace and no trailing or leading |
|
878
|
whitespace is allowed. All cards in the cluster must occur in |
|
879
|
strict lexicographical order. |
|
880
|
|
|
881
|
A cluster consists of one or more "M" cards followed by a single |
|
882
|
"Z" card. Each M card holds an argument which is an artifact ID for an |
|
883
|
artifact in the repository. The Z card has a single argument which is the |
|
884
|
lower-case hexadecimal representation of the MD5 checksum of all |
|
885
|
preceding M cards up to and included the newline character that |
|
886
|
occurred just before the Z that starts the Z card. |
|
887
|
|
|
888
|
Any artifact that does not match the specifications of a cluster |
|
889
|
exactly is not a cluster. There must be no extra whitespace in |
|
890
|
the artifact. There must be one or more M cards. There must be a |
|
891
|
single Z card with a correct MD5 checksum. And all cards must |
|
892
|
be in strict lexicographical order. |
|
893
|
|
|
894
|
<h3 id="unclustered">4.1 The Unclustered Table</h3> |
|
895
|
|
|
896
|
Every repository maintains a table named "<b>unclustered</b>" |
|
897
|
which records the identity of every artifact and phantom it holds that is not |
|
898
|
mentioned in a cluster. The entries in the unclustered table can |
|
899
|
be thought of as leaves on a tree of artifacts. Some of the unclustered |
|
900
|
artifacts will be other clusters. Those clusters may contain other clusters, |
|
901
|
which might contain still more clusters, and so forth. Beginning |
|
902
|
with the artifacts in the unclustered table, one can follow the chain |
|
903
|
of clusters to find every artifact in the repository. |
|
904
|
|
|
905
|
<h2 id="strategies">5.0 Synchronization Strategies</h2> |
|
906
|
|
|
907
|
<h3 id="pull-strategy">5.1 Pull</h3> |
|
908
|
|
|
909
|
A typical pull operation proceeds as shown below. Details |
|
910
|
of the actual implementation may vary slightly but the gist of |
|
911
|
a pull is captured in the following steps: |
|
912
|
|
|
913
|
<ol> |
|
914
|
<li>The client sends login and pull cards. |
|
915
|
<li>The client sends a cookie card if it has previously received a cookie. |
|
916
|
<li>The client sends gimme cards for every phantom that it holds. |
|
917
|
<hr> |
|
918
|
<li>The server checks the login password and rejects the session if |
|
919
|
the user does not have permission to pull. |
|
920
|
<li>If the number of entries in the unclustered table on the server is |
|
921
|
greater than 100, then the server constructs a new cluster artifact to |
|
922
|
cover all those unclustered entries. |
|
923
|
<li>The server sends file cards for every gimme card it received |
|
924
|
from the client. |
|
925
|
<li>The server sends igot cards for every artifact in its unclustered |
|
926
|
table that is not a phantom. |
|
927
|
<hr> |
|
928
|
<li>The client adds the content of file cards to its repository. |
|
929
|
<li>The client creates a phantom for every igot card in the server reply |
|
930
|
that mentions an artifact that the client does not possess. |
|
931
|
<li>The client creates a phantom for the delta source of file cards when |
|
932
|
the delta source is an artifact that the client does not possess. |
|
933
|
</ol> |
|
934
|
|
|
935
|
These ten steps represent a single HTTP round-trip request. |
|
936
|
The first three steps are the processing that occurs on the client |
|
937
|
to generate the request. The middle four steps are processing |
|
938
|
that occurs on the server to interpret the request and generate a |
|
939
|
reply. And the last three steps are the processing that the |
|
940
|
client does to interpret the reply. |
|
941
|
|
|
942
|
During a pull, the client will keep sending HTTP requests |
|
943
|
until it holds all artifacts that exist on the server. |
|
944
|
|
|
945
|
Note that the server tries |
|
946
|
to limit the size of its reply message to something reasonable |
|
947
|
(usually about 1MB) so that it might stop sending file cards as |
|
948
|
described in step (6) if the reply becomes too large. |
|
949
|
|
|
950
|
Step (5) is the only way in which new clusters can be created. |
|
951
|
By only creating clusters on the server, we hope to minimize the |
|
952
|
amount of overlap between clusters in the common configuration where |
|
953
|
there is a single server and many clients. The same synchronization |
|
954
|
protocol will continue to work even if there are multiple servers |
|
955
|
or if servers and clients sometimes change roles. The only negative |
|
956
|
effects of these unusual arrangements is that more than the minimum |
|
957
|
number of clusters might be generated. |
|
958
|
|
|
959
|
<h3 id="push-stragegy">5.2 Push</h3> |
|
960
|
|
|
961
|
A typical push operation proceeds roughly as shown below. As |
|
962
|
with a pull, the actual implementation may vary slightly. |
|
963
|
|
|
964
|
<ol> |
|
965
|
<li>The client sends login and push cards. |
|
966
|
<li>The client sends file cards for any artifacts that it holds that have |
|
967
|
never before been pushed - artifacts that come from local check-ins. |
|
968
|
<li>If this is the second or later cycle in a push, then the |
|
969
|
client sends file cards for any gimme cards that the server sent |
|
970
|
in the previous cycle. |
|
971
|
<li>The client sends igot cards for every artifact in its unclustered table |
|
972
|
that is not a phantom. |
|
973
|
<hr> |
|
974
|
<li>The server checks the login and push cards and issues an error if |
|
975
|
anything is amiss. |
|
976
|
<li>The server accepts file cards from the client and adds those artifacts |
|
977
|
to its repository. |
|
978
|
<li>The server creates phantoms for igot cards that mention artifacts it |
|
979
|
does not possess or for file cards that mention delta source artifacts that |
|
980
|
it does not possess. |
|
981
|
<li>The server issues gimme cards for all phantoms. |
|
982
|
<hr> |
|
983
|
<li>The client remembers the gimme cards from the server so that it |
|
984
|
can generate file cards in reply on the next cycle. |
|
985
|
</ol> |
|
986
|
|
|
987
|
As with a pull, the steps of a push operation repeat until the |
|
988
|
server knows all artifacts that exist on the client. Also, as with |
|
989
|
pull, the client attempts to keep the size of the request from |
|
990
|
growing too large by suppressing file cards once the |
|
991
|
size of the request reaches 1MB. |
|
992
|
|
|
993
|
<h3 id="sync-strategy">5.3 Sync</h3> |
|
994
|
|
|
995
|
A sync is just a pull and a push that happen at the same time. |
|
996
|
The first three steps of a pull are combined with the first five steps |
|
997
|
of a push. Steps (4) through (7) of a pull are combined with steps |
|
998
|
(5) through (8) of a push. And steps (8) through (10) of a pull |
|
999
|
are combined with step (9) of a push. |
|
1000
|
|
|
1001
|
<h3 id="uv-strategy">5.4 Unversioned File Sync</h3> |
|
1002
|
|
|
1003
|
"Unversioned files" are files held in the repository |
|
1004
|
where only the most recent version of the file is kept rather than |
|
1005
|
the entire change history. Unversioned files are intended to be |
|
1006
|
used to store ephemeral content, such as compiled binaries of the |
|
1007
|
most recent release. |
|
1008
|
|
|
1009
|
Unversioned files are identified by name and timestamp (mtime). |
|
1010
|
Only the most recent version of each file (the version with |
|
1011
|
the largest mtime value) is retained. |
|
1012
|
|
|
1013
|
Unversioned files are synchronized using the |
|
1014
|
[/help?cmd=unversioned|fossil unversioned sync] command. |
|
1015
|
|
|
1016
|
A schematic of an unversioned file synchronization is as follows: |
|
1017
|
|
|
1018
|
<ol> |
|
1019
|
<li>The client sends a "pragma uv-hash" card to the server. The argument |
|
1020
|
to the uv-hash pragma is a hash of all filesnames, mtimes, and |
|
1021
|
content hashes for the unversioned files held by the client. |
|
1022
|
<hr> |
|
1023
|
<li>If the unversioned content hash from the client matches the unversioned |
|
1024
|
content hash on the server, then nothing needs to be done and the |
|
1025
|
server no-ops. But if the hashes are different, then the server |
|
1026
|
replies with either a uv-pull-only or a uv-push-ok pragma followed by |
|
1027
|
uvigot cards for all unversioned files held on the server. |
|
1028
|
<hr> |
|
1029
|
<li>The client examines the uvigot cards received from the server and |
|
1030
|
determines which unversioned files need to be exchanged in order |
|
1031
|
to bring the client and server into synchronization. The client |
|
1032
|
then sends appropriate "uvgimme" or "uvfile" cards back to the |
|
1033
|
server. |
|
1034
|
<hr> |
|
1035
|
<li>The server updates its unversioned file store with received "uvfile" |
|
1036
|
cards and answers "uvgimme" cards with "uvfile" cards in its reply. |
|
1037
|
</ol> |
|
1038
|
|
|
1039
|
The last two steps might be repeated multiple |
|
1040
|
times if there is more unversioned content to be transferred than will |
|
1041
|
fit comfortably in a single HTTP request. |
|
1042
|
|
|
1043
|
<h2 id="summary">6.0 Summary</h2> |
|
1044
|
|
|
1045
|
Here are the key points of the synchronization protocol: |
|
1046
|
|
|
1047
|
<ol> |
|
1048
|
<li>The client sends one or more PUSH HTTP requests to the server. |
|
1049
|
The request and reply content type is "application/x-fossil". |
|
1050
|
<li>HTTP request content is compressed using zlib. |
|
1051
|
<li>The content of request and reply consists of cards with one |
|
1052
|
card per line. |
|
1053
|
<li>Card formats are: |
|
1054
|
<ul> |
|
1055
|
<li> <b>login</b> <i>userid nonce signature</i> |
|
1056
|
<li> <b>push</b> <i>servercode projectcode</i> |
|
1057
|
<li> <b>pull</b> <i>servercode projectcode</i> |
|
1058
|
<li> <b>clone</b> |
|
1059
|
<li> <b>clone</b> <i>protocol-version sequence-number</i> |
|
1060
|
<li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> |
|
1061
|
<li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> |
|
1062
|
<li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> |
|
1063
|
<li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> |
|
1064
|
<li> <b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i> |
|
1065
|
<li> <b>private</b> |
|
1066
|
<li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>? |
|
1067
|
<li> <b>uvigot</b> <i>name mtime hash size</i> |
|
1068
|
<li> <b>gimme</b> <i>artifact-id</i> |
|
1069
|
<li> <b>uvgimme</b> <i>name</i> |
|
1070
|
<li> <b>cookie</b> <i>cookie-text</i> |
|
1071
|
<li> <b>reqconfig</b> <i>parameter-name</i> |
|
1072
|
<li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i> |
|
1073
|
<li> <b>pragma</b> <i>name</i> <i>value...</i> |
|
1074
|
<li> <b>error</b> <i>error-message</i> |
|
1075
|
<li> <b>message</b> <i>text-messate</i> |
|
1076
|
<li> <b>#</b> <i>arbitrary-text...</i> |
|
1077
|
</ul> |
|
1078
|
<li>Phantoms are artifacts that a repository knows exist but does not possess. |
|
1079
|
<li>Clusters are artifacts that contain IDs of other artifacts. |
|
1080
|
<li>Clusters are created automatically on the server during a pull. |
|
1081
|
<li>Repositories keep track of all artifacts that are not named in any |
|
1082
|
cluster and send igot messages for those artifacts. |
|
1083
|
<li>Repositories keep track of all the phantoms they hold and send |
|
1084
|
gimme messages for those artifacts. |
|
1085
|
</ol> |
|
1086
|
|
|
1087
|
<h2 id="troubleshooting">7.0 Troubleshooting And Debugging Hints</h2> |
|
1088
|
|
|
1089
|
If you run the [/help?cmd=sync|fossil sync] command |
|
1090
|
(or [/help?cmd=pull|pull] or [/help?cmd=push|push] or |
|
1091
|
[/help?cmd=clone|clone]) with the --httptrace option, Fossil |
|
1092
|
will keep a copy of each HTTP request and reply in files |
|
1093
|
named: |
|
1094
|
<ul> |
|
1095
|
<li> <tt>http-request-</tt><i>N</i><tt>.txt</tt> |
|
1096
|
<li> <tt>http-reply-</tt><i>N</i><tt>.txt</tt> |
|
1097
|
</ul> |
|
1098
|
|
|
1099
|
In the above, <i>N</i> is an integer that increments with each |
|
1100
|
round-trip. If you are having trouble on the server side, |
|
1101
|
you can run the "[/help?cmd=test-http|fossil test-http]" command in a |
|
1102
|
debugger using one the "http-request-N.txt" files as input and |
|
1103
|
single step through the processing performed by the server. |
|
1104
|
|
|
1105
|
The "--transport-command CMD" option on [/help?cmd=sync|fossil sync] |
|
1106
|
(and similar) causes the external program "CMD" to be used to move |
|
1107
|
the sync message to the server and retrieve the sync reply. The |
|
1108
|
CMD is given three arguments: |
|
1109
|
<ol> |
|
1110
|
<li> The URL of the server |
|
1111
|
<li> The name of a temporary file that contains the output-bound sync |
|
1112
|
protocol text, with the HTTP headers |
|
1113
|
<li> The name of a temporary file into which the CMD should write the |
|
1114
|
reply sync protocol text, again without any HTTP headers |
|
1115
|
</ol> |
|
1116
|
|
|
1117
|
In a complex debugging situation, you can run the command |
|
1118
|
"fossil sync --transport-command ./debugging_script" where |
|
1119
|
"debugging_script" is some script of your own that invokes |
|
1120
|
the anomalous behavior you are trying to debug. |
|
1121
|
|