Fossil SCM
Added fragment identifiers to the h2 and h3 levels of the sync doc, for referring to individual sections.
Commit
5ca63acf9f7a99c5c06be584301155646fbe4055841cd60adf54b3e16b3de332
Parent
3ec73a0d0d1ee47…
1 file changed
+34
-37
+34
-37
| --- www/sync.wiki | ||
| +++ www/sync.wiki | ||
| @@ -48,11 +48,11 @@ | ||
| 48 | 48 | |
| 49 | 49 | If you are already familiar with CRDTs and were wondering if Fossil |
| 50 | 50 | used them, the answer is "yes". We just don't call them by that name. |
| 51 | 51 | |
| 52 | 52 | |
| 53 | -<h2>2.0 Transport</h2> | |
| 53 | +<h2 id="transport">2.0 Transport</h2> | |
| 54 | 54 | |
| 55 | 55 | All communication between client and server is via HTTP requests. |
| 56 | 56 | The server is listening for incoming HTTP requests. The client |
| 57 | 57 | issues one or more HTTP requests and receives replies for each |
| 58 | 58 | request. |
| @@ -82,11 +82,11 @@ | ||
| 82 | 82 | processor housed in a datacenter, nor does the client need to be a desktop |
| 83 | 83 | or handheld device. For the purposes of this article "client" simply means |
| 84 | 84 | the repository that initiates the conversation and "server" is the repository |
| 85 | 85 | that responds. Nothing more. |
| 86 | 86 | |
| 87 | -<h4>2.0.1 HTTPS Transport</h4> | |
| 87 | +<h4 id="https">2.0.1 HTTPS Transport</h4> | |
| 88 | 88 | |
| 89 | 89 | HTTPS differs from HTTP only in that the HTTPS protocol is |
| 90 | 90 | encrypted as it travels over the wire. The underlying protocol |
| 91 | 91 | is the same. This document describes only the underlying, unencrypted |
| 92 | 92 | messages that go client to server and back to client. |
| @@ -94,11 +94,11 @@ | ||
| 94 | 94 | in this document. |
| 95 | 95 | |
| 96 | 96 | Fossil includes built-in |
| 97 | 97 | [./ssl-server.md|support for HTTPS encryption] in both client and server. |
| 98 | 98 | |
| 99 | -<h4>2.0.2 SSH Transport</h4> | |
| 99 | +<h4 id="ssh">2.0.2 SSH Transport</h4> | |
| 100 | 100 | |
| 101 | 101 | When doing a sync using an "ssh:..." URL, the same HTTP transport protocol |
| 102 | 102 | is used. Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh] |
| 103 | 103 | to start an instance of the [/help?cmd=test-http|fossil test-http] command |
| 104 | 104 | running on the remote machine. It then sends HTTP requests and gets back HTTP |
| @@ -105,11 +105,11 @@ | ||
| 105 | 105 | replies over the SSH connection, rather than sending and receiving over an |
| 106 | 106 | internet socket. To see the specific "ssh" command that the Fossil client |
| 107 | 107 | runs in order to set up a connection, add either of the the "--httptrace" or |
| 108 | 108 | "--sshtrace" options to the "fossil sync" command line. |
| 109 | 109 | |
| 110 | -<h4>2.0.3 FILE Transport</h4> | |
| 110 | +<h4 id="file">2.0.3 FILE Transport</h4> | |
| 111 | 111 | |
| 112 | 112 | When doing a sync using a "file:..." URL, the same HTTP protocol is |
| 113 | 113 | still used. But instead of sending each HTTP request over a socket or |
| 114 | 114 | via SSH, the HTTP request is written into a temporary file. The client |
| 115 | 115 | then invokes the [/help?cmd=http|fossil http] command in a subprocess |
| @@ -117,11 +117,11 @@ | ||
| 117 | 117 | the HTTP reply out of a temporary file on disk, and deletes the two |
| 118 | 118 | temporary files. To see the specific "fossil http" command that is run |
| 119 | 119 | in order to implement the "file:" transport, add the "--httptrace" |
| 120 | 120 | option to the "fossil sync" command. |
| 121 | 121 | |
| 122 | -<h3>2.1 Server Identification</h3> | |
| 122 | +<h3 id="srv-id">2.1 Server Identification</h3> | |
| 123 | 123 | |
| 124 | 124 | The server is identified by a URL argument that accompanies the |
| 125 | 125 | push, pull, or sync command on the client. (As a convenience to |
| 126 | 126 | users, the URL can be omitted on the client command and the same URL |
| 127 | 127 | from the most recent push, pull, or sync will be reused. This saves |
| @@ -137,11 +137,11 @@ | ||
| 137 | 137 | Then the URL that is really used to do the synchronization will |
| 138 | 138 | be: |
| 139 | 139 | |
| 140 | 140 | <pre>https://fossil-scm.org/fossil/xfer</pre> |
| 141 | 141 | |
| 142 | -<h3>2.2 HTTP Request Format</h3> | |
| 142 | +<h3 id="req-format">2.2 HTTP Request Format</h3> | |
| 143 | 143 | |
| 144 | 144 | The client always sends a POST request to the server. The |
| 145 | 145 | general format of the POST request is as follows: |
| 146 | 146 | |
| 147 | 147 | <pre> |
| @@ -175,17 +175,17 @@ | ||
| 175 | 175 | <i><pre>content...</pre></i> |
| 176 | 176 | |
| 177 | 177 | The content type of the reply is always the same as the content type |
| 178 | 178 | of the request. |
| 179 | 179 | |
| 180 | -<h2>3.0 Fossil Synchronization Content</h2> | |
| 180 | +<h2 id="content">3.0 Fossil Synchronization Content</h2> | |
| 181 | 181 | |
| 182 | 182 | A synchronization request between a client and server consists of |
| 183 | 183 | one or more HTTP requests as described in the previous section. This |
| 184 | 184 | section details the "x-fossil" content type. |
| 185 | 185 | |
| 186 | -<h3>3.1 Line-oriented Format</h3> | |
| 186 | +<h3 id="lines">3.1 Line-oriented Format</h3> | |
| 187 | 187 | |
| 188 | 188 | The x-fossil content type consists of zero or more "cards". Cards |
| 189 | 189 | are separated by the newline character ("\n"). Leading and trailing |
| 190 | 190 | whitespace on a card is ignored. Blank cards are ignored. |
| 191 | 191 | |
| @@ -193,11 +193,11 @@ | ||
| 193 | 193 | The first token on each card is the operator. Subsequent tokens |
| 194 | 194 | are arguments. The set of operators understood by servers is slightly |
| 195 | 195 | different from the operators understood by clients, though the two |
| 196 | 196 | are very similar. |
| 197 | 197 | |
| 198 | -<h3>3.2 Login Cards</h3> | |
| 198 | +<h3 id="login">3.2 Login Cards</h3> | |
| 199 | 199 | |
| 200 | 200 | Every message from client to server begins with one or more login |
| 201 | 201 | cards. Each login card has the following format: |
| 202 | 202 | |
| 203 | 203 | <pre><b>login</b> <i>userid nonce signature</i></pre> |
| @@ -217,21 +217,20 @@ | ||
| 217 | 217 | |
| 218 | 218 | Privileges are cumulative. There can be multiple successful |
| 219 | 219 | login cards. The session privilege is the union of all |
| 220 | 220 | privileges from all login cards. |
| 221 | 221 | |
| 222 | -<h3>3.3 File Cards</h3> | |
| 222 | +<h3 id="file">3.3 File Cards</h3> | |
| 223 | 223 | |
| 224 | 224 | Artifacts are transferred using either "file" cards, or "cfile" |
| 225 | 225 | or "uvfile" cards. |
| 226 | 226 | The name "file" card comes from the fact that most artifacts correspond to |
| 227 | 227 | files that are under version control. |
| 228 | 228 | The "cfile" name is an abbreviation for "compressed file". |
| 229 | 229 | The "uvfile" name is an abbreviation for "unversioned file". |
| 230 | 230 | |
| 231 | - | |
| 232 | -<h4>3.3.1 Ordinary File Cards</h4> | |
| 231 | +<h4 id="ordinary-fc">3.3.1 Ordinary File Cards</h4> | |
| 233 | 232 | |
| 234 | 233 | For sync protocols, artifacts are transferred using "file" |
| 235 | 234 | cards. File cards come in two different formats depending |
| 236 | 235 | on whether the artifact is sent directly or as a |
| 237 | 236 | [./delta_format.wiki|delta] from some |
| @@ -263,11 +262,11 @@ | ||
| 263 | 262 | File cards are sent in both directions: client to server and |
| 264 | 263 | server to client. A delta might be sent before the source of |
| 265 | 264 | the delta, so both client and server should remember deltas |
| 266 | 265 | and be able to apply them when their source arrives. |
| 267 | 266 | |
| 268 | -<h4>3.3.2 Compressed File Cards</h4> | |
| 267 | +<h4 id="compressed-fc">3.3.2 Compressed File Cards</h4> | |
| 269 | 268 | |
| 270 | 269 | A client that sends a clone protocol version "3" or greater will |
| 271 | 270 | receive artifacts as "cfile" cards while cloning. This card was |
| 272 | 271 | introduced to improve the speed of the transfer of content by sending the |
| 273 | 272 | compressed artifact directly from the server database to the client. |
| @@ -296,25 +295,24 @@ | ||
| 296 | 295 | delta artifact. |
| 297 | 296 | |
| 298 | 297 | Unlike file cards, cfile cards are only sent in one direction during a |
| 299 | 298 | clone from server to client for clone protocol version "3" or greater. |
| 300 | 299 | |
| 301 | -<h4>3.3.3 Private artifacts</h4> | |
| 300 | +<h4 id="private">3.3.3 Private artifacts</h4> | |
| 302 | 301 | |
| 303 | 302 | "Private" content consist of artifacts that are not normally synced. |
| 304 | 303 | However, private content will be synced when the |
| 305 | 304 | the [/help?cmd=sync|fossil sync] command includes the "--private" option. |
| 306 | - | |
| 307 | 305 | |
| 308 | 306 | Private content is marked by a "private" card: |
| 309 | 307 | |
| 310 | 308 | <pre><b>private</b></pre> |
| 311 | 309 | |
| 312 | 310 | The private card has no arguments and must directly precede a |
| 313 | 311 | file card that contains the private content. |
| 314 | 312 | |
| 315 | -<h4>3.3.4 Unversioned File Cards</h4> | |
| 313 | +<h4 id="uv-fc">3.3.4 Unversioned File Cards</h4> | |
| 316 | 314 | |
| 317 | 315 | Unversioned content is sent in both directions (client to server and |
| 318 | 316 | server to client) using "uvfile" cards in the following format: |
| 319 | 317 | |
| 320 | 318 | <pre><b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i></pre> |
| @@ -342,11 +340,11 @@ | ||
| 342 | 340 | |
| 343 | 341 | Servers send uvfile cards in response to uvgimme cards received from |
| 344 | 342 | the client. Clients send uvfile cards when they determine that the server |
| 345 | 343 | needs the content based on uvigot cards previously received from the server. |
| 346 | 344 | |
| 347 | -<h3>3.4 Push and Pull Cards</h3> | |
| 345 | +<h3 id="push" name="pull">3.4 Push and Pull Cards</h3> | |
| 348 | 346 | |
| 349 | 347 | Among the first cards in a client-to-server message are |
| 350 | 348 | the push and pull cards. The push card tells the server that |
| 351 | 349 | the client is pushing content. The pull card tells the server |
| 352 | 350 | that the client wants to pull content. In the event of a sync, |
| @@ -367,11 +365,11 @@ | ||
| 367 | 365 | during a clone. This is how the client determines what project |
| 368 | 366 | code to put in the new repository it is constructing. |
| 369 | 367 | |
| 370 | 368 | The <i>servercode</i> argument is currently unused. |
| 371 | 369 | |
| 372 | -<h3>3.5 Clone Cards</h3> | |
| 370 | +<h3 id="clones">3.5 Clone Cards</h3> | |
| 373 | 371 | |
| 374 | 372 | A clone card works like a pull card in that it is sent from |
| 375 | 373 | client to server in order to tell the server that the client |
| 376 | 374 | wants to pull content. The clone card comes in two formats. Older |
| 377 | 375 | clients use the no-argument format and newer clients use the |
| @@ -424,11 +422,11 @@ | ||
| 424 | 422 | artifacts) but is too slow and unwieldy for larger repositories. |
| 425 | 423 | The version 2 protocol is an effort to improve performance. Further |
| 426 | 424 | performance improvements with higher-numbered clone protocols are |
| 427 | 425 | possible in future versions of Fossil. |
| 428 | 426 | |
| 429 | -<h3>3.6 Igot Cards</h3> | |
| 427 | +<h3 id="igot">3.6 Igot Cards</h3> | |
| 430 | 428 | |
| 431 | 429 | An igot card can be sent from either client to server or from |
| 432 | 430 | server to client in order to indicate that the sender holds a copy |
| 433 | 431 | of a particular artifact. The format is: |
| 434 | 432 | |
| @@ -479,11 +477,11 @@ | ||
| 479 | 477 | If a client-to-server transmission is needed, the client schedules that |
| 480 | 478 | transfer to occur on a subsequent HTTP request. If a server-to-client |
| 481 | 479 | transfer is needed, then the client sends a "uvgimme" card back to the |
| 482 | 480 | server to request the file content. |
| 483 | 481 | |
| 484 | -<h3>3.7 Gimme Cards</h3> | |
| 482 | +<h3 id="gimme">3.7 Gimme Cards</h3> | |
| 485 | 483 | |
| 486 | 484 | A gimme card is sent from either client to server or from server |
| 487 | 485 | to client. The gimme card asks the receiver to send a particular |
| 488 | 486 | artifact back to the sender. The format of a gimme card is this: |
| 489 | 487 | |
| @@ -514,11 +512,11 @@ | ||
| 514 | 512 | The <i>name</i> is the name of the unversioned file found on the |
| 515 | 513 | server that the client would like to have. When a server sees a |
| 516 | 514 | uvgimme card, it normally responses with a uvfile card, though it might |
| 517 | 515 | also send another uvigot card if the HTTP reply is already oversized. |
| 518 | 516 | |
| 519 | -<h3>3.8 Cookie Cards</h3> | |
| 517 | +<h3 id="cookie">3.8 Cookie Cards</h3> | |
| 520 | 518 | |
| 521 | 519 | A cookie card can be used by a server to record a small amount |
| 522 | 520 | of state information on a client. The server sends a cookie to the |
| 523 | 521 | client. The client sends the same cookie back to the server on |
| 524 | 522 | its next request. The cookie card has a single argument which |
| @@ -534,11 +532,11 @@ | ||
| 534 | 532 | cookie and the server must structure the cookie payload in such |
| 535 | 533 | a way that it can tell if the cookie it sees is its own cookie or |
| 536 | 534 | a cookie from another server. (Typically the server will embed |
| 537 | 535 | its servercode as part of the cookie.) |
| 538 | 536 | |
| 539 | -<h3>3.9 Request-Configuration Cards</h3> | |
| 537 | +<h3 id="reqconfig">3.9 Request-Configuration Cards</h3> | |
| 540 | 538 | |
| 541 | 539 | A request-configuration or "reqconfig" card is sent from client to |
| 542 | 540 | server in order to request that the server send back "configuration" |
| 543 | 541 | data. "Configuration" data is information about users or website |
| 544 | 542 | appearance or other administrative details which are not part of the |
| @@ -643,11 +641,11 @@ | ||
| 643 | 641 | intended to be evaluated by the client. |
| 644 | 642 | |
| 645 | 643 | The @user and @concealed configuration items contain sensitive information |
| 646 | 644 | and are ignored for clients without sufficient privilege. |
| 647 | 645 | |
| 648 | -<h3>3.10 Configuration Cards</h3> | |
| 646 | +<h3 id="config">3.10 Configuration Cards</h3> | |
| 649 | 647 | |
| 650 | 648 | A "config" card is used to send configuration information from client |
| 651 | 649 | to server (in response to a "fossil configuration push" command) or |
| 652 | 650 | from server to client (in response to a "fossil configuration pull" or |
| 653 | 651 | "fossil clone" command). The format is as follows: |
| @@ -661,11 +659,11 @@ | ||
| 661 | 659 | it had sent a corresponding reqconfig card in its request. |
| 662 | 660 | |
| 663 | 661 | The content of the configuration item is used to overwrite the |
| 664 | 662 | corresponding configuration data in the receiver. |
| 665 | 663 | |
| 666 | -<h3>3.11 Pragma Cards</h3> | |
| 664 | +<h3 id="pragma">3.11 Pragma Cards</h3> | |
| 667 | 665 | |
| 668 | 666 | The client may try to influence the behavior of the server by |
| 669 | 667 | issuing a pragma card: |
| 670 | 668 | |
| 671 | 669 | <pre> |
| @@ -753,16 +751,16 @@ | ||
| 753 | 751 | The ci-unlock pragma helps to avoid false-positive lock warnings |
| 754 | 752 | that might arise if a check-in is aborted and then restarted |
| 755 | 753 | on a branch. |
| 756 | 754 | </ol> |
| 757 | 755 | |
| 758 | -<h3>3.12 Comment Cards</h3> | |
| 756 | +<h3 id="comment">3.12 Comment Cards</h3> | |
| 759 | 757 | |
| 760 | 758 | Any card that begins with "#" (ASCII 0x23) is a comment card and |
| 761 | 759 | is silently ignored. |
| 762 | 760 | |
| 763 | -<h3>3.13 Message and Error Cards</h3> | |
| 761 | +<h3 id="error">3.13 Message and Error Cards</h3> | |
| 764 | 762 | |
| 765 | 763 | If the server discovers anything wrong with a request, it generates |
| 766 | 764 | an error card in its reply. When the client sees the error card, |
| 767 | 765 | it displays an error message to the user and aborts the sync |
| 768 | 766 | operation. An error card looks like this: |
| @@ -787,16 +785,16 @@ | ||
| 787 | 785 | <b>message</b> <i>message-text</i> |
| 788 | 786 | </pre> |
| 789 | 787 | |
| 790 | 788 | The message-text uses the same format as an error message. |
| 791 | 789 | |
| 792 | -<h3>3.14 Unknown Cards</h3> | |
| 790 | +<h3 id="unknown">3.14 Unknown Cards</h3> | |
| 793 | 791 | |
| 794 | 792 | If either the client or the server sees a card that is not |
| 795 | 793 | described above, then it generates an error and aborts. |
| 796 | 794 | |
| 797 | -<h2>4.0 Phantoms And Clusters</h2> | |
| 795 | +<h2 id="phantoms" name="clusters">4.0 Phantoms And Clusters</h2> | |
| 798 | 796 | |
| 799 | 797 | When a repository knows that an artifact exists and knows the ID of |
| 800 | 798 | that artifact, but it does not know the artifact content, then it stores that |
| 801 | 799 | artifact as a "phantom". A repository will typically create a phantom when |
| 802 | 800 | it receives an igot card for an artifact that it does not hold or when it |
| @@ -827,11 +825,11 @@ | ||
| 827 | 825 | exactly is not a cluster. There must be no extra whitespace in |
| 828 | 826 | the artifact. There must be one or more M cards. There must be a |
| 829 | 827 | single Z card with a correct MD5 checksum. And all cards must |
| 830 | 828 | be in strict lexicographical order. |
| 831 | 829 | |
| 832 | -<h3>4.1 The Unclustered Table</h3> | |
| 830 | +<h3 id="unclustered">4.1 The Unclustered Table</h3> | |
| 833 | 831 | |
| 834 | 832 | Every repository maintains a table named "<b>unclustered</b>" |
| 835 | 833 | which records the identity of every artifact and phantom it holds that is not |
| 836 | 834 | mentioned in a cluster. The entries in the unclustered table can |
| 837 | 835 | be thought of as leaves on a tree of artifacts. Some of the unclustered |
| @@ -838,13 +836,13 @@ | ||
| 838 | 836 | artifacts will be other clusters. Those clusters may contain other clusters, |
| 839 | 837 | which might contain still more clusters, and so forth. Beginning |
| 840 | 838 | with the artifacts in the unclustered table, one can follow the chain |
| 841 | 839 | of clusters to find every artifact in the repository. |
| 842 | 840 | |
| 843 | -<h2>5.0 Synchronization Strategies</h2> | |
| 841 | +<h2 id="strategies">5.0 Synchronization Strategies</h2> | |
| 844 | 842 | |
| 845 | -<h3>5.1 Pull</h3> | |
| 843 | +<h3 id="pull-strategy">5.1 Pull</h3> | |
| 846 | 844 | |
| 847 | 845 | A typical pull operation proceeds as shown below. Details |
| 848 | 846 | of the actual implementation may very slightly but the gist of |
| 849 | 847 | a pull is captured in the following steps: |
| 850 | 848 | |
| @@ -892,11 +890,11 @@ | ||
| 892 | 890 | protocol will continue to work even if there are multiple servers |
| 893 | 891 | or if servers and clients sometimes change roles. The only negative |
| 894 | 892 | effects of these unusual arrangements is that more than the minimum |
| 895 | 893 | number of clusters might be generated. |
| 896 | 894 | |
| 897 | -<h3>5.2 Push</h3> | |
| 895 | +<h3 id="push-stragegy">5.2 Push</h3> | |
| 898 | 896 | |
| 899 | 897 | A typical push operation proceeds roughly as shown below. As |
| 900 | 898 | with a pull, the actual implementation may vary slightly. |
| 901 | 899 | |
| 902 | 900 | <ol> |
| @@ -926,19 +924,19 @@ | ||
| 926 | 924 | server knows all artifacts that exist on the client. Also, as with |
| 927 | 925 | pull, the client attempts to keep the size of the request from |
| 928 | 926 | growing too large by suppressing file cards once the |
| 929 | 927 | size of the request reaches 1MB. |
| 930 | 928 | |
| 931 | -<h3 id="sync">5.3 Sync</h3> | |
| 929 | +<h3 id="sync-strategy">5.3 Sync</h3> | |
| 932 | 930 | |
| 933 | 931 | A sync is just a pull and a push that happen at the same time. |
| 934 | 932 | The first three steps of a pull are combined with the first five steps |
| 935 | 933 | of a push. Steps (4) through (7) of a pull are combined with steps |
| 936 | 934 | (5) through (8) of a push. And steps (8) through (10) of a pull |
| 937 | 935 | are combined with step (9) of a push. |
| 938 | 936 | |
| 939 | -<h3>5.4 Unversioned File Sync</h3> | |
| 937 | +<h3 id="uv-strategy">5.4 Unversioned File Sync</h3> | |
| 940 | 938 | |
| 941 | 939 | "Unversioned files" are files held in the repository |
| 942 | 940 | where only the most recent version of the file is kept rather than |
| 943 | 941 | the entire change history. Unversioned files are intended to be |
| 944 | 942 | used to store ephemeral content, such as compiled binaries of the |
| @@ -976,11 +974,11 @@ | ||
| 976 | 974 | |
| 977 | 975 | The last two steps might be repeated multiple |
| 978 | 976 | times if there is more unversioned content to be transferred than will |
| 979 | 977 | fit comfortably in a single HTTP request. |
| 980 | 978 | |
| 981 | -<h2>6.0 Summary</h2> | |
| 979 | +<h2 id="summary">6.0 Summary</h2> | |
| 982 | 980 | |
| 983 | 981 | Here are the key points of the synchronization protocol: |
| 984 | 982 | |
| 985 | 983 | <ol> |
| 986 | 984 | <li>The client sends one or more PUSH HTTP requests to the server. |
| @@ -1020,12 +1018,11 @@ | ||
| 1020 | 1018 | cluster and send igot messages for those artifacts. |
| 1021 | 1019 | <li>Repositories keep track of all the phantoms they hold and send |
| 1022 | 1020 | gimme messages for those artifacts. |
| 1023 | 1021 | </ol> |
| 1024 | 1022 | |
| 1025 | -<h2>7.0 Troubleshooting And Debugging Hints</h2> | |
| 1026 | - | |
| 1023 | +<h2 id="troubleshooting">7.0 Troubleshooting And Debugging Hints</h2> | |
| 1027 | 1024 | |
| 1028 | 1025 | If you run the [/help?cmd=sync|fossil sync] command |
| 1029 | 1026 | (or [/help?cmd=pull|pull] or [/help?cmd=push|push] or |
| 1030 | 1027 | [/help?cmd=clone|clone]) with the --httptrace option, Fossil |
| 1031 | 1028 | will keep a copy of each HTTP request and reply in files |
| 1032 | 1029 |
| --- www/sync.wiki | |
| +++ www/sync.wiki | |
| @@ -48,11 +48,11 @@ | |
| 48 | |
| 49 | If you are already familiar with CRDTs and were wondering if Fossil |
| 50 | used them, the answer is "yes". We just don't call them by that name. |
| 51 | |
| 52 | |
| 53 | <h2>2.0 Transport</h2> |
| 54 | |
| 55 | All communication between client and server is via HTTP requests. |
| 56 | The server is listening for incoming HTTP requests. The client |
| 57 | issues one or more HTTP requests and receives replies for each |
| 58 | request. |
| @@ -82,11 +82,11 @@ | |
| 82 | processor housed in a datacenter, nor does the client need to be a desktop |
| 83 | or handheld device. For the purposes of this article "client" simply means |
| 84 | the repository that initiates the conversation and "server" is the repository |
| 85 | that responds. Nothing more. |
| 86 | |
| 87 | <h4>2.0.1 HTTPS Transport</h4> |
| 88 | |
| 89 | HTTPS differs from HTTP only in that the HTTPS protocol is |
| 90 | encrypted as it travels over the wire. The underlying protocol |
| 91 | is the same. This document describes only the underlying, unencrypted |
| 92 | messages that go client to server and back to client. |
| @@ -94,11 +94,11 @@ | |
| 94 | in this document. |
| 95 | |
| 96 | Fossil includes built-in |
| 97 | [./ssl-server.md|support for HTTPS encryption] in both client and server. |
| 98 | |
| 99 | <h4>2.0.2 SSH Transport</h4> |
| 100 | |
| 101 | When doing a sync using an "ssh:..." URL, the same HTTP transport protocol |
| 102 | is used. Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh] |
| 103 | to start an instance of the [/help?cmd=test-http|fossil test-http] command |
| 104 | running on the remote machine. It then sends HTTP requests and gets back HTTP |
| @@ -105,11 +105,11 @@ | |
| 105 | replies over the SSH connection, rather than sending and receiving over an |
| 106 | internet socket. To see the specific "ssh" command that the Fossil client |
| 107 | runs in order to set up a connection, add either of the the "--httptrace" or |
| 108 | "--sshtrace" options to the "fossil sync" command line. |
| 109 | |
| 110 | <h4>2.0.3 FILE Transport</h4> |
| 111 | |
| 112 | When doing a sync using a "file:..." URL, the same HTTP protocol is |
| 113 | still used. But instead of sending each HTTP request over a socket or |
| 114 | via SSH, the HTTP request is written into a temporary file. The client |
| 115 | then invokes the [/help?cmd=http|fossil http] command in a subprocess |
| @@ -117,11 +117,11 @@ | |
| 117 | the HTTP reply out of a temporary file on disk, and deletes the two |
| 118 | temporary files. To see the specific "fossil http" command that is run |
| 119 | in order to implement the "file:" transport, add the "--httptrace" |
| 120 | option to the "fossil sync" command. |
| 121 | |
| 122 | <h3>2.1 Server Identification</h3> |
| 123 | |
| 124 | The server is identified by a URL argument that accompanies the |
| 125 | push, pull, or sync command on the client. (As a convenience to |
| 126 | users, the URL can be omitted on the client command and the same URL |
| 127 | from the most recent push, pull, or sync will be reused. This saves |
| @@ -137,11 +137,11 @@ | |
| 137 | Then the URL that is really used to do the synchronization will |
| 138 | be: |
| 139 | |
| 140 | <pre>https://fossil-scm.org/fossil/xfer</pre> |
| 141 | |
| 142 | <h3>2.2 HTTP Request Format</h3> |
| 143 | |
| 144 | The client always sends a POST request to the server. The |
| 145 | general format of the POST request is as follows: |
| 146 | |
| 147 | <pre> |
| @@ -175,17 +175,17 @@ | |
| 175 | <i><pre>content...</pre></i> |
| 176 | |
| 177 | The content type of the reply is always the same as the content type |
| 178 | of the request. |
| 179 | |
| 180 | <h2>3.0 Fossil Synchronization Content</h2> |
| 181 | |
| 182 | A synchronization request between a client and server consists of |
| 183 | one or more HTTP requests as described in the previous section. This |
| 184 | section details the "x-fossil" content type. |
| 185 | |
| 186 | <h3>3.1 Line-oriented Format</h3> |
| 187 | |
| 188 | The x-fossil content type consists of zero or more "cards". Cards |
| 189 | are separated by the newline character ("\n"). Leading and trailing |
| 190 | whitespace on a card is ignored. Blank cards are ignored. |
| 191 | |
| @@ -193,11 +193,11 @@ | |
| 193 | The first token on each card is the operator. Subsequent tokens |
| 194 | are arguments. The set of operators understood by servers is slightly |
| 195 | different from the operators understood by clients, though the two |
| 196 | are very similar. |
| 197 | |
| 198 | <h3>3.2 Login Cards</h3> |
| 199 | |
| 200 | Every message from client to server begins with one or more login |
| 201 | cards. Each login card has the following format: |
| 202 | |
| 203 | <pre><b>login</b> <i>userid nonce signature</i></pre> |
| @@ -217,21 +217,20 @@ | |
| 217 | |
| 218 | Privileges are cumulative. There can be multiple successful |
| 219 | login cards. The session privilege is the union of all |
| 220 | privileges from all login cards. |
| 221 | |
| 222 | <h3>3.3 File Cards</h3> |
| 223 | |
| 224 | Artifacts are transferred using either "file" cards, or "cfile" |
| 225 | or "uvfile" cards. |
| 226 | The name "file" card comes from the fact that most artifacts correspond to |
| 227 | files that are under version control. |
| 228 | The "cfile" name is an abbreviation for "compressed file". |
| 229 | The "uvfile" name is an abbreviation for "unversioned file". |
| 230 | |
| 231 | |
| 232 | <h4>3.3.1 Ordinary File Cards</h4> |
| 233 | |
| 234 | For sync protocols, artifacts are transferred using "file" |
| 235 | cards. File cards come in two different formats depending |
| 236 | on whether the artifact is sent directly or as a |
| 237 | [./delta_format.wiki|delta] from some |
| @@ -263,11 +262,11 @@ | |
| 263 | File cards are sent in both directions: client to server and |
| 264 | server to client. A delta might be sent before the source of |
| 265 | the delta, so both client and server should remember deltas |
| 266 | and be able to apply them when their source arrives. |
| 267 | |
| 268 | <h4>3.3.2 Compressed File Cards</h4> |
| 269 | |
| 270 | A client that sends a clone protocol version "3" or greater will |
| 271 | receive artifacts as "cfile" cards while cloning. This card was |
| 272 | introduced to improve the speed of the transfer of content by sending the |
| 273 | compressed artifact directly from the server database to the client. |
| @@ -296,25 +295,24 @@ | |
| 296 | delta artifact. |
| 297 | |
| 298 | Unlike file cards, cfile cards are only sent in one direction during a |
| 299 | clone from server to client for clone protocol version "3" or greater. |
| 300 | |
| 301 | <h4>3.3.3 Private artifacts</h4> |
| 302 | |
| 303 | "Private" content consist of artifacts that are not normally synced. |
| 304 | However, private content will be synced when the |
| 305 | the [/help?cmd=sync|fossil sync] command includes the "--private" option. |
| 306 | |
| 307 | |
| 308 | Private content is marked by a "private" card: |
| 309 | |
| 310 | <pre><b>private</b></pre> |
| 311 | |
| 312 | The private card has no arguments and must directly precede a |
| 313 | file card that contains the private content. |
| 314 | |
| 315 | <h4>3.3.4 Unversioned File Cards</h4> |
| 316 | |
| 317 | Unversioned content is sent in both directions (client to server and |
| 318 | server to client) using "uvfile" cards in the following format: |
| 319 | |
| 320 | <pre><b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i></pre> |
| @@ -342,11 +340,11 @@ | |
| 342 | |
| 343 | Servers send uvfile cards in response to uvgimme cards received from |
| 344 | the client. Clients send uvfile cards when they determine that the server |
| 345 | needs the content based on uvigot cards previously received from the server. |
| 346 | |
| 347 | <h3>3.4 Push and Pull Cards</h3> |
| 348 | |
| 349 | Among the first cards in a client-to-server message are |
| 350 | the push and pull cards. The push card tells the server that |
| 351 | the client is pushing content. The pull card tells the server |
| 352 | that the client wants to pull content. In the event of a sync, |
| @@ -367,11 +365,11 @@ | |
| 367 | during a clone. This is how the client determines what project |
| 368 | code to put in the new repository it is constructing. |
| 369 | |
| 370 | The <i>servercode</i> argument is currently unused. |
| 371 | |
| 372 | <h3>3.5 Clone Cards</h3> |
| 373 | |
| 374 | A clone card works like a pull card in that it is sent from |
| 375 | client to server in order to tell the server that the client |
| 376 | wants to pull content. The clone card comes in two formats. Older |
| 377 | clients use the no-argument format and newer clients use the |
| @@ -424,11 +422,11 @@ | |
| 424 | artifacts) but is too slow and unwieldy for larger repositories. |
| 425 | The version 2 protocol is an effort to improve performance. Further |
| 426 | performance improvements with higher-numbered clone protocols are |
| 427 | possible in future versions of Fossil. |
| 428 | |
| 429 | <h3>3.6 Igot Cards</h3> |
| 430 | |
| 431 | An igot card can be sent from either client to server or from |
| 432 | server to client in order to indicate that the sender holds a copy |
| 433 | of a particular artifact. The format is: |
| 434 | |
| @@ -479,11 +477,11 @@ | |
| 479 | If a client-to-server transmission is needed, the client schedules that |
| 480 | transfer to occur on a subsequent HTTP request. If a server-to-client |
| 481 | transfer is needed, then the client sends a "uvgimme" card back to the |
| 482 | server to request the file content. |
| 483 | |
| 484 | <h3>3.7 Gimme Cards</h3> |
| 485 | |
| 486 | A gimme card is sent from either client to server or from server |
| 487 | to client. The gimme card asks the receiver to send a particular |
| 488 | artifact back to the sender. The format of a gimme card is this: |
| 489 | |
| @@ -514,11 +512,11 @@ | |
| 514 | The <i>name</i> is the name of the unversioned file found on the |
| 515 | server that the client would like to have. When a server sees a |
| 516 | uvgimme card, it normally responses with a uvfile card, though it might |
| 517 | also send another uvigot card if the HTTP reply is already oversized. |
| 518 | |
| 519 | <h3>3.8 Cookie Cards</h3> |
| 520 | |
| 521 | A cookie card can be used by a server to record a small amount |
| 522 | of state information on a client. The server sends a cookie to the |
| 523 | client. The client sends the same cookie back to the server on |
| 524 | its next request. The cookie card has a single argument which |
| @@ -534,11 +532,11 @@ | |
| 534 | cookie and the server must structure the cookie payload in such |
| 535 | a way that it can tell if the cookie it sees is its own cookie or |
| 536 | a cookie from another server. (Typically the server will embed |
| 537 | its servercode as part of the cookie.) |
| 538 | |
| 539 | <h3>3.9 Request-Configuration Cards</h3> |
| 540 | |
| 541 | A request-configuration or "reqconfig" card is sent from client to |
| 542 | server in order to request that the server send back "configuration" |
| 543 | data. "Configuration" data is information about users or website |
| 544 | appearance or other administrative details which are not part of the |
| @@ -643,11 +641,11 @@ | |
| 643 | intended to be evaluated by the client. |
| 644 | |
| 645 | The @user and @concealed configuration items contain sensitive information |
| 646 | and are ignored for clients without sufficient privilege. |
| 647 | |
| 648 | <h3>3.10 Configuration Cards</h3> |
| 649 | |
| 650 | A "config" card is used to send configuration information from client |
| 651 | to server (in response to a "fossil configuration push" command) or |
| 652 | from server to client (in response to a "fossil configuration pull" or |
| 653 | "fossil clone" command). The format is as follows: |
| @@ -661,11 +659,11 @@ | |
| 661 | it had sent a corresponding reqconfig card in its request. |
| 662 | |
| 663 | The content of the configuration item is used to overwrite the |
| 664 | corresponding configuration data in the receiver. |
| 665 | |
| 666 | <h3>3.11 Pragma Cards</h3> |
| 667 | |
| 668 | The client may try to influence the behavior of the server by |
| 669 | issuing a pragma card: |
| 670 | |
| 671 | <pre> |
| @@ -753,16 +751,16 @@ | |
| 753 | The ci-unlock pragma helps to avoid false-positive lock warnings |
| 754 | that might arise if a check-in is aborted and then restarted |
| 755 | on a branch. |
| 756 | </ol> |
| 757 | |
| 758 | <h3>3.12 Comment Cards</h3> |
| 759 | |
| 760 | Any card that begins with "#" (ASCII 0x23) is a comment card and |
| 761 | is silently ignored. |
| 762 | |
| 763 | <h3>3.13 Message and Error Cards</h3> |
| 764 | |
| 765 | If the server discovers anything wrong with a request, it generates |
| 766 | an error card in its reply. When the client sees the error card, |
| 767 | it displays an error message to the user and aborts the sync |
| 768 | operation. An error card looks like this: |
| @@ -787,16 +785,16 @@ | |
| 787 | <b>message</b> <i>message-text</i> |
| 788 | </pre> |
| 789 | |
| 790 | The message-text uses the same format as an error message. |
| 791 | |
| 792 | <h3>3.14 Unknown Cards</h3> |
| 793 | |
| 794 | If either the client or the server sees a card that is not |
| 795 | described above, then it generates an error and aborts. |
| 796 | |
| 797 | <h2>4.0 Phantoms And Clusters</h2> |
| 798 | |
| 799 | When a repository knows that an artifact exists and knows the ID of |
| 800 | that artifact, but it does not know the artifact content, then it stores that |
| 801 | artifact as a "phantom". A repository will typically create a phantom when |
| 802 | it receives an igot card for an artifact that it does not hold or when it |
| @@ -827,11 +825,11 @@ | |
| 827 | exactly is not a cluster. There must be no extra whitespace in |
| 828 | the artifact. There must be one or more M cards. There must be a |
| 829 | single Z card with a correct MD5 checksum. And all cards must |
| 830 | be in strict lexicographical order. |
| 831 | |
| 832 | <h3>4.1 The Unclustered Table</h3> |
| 833 | |
| 834 | Every repository maintains a table named "<b>unclustered</b>" |
| 835 | which records the identity of every artifact and phantom it holds that is not |
| 836 | mentioned in a cluster. The entries in the unclustered table can |
| 837 | be thought of as leaves on a tree of artifacts. Some of the unclustered |
| @@ -838,13 +836,13 @@ | |
| 838 | artifacts will be other clusters. Those clusters may contain other clusters, |
| 839 | which might contain still more clusters, and so forth. Beginning |
| 840 | with the artifacts in the unclustered table, one can follow the chain |
| 841 | of clusters to find every artifact in the repository. |
| 842 | |
| 843 | <h2>5.0 Synchronization Strategies</h2> |
| 844 | |
| 845 | <h3>5.1 Pull</h3> |
| 846 | |
| 847 | A typical pull operation proceeds as shown below. Details |
| 848 | of the actual implementation may very slightly but the gist of |
| 849 | a pull is captured in the following steps: |
| 850 | |
| @@ -892,11 +890,11 @@ | |
| 892 | protocol will continue to work even if there are multiple servers |
| 893 | or if servers and clients sometimes change roles. The only negative |
| 894 | effects of these unusual arrangements is that more than the minimum |
| 895 | number of clusters might be generated. |
| 896 | |
| 897 | <h3>5.2 Push</h3> |
| 898 | |
| 899 | A typical push operation proceeds roughly as shown below. As |
| 900 | with a pull, the actual implementation may vary slightly. |
| 901 | |
| 902 | <ol> |
| @@ -926,19 +924,19 @@ | |
| 926 | server knows all artifacts that exist on the client. Also, as with |
| 927 | pull, the client attempts to keep the size of the request from |
| 928 | growing too large by suppressing file cards once the |
| 929 | size of the request reaches 1MB. |
| 930 | |
| 931 | <h3 id="sync">5.3 Sync</h3> |
| 932 | |
| 933 | A sync is just a pull and a push that happen at the same time. |
| 934 | The first three steps of a pull are combined with the first five steps |
| 935 | of a push. Steps (4) through (7) of a pull are combined with steps |
| 936 | (5) through (8) of a push. And steps (8) through (10) of a pull |
| 937 | are combined with step (9) of a push. |
| 938 | |
| 939 | <h3>5.4 Unversioned File Sync</h3> |
| 940 | |
| 941 | "Unversioned files" are files held in the repository |
| 942 | where only the most recent version of the file is kept rather than |
| 943 | the entire change history. Unversioned files are intended to be |
| 944 | used to store ephemeral content, such as compiled binaries of the |
| @@ -976,11 +974,11 @@ | |
| 976 | |
| 977 | The last two steps might be repeated multiple |
| 978 | times if there is more unversioned content to be transferred than will |
| 979 | fit comfortably in a single HTTP request. |
| 980 | |
| 981 | <h2>6.0 Summary</h2> |
| 982 | |
| 983 | Here are the key points of the synchronization protocol: |
| 984 | |
| 985 | <ol> |
| 986 | <li>The client sends one or more PUSH HTTP requests to the server. |
| @@ -1020,12 +1018,11 @@ | |
| 1020 | cluster and send igot messages for those artifacts. |
| 1021 | <li>Repositories keep track of all the phantoms they hold and send |
| 1022 | gimme messages for those artifacts. |
| 1023 | </ol> |
| 1024 | |
| 1025 | <h2>7.0 Troubleshooting And Debugging Hints</h2> |
| 1026 | |
| 1027 | |
| 1028 | If you run the [/help?cmd=sync|fossil sync] command |
| 1029 | (or [/help?cmd=pull|pull] or [/help?cmd=push|push] or |
| 1030 | [/help?cmd=clone|clone]) with the --httptrace option, Fossil |
| 1031 | will keep a copy of each HTTP request and reply in files |
| 1032 |
| --- www/sync.wiki | |
| +++ www/sync.wiki | |
| @@ -48,11 +48,11 @@ | |
| 48 | |
| 49 | If you are already familiar with CRDTs and were wondering if Fossil |
| 50 | used them, the answer is "yes". We just don't call them by that name. |
| 51 | |
| 52 | |
| 53 | <h2 id="transport">2.0 Transport</h2> |
| 54 | |
| 55 | All communication between client and server is via HTTP requests. |
| 56 | The server is listening for incoming HTTP requests. The client |
| 57 | issues one or more HTTP requests and receives replies for each |
| 58 | request. |
| @@ -82,11 +82,11 @@ | |
| 82 | processor housed in a datacenter, nor does the client need to be a desktop |
| 83 | or handheld device. For the purposes of this article "client" simply means |
| 84 | the repository that initiates the conversation and "server" is the repository |
| 85 | that responds. Nothing more. |
| 86 | |
| 87 | <h4 id="https">2.0.1 HTTPS Transport</h4> |
| 88 | |
| 89 | HTTPS differs from HTTP only in that the HTTPS protocol is |
| 90 | encrypted as it travels over the wire. The underlying protocol |
| 91 | is the same. This document describes only the underlying, unencrypted |
| 92 | messages that go client to server and back to client. |
| @@ -94,11 +94,11 @@ | |
| 94 | in this document. |
| 95 | |
| 96 | Fossil includes built-in |
| 97 | [./ssl-server.md|support for HTTPS encryption] in both client and server. |
| 98 | |
| 99 | <h4 id="ssh">2.0.2 SSH Transport</h4> |
| 100 | |
| 101 | When doing a sync using an "ssh:..." URL, the same HTTP transport protocol |
| 102 | is used. Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh] |
| 103 | to start an instance of the [/help?cmd=test-http|fossil test-http] command |
| 104 | running on the remote machine. It then sends HTTP requests and gets back HTTP |
| @@ -105,11 +105,11 @@ | |
| 105 | replies over the SSH connection, rather than sending and receiving over an |
| 106 | internet socket. To see the specific "ssh" command that the Fossil client |
| 107 | runs in order to set up a connection, add either of the the "--httptrace" or |
| 108 | "--sshtrace" options to the "fossil sync" command line. |
| 109 | |
| 110 | <h4 id="file">2.0.3 FILE Transport</h4> |
| 111 | |
| 112 | When doing a sync using a "file:..." URL, the same HTTP protocol is |
| 113 | still used. But instead of sending each HTTP request over a socket or |
| 114 | via SSH, the HTTP request is written into a temporary file. The client |
| 115 | then invokes the [/help?cmd=http|fossil http] command in a subprocess |
| @@ -117,11 +117,11 @@ | |
| 117 | the HTTP reply out of a temporary file on disk, and deletes the two |
| 118 | temporary files. To see the specific "fossil http" command that is run |
| 119 | in order to implement the "file:" transport, add the "--httptrace" |
| 120 | option to the "fossil sync" command. |
| 121 | |
| 122 | <h3 id="srv-id">2.1 Server Identification</h3> |
| 123 | |
| 124 | The server is identified by a URL argument that accompanies the |
| 125 | push, pull, or sync command on the client. (As a convenience to |
| 126 | users, the URL can be omitted on the client command and the same URL |
| 127 | from the most recent push, pull, or sync will be reused. This saves |
| @@ -137,11 +137,11 @@ | |
| 137 | Then the URL that is really used to do the synchronization will |
| 138 | be: |
| 139 | |
| 140 | <pre>https://fossil-scm.org/fossil/xfer</pre> |
| 141 | |
| 142 | <h3 id="req-format">2.2 HTTP Request Format</h3> |
| 143 | |
| 144 | The client always sends a POST request to the server. The |
| 145 | general format of the POST request is as follows: |
| 146 | |
| 147 | <pre> |
| @@ -175,17 +175,17 @@ | |
| 175 | <i><pre>content...</pre></i> |
| 176 | |
| 177 | The content type of the reply is always the same as the content type |
| 178 | of the request. |
| 179 | |
| 180 | <h2 id="content">3.0 Fossil Synchronization Content</h2> |
| 181 | |
| 182 | A synchronization request between a client and server consists of |
| 183 | one or more HTTP requests as described in the previous section. This |
| 184 | section details the "x-fossil" content type. |
| 185 | |
| 186 | <h3 id="lines">3.1 Line-oriented Format</h3> |
| 187 | |
| 188 | The x-fossil content type consists of zero or more "cards". Cards |
| 189 | are separated by the newline character ("\n"). Leading and trailing |
| 190 | whitespace on a card is ignored. Blank cards are ignored. |
| 191 | |
| @@ -193,11 +193,11 @@ | |
| 193 | The first token on each card is the operator. Subsequent tokens |
| 194 | are arguments. The set of operators understood by servers is slightly |
| 195 | different from the operators understood by clients, though the two |
| 196 | are very similar. |
| 197 | |
| 198 | <h3 id="login">3.2 Login Cards</h3> |
| 199 | |
| 200 | Every message from client to server begins with one or more login |
| 201 | cards. Each login card has the following format: |
| 202 | |
| 203 | <pre><b>login</b> <i>userid nonce signature</i></pre> |
| @@ -217,21 +217,20 @@ | |
| 217 | |
| 218 | Privileges are cumulative. There can be multiple successful |
| 219 | login cards. The session privilege is the union of all |
| 220 | privileges from all login cards. |
| 221 | |
| 222 | <h3 id="file">3.3 File Cards</h3> |
| 223 | |
| 224 | Artifacts are transferred using either "file" cards, or "cfile" |
| 225 | or "uvfile" cards. |
| 226 | The name "file" card comes from the fact that most artifacts correspond to |
| 227 | files that are under version control. |
| 228 | The "cfile" name is an abbreviation for "compressed file". |
| 229 | The "uvfile" name is an abbreviation for "unversioned file". |
| 230 | |
| 231 | <h4 id="ordinary-fc">3.3.1 Ordinary File Cards</h4> |
| 232 | |
| 233 | For sync protocols, artifacts are transferred using "file" |
| 234 | cards. File cards come in two different formats depending |
| 235 | on whether the artifact is sent directly or as a |
| 236 | [./delta_format.wiki|delta] from some |
| @@ -263,11 +262,11 @@ | |
| 262 | File cards are sent in both directions: client to server and |
| 263 | server to client. A delta might be sent before the source of |
| 264 | the delta, so both client and server should remember deltas |
| 265 | and be able to apply them when their source arrives. |
| 266 | |
| 267 | <h4 id="compressed-fc">3.3.2 Compressed File Cards</h4> |
| 268 | |
| 269 | A client that sends a clone protocol version "3" or greater will |
| 270 | receive artifacts as "cfile" cards while cloning. This card was |
| 271 | introduced to improve the speed of the transfer of content by sending the |
| 272 | compressed artifact directly from the server database to the client. |
| @@ -296,25 +295,24 @@ | |
| 295 | delta artifact. |
| 296 | |
| 297 | Unlike file cards, cfile cards are only sent in one direction during a |
| 298 | clone from server to client for clone protocol version "3" or greater. |
| 299 | |
| 300 | <h4 id="private">3.3.3 Private artifacts</h4> |
| 301 | |
| 302 | "Private" content consist of artifacts that are not normally synced. |
| 303 | However, private content will be synced when the |
| 304 | the [/help?cmd=sync|fossil sync] command includes the "--private" option. |
| 305 | |
| 306 | Private content is marked by a "private" card: |
| 307 | |
| 308 | <pre><b>private</b></pre> |
| 309 | |
| 310 | The private card has no arguments and must directly precede a |
| 311 | file card that contains the private content. |
| 312 | |
| 313 | <h4 id="uv-fc">3.3.4 Unversioned File Cards</h4> |
| 314 | |
| 315 | Unversioned content is sent in both directions (client to server and |
| 316 | server to client) using "uvfile" cards in the following format: |
| 317 | |
| 318 | <pre><b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i></pre> |
| @@ -342,11 +340,11 @@ | |
| 340 | |
| 341 | Servers send uvfile cards in response to uvgimme cards received from |
| 342 | the client. Clients send uvfile cards when they determine that the server |
| 343 | needs the content based on uvigot cards previously received from the server. |
| 344 | |
| 345 | <h3 id="push" name="pull">3.4 Push and Pull Cards</h3> |
| 346 | |
| 347 | Among the first cards in a client-to-server message are |
| 348 | the push and pull cards. The push card tells the server that |
| 349 | the client is pushing content. The pull card tells the server |
| 350 | that the client wants to pull content. In the event of a sync, |
| @@ -367,11 +365,11 @@ | |
| 365 | during a clone. This is how the client determines what project |
| 366 | code to put in the new repository it is constructing. |
| 367 | |
| 368 | The <i>servercode</i> argument is currently unused. |
| 369 | |
| 370 | <h3 id="clones">3.5 Clone Cards</h3> |
| 371 | |
| 372 | A clone card works like a pull card in that it is sent from |
| 373 | client to server in order to tell the server that the client |
| 374 | wants to pull content. The clone card comes in two formats. Older |
| 375 | clients use the no-argument format and newer clients use the |
| @@ -424,11 +422,11 @@ | |
| 422 | artifacts) but is too slow and unwieldy for larger repositories. |
| 423 | The version 2 protocol is an effort to improve performance. Further |
| 424 | performance improvements with higher-numbered clone protocols are |
| 425 | possible in future versions of Fossil. |
| 426 | |
| 427 | <h3 id="igot">3.6 Igot Cards</h3> |
| 428 | |
| 429 | An igot card can be sent from either client to server or from |
| 430 | server to client in order to indicate that the sender holds a copy |
| 431 | of a particular artifact. The format is: |
| 432 | |
| @@ -479,11 +477,11 @@ | |
| 477 | If a client-to-server transmission is needed, the client schedules that |
| 478 | transfer to occur on a subsequent HTTP request. If a server-to-client |
| 479 | transfer is needed, then the client sends a "uvgimme" card back to the |
| 480 | server to request the file content. |
| 481 | |
| 482 | <h3 id="gimme">3.7 Gimme Cards</h3> |
| 483 | |
| 484 | A gimme card is sent from either client to server or from server |
| 485 | to client. The gimme card asks the receiver to send a particular |
| 486 | artifact back to the sender. The format of a gimme card is this: |
| 487 | |
| @@ -514,11 +512,11 @@ | |
| 512 | The <i>name</i> is the name of the unversioned file found on the |
| 513 | server that the client would like to have. When a server sees a |
| 514 | uvgimme card, it normally responses with a uvfile card, though it might |
| 515 | also send another uvigot card if the HTTP reply is already oversized. |
| 516 | |
| 517 | <h3 id="cookie">3.8 Cookie Cards</h3> |
| 518 | |
| 519 | A cookie card can be used by a server to record a small amount |
| 520 | of state information on a client. The server sends a cookie to the |
| 521 | client. The client sends the same cookie back to the server on |
| 522 | its next request. The cookie card has a single argument which |
| @@ -534,11 +532,11 @@ | |
| 532 | cookie and the server must structure the cookie payload in such |
| 533 | a way that it can tell if the cookie it sees is its own cookie or |
| 534 | a cookie from another server. (Typically the server will embed |
| 535 | its servercode as part of the cookie.) |
| 536 | |
| 537 | <h3 id="reqconfig">3.9 Request-Configuration Cards</h3> |
| 538 | |
| 539 | A request-configuration or "reqconfig" card is sent from client to |
| 540 | server in order to request that the server send back "configuration" |
| 541 | data. "Configuration" data is information about users or website |
| 542 | appearance or other administrative details which are not part of the |
| @@ -643,11 +641,11 @@ | |
| 641 | intended to be evaluated by the client. |
| 642 | |
| 643 | The @user and @concealed configuration items contain sensitive information |
| 644 | and are ignored for clients without sufficient privilege. |
| 645 | |
| 646 | <h3 id="config">3.10 Configuration Cards</h3> |
| 647 | |
| 648 | A "config" card is used to send configuration information from client |
| 649 | to server (in response to a "fossil configuration push" command) or |
| 650 | from server to client (in response to a "fossil configuration pull" or |
| 651 | "fossil clone" command). The format is as follows: |
| @@ -661,11 +659,11 @@ | |
| 659 | it had sent a corresponding reqconfig card in its request. |
| 660 | |
| 661 | The content of the configuration item is used to overwrite the |
| 662 | corresponding configuration data in the receiver. |
| 663 | |
| 664 | <h3 id="pragma">3.11 Pragma Cards</h3> |
| 665 | |
| 666 | The client may try to influence the behavior of the server by |
| 667 | issuing a pragma card: |
| 668 | |
| 669 | <pre> |
| @@ -753,16 +751,16 @@ | |
| 751 | The ci-unlock pragma helps to avoid false-positive lock warnings |
| 752 | that might arise if a check-in is aborted and then restarted |
| 753 | on a branch. |
| 754 | </ol> |
| 755 | |
| 756 | <h3 id="comment">3.12 Comment Cards</h3> |
| 757 | |
| 758 | Any card that begins with "#" (ASCII 0x23) is a comment card and |
| 759 | is silently ignored. |
| 760 | |
| 761 | <h3 id="error">3.13 Message and Error Cards</h3> |
| 762 | |
| 763 | If the server discovers anything wrong with a request, it generates |
| 764 | an error card in its reply. When the client sees the error card, |
| 765 | it displays an error message to the user and aborts the sync |
| 766 | operation. An error card looks like this: |
| @@ -787,16 +785,16 @@ | |
| 785 | <b>message</b> <i>message-text</i> |
| 786 | </pre> |
| 787 | |
| 788 | The message-text uses the same format as an error message. |
| 789 | |
| 790 | <h3 id="unknown">3.14 Unknown Cards</h3> |
| 791 | |
| 792 | If either the client or the server sees a card that is not |
| 793 | described above, then it generates an error and aborts. |
| 794 | |
| 795 | <h2 id="phantoms" name="clusters">4.0 Phantoms And Clusters</h2> |
| 796 | |
| 797 | When a repository knows that an artifact exists and knows the ID of |
| 798 | that artifact, but it does not know the artifact content, then it stores that |
| 799 | artifact as a "phantom". A repository will typically create a phantom when |
| 800 | it receives an igot card for an artifact that it does not hold or when it |
| @@ -827,11 +825,11 @@ | |
| 825 | exactly is not a cluster. There must be no extra whitespace in |
| 826 | the artifact. There must be one or more M cards. There must be a |
| 827 | single Z card with a correct MD5 checksum. And all cards must |
| 828 | be in strict lexicographical order. |
| 829 | |
| 830 | <h3 id="unclustered">4.1 The Unclustered Table</h3> |
| 831 | |
| 832 | Every repository maintains a table named "<b>unclustered</b>" |
| 833 | which records the identity of every artifact and phantom it holds that is not |
| 834 | mentioned in a cluster. The entries in the unclustered table can |
| 835 | be thought of as leaves on a tree of artifacts. Some of the unclustered |
| @@ -838,13 +836,13 @@ | |
| 836 | artifacts will be other clusters. Those clusters may contain other clusters, |
| 837 | which might contain still more clusters, and so forth. Beginning |
| 838 | with the artifacts in the unclustered table, one can follow the chain |
| 839 | of clusters to find every artifact in the repository. |
| 840 | |
| 841 | <h2 id="strategies">5.0 Synchronization Strategies</h2> |
| 842 | |
| 843 | <h3 id="pull-strategy">5.1 Pull</h3> |
| 844 | |
| 845 | A typical pull operation proceeds as shown below. Details |
| 846 | of the actual implementation may very slightly but the gist of |
| 847 | a pull is captured in the following steps: |
| 848 | |
| @@ -892,11 +890,11 @@ | |
| 890 | protocol will continue to work even if there are multiple servers |
| 891 | or if servers and clients sometimes change roles. The only negative |
| 892 | effects of these unusual arrangements is that more than the minimum |
| 893 | number of clusters might be generated. |
| 894 | |
| 895 | <h3 id="push-stragegy">5.2 Push</h3> |
| 896 | |
| 897 | A typical push operation proceeds roughly as shown below. As |
| 898 | with a pull, the actual implementation may vary slightly. |
| 899 | |
| 900 | <ol> |
| @@ -926,19 +924,19 @@ | |
| 924 | server knows all artifacts that exist on the client. Also, as with |
| 925 | pull, the client attempts to keep the size of the request from |
| 926 | growing too large by suppressing file cards once the |
| 927 | size of the request reaches 1MB. |
| 928 | |
| 929 | <h3 id="sync-strategy">5.3 Sync</h3> |
| 930 | |
| 931 | A sync is just a pull and a push that happen at the same time. |
| 932 | The first three steps of a pull are combined with the first five steps |
| 933 | of a push. Steps (4) through (7) of a pull are combined with steps |
| 934 | (5) through (8) of a push. And steps (8) through (10) of a pull |
| 935 | are combined with step (9) of a push. |
| 936 | |
| 937 | <h3 id="uv-strategy">5.4 Unversioned File Sync</h3> |
| 938 | |
| 939 | "Unversioned files" are files held in the repository |
| 940 | where only the most recent version of the file is kept rather than |
| 941 | the entire change history. Unversioned files are intended to be |
| 942 | used to store ephemeral content, such as compiled binaries of the |
| @@ -976,11 +974,11 @@ | |
| 974 | |
| 975 | The last two steps might be repeated multiple |
| 976 | times if there is more unversioned content to be transferred than will |
| 977 | fit comfortably in a single HTTP request. |
| 978 | |
| 979 | <h2 id="summary">6.0 Summary</h2> |
| 980 | |
| 981 | Here are the key points of the synchronization protocol: |
| 982 | |
| 983 | <ol> |
| 984 | <li>The client sends one or more PUSH HTTP requests to the server. |
| @@ -1020,12 +1018,11 @@ | |
| 1018 | cluster and send igot messages for those artifacts. |
| 1019 | <li>Repositories keep track of all the phantoms they hold and send |
| 1020 | gimme messages for those artifacts. |
| 1021 | </ol> |
| 1022 | |
| 1023 | <h2 id="troubleshooting">7.0 Troubleshooting And Debugging Hints</h2> |
| 1024 | |
| 1025 | If you run the [/help?cmd=sync|fossil sync] command |
| 1026 | (or [/help?cmd=pull|pull] or [/help?cmd=push|push] or |
| 1027 | [/help?cmd=clone|clone]) with the --httptrace option, Fossil |
| 1028 | will keep a copy of each HTTP request and reply in files |
| 1029 |