|
1
|
Backoffice |
|
2
|
========== |
|
3
|
|
|
4
|
This is technical documentation about the internal workings of Fossil. |
|
5
|
Ordinary Fossil users do not need to know about anything covered by this |
|
6
|
document. The information here is intended for people who want to enhance |
|
7
|
or extend the Fossil code, or who just want a deeper understanding of |
|
8
|
the internal workings of Fossil. |
|
9
|
|
|
10
|
What Is The Backoffice |
|
11
|
---------------------- |
|
12
|
|
|
13
|
The backoffice is a mechanism used by a |
|
14
|
[Fossil server](./server/) to do low-priority |
|
15
|
background work that is not directly related to the user interface. Here |
|
16
|
are some examples of the kinds of work that backoffice performs: |
|
17
|
|
|
18
|
1. Sending email alerts and notifications |
|
19
|
2. Sending out daily digests of email notifications |
|
20
|
3. Other background email handling chores |
|
21
|
4. Automatic syncing of peer repositories |
|
22
|
5. Repository maintenance and optimization |
|
23
|
|
|
24
|
(As of 2018-08-07, only items 1 and 2 have actually been implemented.) |
|
25
|
The idea is that the backoffice handles behind-the-scenes work that does |
|
26
|
not have tight latency requirements. |
|
27
|
|
|
28
|
When Backoffice Runs |
|
29
|
-------------------- |
|
30
|
|
|
31
|
A backoffice process is usually launched automatically by a webpage |
|
32
|
request. After each webpage is generated, Fossil checks to see if any |
|
33
|
backoffice work needs to be done. If there is work to do, and no other |
|
34
|
process is already assigned to do the work, then a new backoffice process |
|
35
|
is started to do the work. |
|
36
|
|
|
37
|
This happens for every webpage, regardless of how that webpage is launched, |
|
38
|
and regardless of the purpose of the webpage. This also happens on the |
|
39
|
server for "[fossil sync](/help/sync)" and |
|
40
|
"[fossil clone](/help/clone)" commands which are implemented as |
|
41
|
web requests - albeit requests that the human user never sees. |
|
42
|
Web requests can arrive at the Fossil server via direct TCP/IP (for example |
|
43
|
when Fossil is started using commands like "[fossil server](/help/server)") |
|
44
|
or via [CGI](./server/any/cgi.md) or |
|
45
|
[SCGI](./server/any/scgi.md) or via SSH. |
|
46
|
A backoffice process might be started regardless of the origin of the |
|
47
|
request. |
|
48
|
|
|
49
|
The backoffice is not a daemon. Each backoffice process runs for a short |
|
50
|
while and then exits. This helps keep Fossil easy to manage, since there |
|
51
|
are no daemons to start and stop. To upgrade Fossil to a new version, |
|
52
|
you simply replace the older "fossil" executable with the newer one, and |
|
53
|
the backoffice processes will (within a minute or so) start using the new |
|
54
|
one. (Upgrading the executable on Windows is more complicated, since on |
|
55
|
Windows it is not possible to replace an executable file that is in active |
|
56
|
use. But Windows users probably already know this.) |
|
57
|
|
|
58
|
The backoffice is serialized and rate limited. No more than a single |
|
59
|
backoffice process will be running at once, and backoffice runs will not |
|
60
|
occur more frequently than once every 60 seconds. (The 60-second spacing |
|
61
|
is controlled by the BKOFCE_LEASE_TIME macro in the |
|
62
|
[backoffice.c](/file/src/backoffice.c) source file.) |
|
63
|
|
|
64
|
If a Fossil server is idle, then no backoffice processes will be running. |
|
65
|
That means there are no extra processes sitting around taking up memory |
|
66
|
and process table slots for seldom accessed repositories. |
|
67
|
The backoffice is an on-demand system. |
|
68
|
A busy repository will usually have a backoffice |
|
69
|
running at all times. But an infrequently accessed repository will only have |
|
70
|
backoffice processes running for a minute or two following the most recent |
|
71
|
access. |
|
72
|
|
|
73
|
Manually Running The Backoffice |
|
74
|
------------------------------- |
|
75
|
|
|
76
|
The automatic backoffice runs are sufficient for most installations. |
|
77
|
However, the daily digest of email notifications is handled by the |
|
78
|
backoffice. If a Fossil server can sometimes go more than a day without |
|
79
|
being accessed, then the automatic backoffice will never run, and the |
|
80
|
daily digest might not go out until somebody does visit a webpage. |
|
81
|
If this is a problem, an administrator can set up a cron job to |
|
82
|
periodically run: |
|
83
|
|
|
84
|
fossil backoffice _REPOSITORY_ |
|
85
|
|
|
86
|
That command will cause backoffice processing to occur immediately. |
|
87
|
Note that this is almost never necessary for an internet-facing |
|
88
|
Fossil repository, since most repositories will get multiple accesses |
|
89
|
per day from random robots, which will be sufficient to kick off the |
|
90
|
daily digest emails. And even for a private server, if there is very |
|
91
|
little traffic, then the daily digests are probably a no-op anyhow |
|
92
|
and won't be missed. |
|
93
|
|
|
94
|
Automatic Backoffice Does Not Work On Some Systems |
|
95
|
-------------------------------------------------- |
|
96
|
|
|
97
|
We have observed that the automatic backoffice does not work on |
|
98
|
some systems - OpenBSD in particular. We still do not understand why |
|
99
|
this is. (If you have insights, please share them on the |
|
100
|
[Fossil Forum](https://fossil-scm.org/forum) so that we can perhaps |
|
101
|
fix the problem.) For now, the backoffice must be run manually |
|
102
|
on OpenBSD systems. |
|
103
|
|
|
104
|
To set up a fully-manual backoffice, first disable the automatic backoffice |
|
105
|
using the "[backoffice-disable](/help/backoffice-disable)" setting. |
|
106
|
|
|
107
|
fossil setting backoffice-disable on |
|
108
|
|
|
109
|
Then arrange to invoke the backoffice separately using a command |
|
110
|
like this: |
|
111
|
|
|
112
|
fossil backoffice --poll 30 _REPOSITORY-LIST_ |
|
113
|
|
|
114
|
Multiple repositories can be named. This one command will handle |
|
115
|
launching the backoffice for all of them. There are additional useful |
|
116
|
command-line options. See the "[fossil backoffice](/help/backoffice)" |
|
117
|
documentation for details. |
|
118
|
|
|
119
|
The backoffice processes that are run manually using the "fossil backoffice" |
|
120
|
command do not normally use a lease. That means that if you run the |
|
121
|
"fossil backoffice" command with --poll and you forget to disable |
|
122
|
automatic backoffice by setting the "backoffice-disable" flag, then |
|
123
|
you might have one backoffice running due to a command and another due |
|
124
|
to a webpage access, both at the same time. This is harmless. The |
|
125
|
only downside is that it uses extra CPU time. |
|
126
|
|
|
127
|
How Backoffice Is Implemented |
|
128
|
----------------------------- |
|
129
|
|
|
130
|
The backoffice is implemented by the |
|
131
|
"[backoffice.c](/file/src/backoffice.c)" source file. |
|
132
|
|
|
133
|
Serialization and rate limiting is handled by a single entry in the |
|
134
|
repository database CONFIG table named "backoffice". This entry is |
|
135
|
called "the lease". The value of the lease |
|
136
|
is a text string representing four integers, which |
|
137
|
are respectively: |
|
138
|
|
|
139
|
1. The process id of the "current" backoffice process |
|
140
|
2. The lease expiration time of the current backoffice process |
|
141
|
3. The process id of the "next" backoffice process |
|
142
|
4. The lease expiration time for the next backoffice process |
|
143
|
|
|
144
|
Times are expressed in seconds since 1970. A process id of zero means |
|
145
|
"no process". Sometimes the process id will be non-zero even if there |
|
146
|
is no corresponding process. Fossil knows how to figure out whether or |
|
147
|
not a process still exists. |
|
148
|
|
|
149
|
You can print out a decoded copy of the current backoffice lease using |
|
150
|
this command: |
|
151
|
|
|
152
|
fossil test-backoffice-lease -R _REPOSITORY_ |
|
153
|
|
|
154
|
If a system has been idle for a long time, then there will be no |
|
155
|
backoffice processes. (Either the process id entries in the lease |
|
156
|
will be zero, or there will exist no process associated with the |
|
157
|
process id.) When a new web request comes in, the system |
|
158
|
sees that no backoffice process is active and so it kicks off a separate |
|
159
|
process to run backoffice. |
|
160
|
|
|
161
|
The new backoffice process becomes the "current" process. It sets a |
|
162
|
lease expiration time for itself to be 60 seconds in the future. |
|
163
|
Then it does the backoffice processing and exits. Note that usually |
|
164
|
the backoffice process will exit long before its lease expires. That |
|
165
|
is ok. The lease is there to limit the rate at which backoffice processes |
|
166
|
run. |
|
167
|
|
|
168
|
If a new backoffice process starts up and sees that the "current" lease has |
|
169
|
yet to expire, the new process makes itself the "next" backoffice process |
|
170
|
and sets its expiration time to be 60 seconds past the expiration time of |
|
171
|
the "current" backoffice process. The "next" process then puts itself to |
|
172
|
sleep until the "current" lease expires. After the "current" |
|
173
|
lease expires and the "current" process has itself exited, then |
|
174
|
the "next" process promotes itself to the new "current" process. It |
|
175
|
sets the current lease expiration to be 60 seconds in the future, runs |
|
176
|
whatever backoffice work is needed, then exits. |
|
177
|
|
|
178
|
If a new backoffice process starts up and finds that there is already |
|
179
|
a "current" lease and a "next" process, it exits without doing anything. |
|
180
|
This should happen only rarely, since the lease information is checked |
|
181
|
prior to spawning the backoffice process, so a conflict will only happen |
|
182
|
in a race. |
|
183
|
|
|
184
|
Because the "backoffice" entry of the CONFIG table is in the repository |
|
185
|
database, access to the lease is serialized. The lease prevents more |
|
186
|
than one backoffice process from running at a time. It prevents |
|
187
|
backoffice processes from running more frequently than once every 60 seconds. |
|
188
|
And, it guarantees (assuming processes are not killed out-of-band) that |
|
189
|
every web request will be followed within 60 seconds by a backoffice |
|
190
|
run. |
|
191
|
|
|
192
|
Debugging The Backoffice |
|
193
|
------------------------ |
|
194
|
|
|
195
|
The backoffice should "just work". It should not require administrator |
|
196
|
attention. However, if you suspect that something is not working right, |
|
197
|
there are some debugging aids. |
|
198
|
|
|
199
|
We have already mentioned the command that shows the backoffice lease |
|
200
|
for a repository: |
|
201
|
|
|
202
|
fossil test-backoffice-lease -R _REPOSITORY_ |
|
203
|
|
|
204
|
Running that command every few seconds should show what is going on with |
|
205
|
backoffice processing in a particular repository. |
|
206
|
|
|
207
|
There are also settings that control backoffice behavior. The |
|
208
|
"backoffice-nodelay" setting prevents the "next" process from taking a |
|
209
|
lease and sleeping. If "backoffice-nodelay" is set, that causes all |
|
210
|
backoffice processes to exit either immediately or after doing whatever |
|
211
|
backoffice works needs to be done. If something is going wrong and |
|
212
|
backoffice leases are causing delays in webpage processing, then setting |
|
213
|
"backoffice-nodelay" to true can work around the problem until the bug |
|
214
|
can be fixed. The "backoffice-logfile" setting is the name of a log |
|
215
|
file onto which is appended a short message every time a backoffice |
|
216
|
process actually starts to do the backoffice work. This log file can |
|
217
|
be used to verify that backoffice really is running, if there is any |
|
218
|
doubt. The "backoffice-disable" setting prevents automatic backoffice |
|
219
|
processing, if true. Use this to completely disable backoffice processing |
|
220
|
that occurs automatically after each HTTP request. The "backoffice-disable" |
|
221
|
setting does not affect the operation of the manual |
|
222
|
"fossil backoffice" command. |
|
223
|
Most installations should leave "backoffice-nodelay" and "backoffice-disable" |
|
224
|
set to their default values of off and |
|
225
|
leave "backoffice-logfile" unset or set to an empty string. |
|
226
|
|