|
1
|
<title>CGI Server Extensions</title> |
|
2
|
|
|
3
|
<h2>1.0 Introduction</h2> |
|
4
|
|
|
5
|
If you have a [./server/|Fossil server] for your project, |
|
6
|
you can add [./aboutcgi.wiki|CGI] |
|
7
|
extensions to that server. These extensions work like |
|
8
|
any other CGI program, except that they also have access to the Fossil |
|
9
|
login information and can (optionally) leverage the "[./customskin.md|skins]" |
|
10
|
of Fossil so that they appear to be more tightly integrated into the project. |
|
11
|
|
|
12
|
An example of where this is useful is the |
|
13
|
[https://sqlite.org/src/ext/checklist|checklist application] on |
|
14
|
the [https://sqlite.org/|SQLite] project. The checklist |
|
15
|
helps the SQLite developers track which release tests have passed, |
|
16
|
or failed, or are still to be done. The checklist program began as a |
|
17
|
stand-alone CGI which kept its own private user database and implemented |
|
18
|
its own permissions and login system and provided its own CSS. By |
|
19
|
converting checklist into a Fossil extension, the same login that works |
|
20
|
for the [https://sqlite.org/src|main SQLite source repository] also works |
|
21
|
for the checklist. Permission to change elements of the checklist |
|
22
|
is tied on permission to check-in to the main source repository. And |
|
23
|
the standard Fossil header menu and footer appear on each page of |
|
24
|
the checklist. |
|
25
|
|
|
26
|
<h2>2.0 How It Works</h2> |
|
27
|
|
|
28
|
CGI Extensions are disabled by default. |
|
29
|
An administrator activates the CGI extension mechanism by specifying |
|
30
|
an "Extension Root Directory" or "extroot" as part of the |
|
31
|
[./server/index.html|server setup]. |
|
32
|
If the Fossil server is itself run as |
|
33
|
[./server/any/cgi.md|CGI], then add a line to the |
|
34
|
[./cgi.wiki#extroot|CGI script file] that says: |
|
35
|
|
|
36
|
<pre> |
|
37
|
extroot: <i>DIRECTORY</i> |
|
38
|
</pre> |
|
39
|
|
|
40
|
Or, if the Fossil server is being run using the |
|
41
|
"[./server/any/none.md|fossil server]" or |
|
42
|
"[./server/any/none.md|fossil ui]" or |
|
43
|
"[./server/any/inetd.md|fossil http]" commands, then add an extra |
|
44
|
"--extroot <i>DIRECTORY</i>" option to that command. |
|
45
|
|
|
46
|
The <i>DIRECTORY</i> is the DOCUMENT_ROOT for the CGI. |
|
47
|
Files in the DOCUMENT_ROOT are accessed via URLs like this: |
|
48
|
|
|
49
|
<pre> |
|
50
|
https://example-project.org/ext/<i>FILENAME</i> |
|
51
|
</pre> |
|
52
|
|
|
53
|
In other words, access files in DOCUMENT_ROOT by appending the filename |
|
54
|
relative to DOCUMENT_ROOT to the [/help/www/ext|/ext] |
|
55
|
page of the Fossil server. |
|
56
|
|
|
57
|
* Files that are readable but not executable are returned as static |
|
58
|
content. |
|
59
|
|
|
60
|
* Files that are executable are run as CGI. |
|
61
|
|
|
62
|
<h3>2.1 Example #1</h3> |
|
63
|
|
|
64
|
The source code repository for SQLite is a Fossil server that is run |
|
65
|
as CGI. The URL for the source code repository is [https://sqlite.org/src]. |
|
66
|
The CGI script looks like this: |
|
67
|
|
|
68
|
<verbatim> |
|
69
|
#!/usr/bin/fossil |
|
70
|
repository: /fossil/sqlite.fossil |
|
71
|
errorlog: /logs/errors.txt |
|
72
|
extroot: /sqlite-src-ext |
|
73
|
</verbatim> |
|
74
|
|
|
75
|
The "extroot: /sqlite-src-ext" line tells Fossil that it should look for |
|
76
|
extension CGIs in the /sqlite-src-ext directory. (All of this is happening |
|
77
|
inside of a chroot jail, so putting the document root in a top-level |
|
78
|
directory is a reasonable thing to do.) |
|
79
|
|
|
80
|
When a URL like "https://sqlite.org/src/ext/checklist" is received by the |
|
81
|
main webserver, it figures out that the /src part refers to the main |
|
82
|
Fossil CGI script and so it runs that script. Fossil gets the remainder |
|
83
|
of the URL to work with: "/ext/checklist". Fossil extracts the "/ext" |
|
84
|
prefix and uses that to determine that this a CGI extension request. |
|
85
|
Then it takes the leftover "/checklist" part and appends it to the |
|
86
|
"extroot" to get the filename "/sqlite-src-ext/checklist". Fossil finds |
|
87
|
that file to be executable, so it runs it as CGI and returns the result. |
|
88
|
|
|
89
|
The /sqlite-src-ext/checklist file is a |
|
90
|
[https://wapp.tcl.tk|Wapp program]. The current source code to the |
|
91
|
this program can be seen at |
|
92
|
[https://www.sqlite.org/src/ext/checklist/3070700/self] and |
|
93
|
recent historical versions are available at |
|
94
|
[https://sqlite.org/docsrc/finfo/misc/checklist.tcl] with |
|
95
|
older legacy at [https://sqlite.org/checklistapp/timeline?n1=all] |
|
96
|
|
|
97
|
There is a cascade of CGIs happening here. The web server that receives |
|
98
|
the initial HTTP request runs Fossil as a CGI based on the |
|
99
|
"https://sqlite.org/src" portion of the URL. The Fossil instance then |
|
100
|
runs the checklist sub-CGI based on the "/ext/checklists" suffix. The |
|
101
|
output of the sub-CGI is read by Fossil and then relayed on to the |
|
102
|
main web server which in turn relays the result back to the original client. |
|
103
|
|
|
104
|
<h3>2.2 Example #2</h3> |
|
105
|
|
|
106
|
The [https://fossil-scm.org/home|Fossil self-hosting repository] is also |
|
107
|
a CGI that looks like this: |
|
108
|
|
|
109
|
<verbatim> |
|
110
|
#!/usr/bin/fossil |
|
111
|
repository: /fossil/fossil.fossil |
|
112
|
errorlog: /logs/errors.txt |
|
113
|
extroot: /fossil-extroot |
|
114
|
</verbatim> |
|
115
|
|
|
116
|
The extroot for this Fossil server is /fossil-extroot and in that directory |
|
117
|
is an executable file named "fileup1" - another [https://wapp.tcl.tk|Wapp] |
|
118
|
script. (The extension mechanism is not required to use Wapp. You can use |
|
119
|
any kind of program you like. But the creator of SQLite and Fossil is fond |
|
120
|
of [https://www.tcl.tk|Tcl/Tk] and so he tends to gravitate toward Tcl-based |
|
121
|
technologies like Wapp.) The fileup1 script is a demo program that lets |
|
122
|
the user upload a file using a form, and then displays that file in the reply. |
|
123
|
There is a link on the page that causes the fileup1 script to return a copy |
|
124
|
of its own source-code, so you can see how it works. |
|
125
|
|
|
126
|
<h3>2.3 Example #3</h3> |
|
127
|
|
|
128
|
For Fossil versions dated 2025-03-23 and later, the "--extpage FILENAME" |
|
129
|
option to the [/help/ui|fossil ui] command is a short cut that treats |
|
130
|
FILENAME as a CGI extension. When the ui command starts up a new web browser |
|
131
|
pages, it points that page to the FILENAME extension. So if FILENAME is |
|
132
|
a static content file (such as an HTML file or |
|
133
|
[/md_rules|Markdown] or [/wiki_rules|Wiki] document), then the |
|
134
|
rendered content of the file is displayed. Meanwhile, the user can be |
|
135
|
editing the source text for that document in a separate window, and |
|
136
|
periodically pressing "Reload" on the web browser to instantly view the |
|
137
|
rendered results. |
|
138
|
|
|
139
|
For example, the author of this documentation page is running |
|
140
|
"<tt>fossil ui --extpage www/serverext.wiki</tt>" while editing this |
|
141
|
very paragraph, and presses Reload from time to time to view his |
|
142
|
edits. |
|
143
|
|
|
144
|
The same idea applies when developing new CGI applications using a script |
|
145
|
language (for example using [https://wapp.tcl.tk|Wapp]). Run the |
|
146
|
command "<tt>fossil ui --extpage SCRIPT</tt>" where SCRIPT is the name |
|
147
|
of the application script, while editing that script in a separate |
|
148
|
window, then press Reload periodically on the web browser to test the |
|
149
|
script. |
|
150
|
|
|
151
|
<h2 id="cgi-inputs">3.0 CGI Inputs</h2> |
|
152
|
|
|
153
|
The /ext extension mechanism is an ordinary CGI interface. Parameters |
|
154
|
are passed to the CGI program using environment variables. The following |
|
155
|
standard CGI environment variables are supplied: |
|
156
|
|
|
157
|
* AUTH_TYPE |
|
158
|
* AUTH_CONTENT |
|
159
|
* CONTENT_LENGTH |
|
160
|
* CONTENT_TYPE |
|
161
|
* DOCUMENT_ROOT |
|
162
|
* GATEWAY_INTERFACE |
|
163
|
* HTTPS |
|
164
|
* HTTP_ACCEPT |
|
165
|
* HTTP_ACCEPT_ENCODING |
|
166
|
* HTTP_COOKIE |
|
167
|
* HTTP_HOST |
|
168
|
* HTTP_IF_MODIFIED_SINCE |
|
169
|
* HTTP_IF_NONE_MATCH |
|
170
|
* HTTP_REFERER |
|
171
|
* HTTP_USER_AGENT |
|
172
|
* PATH_INFO |
|
173
|
* QUERY_STRING |
|
174
|
* REMOTE_ADDR |
|
175
|
* REMOTE_USER |
|
176
|
* REQUEST_METHOD |
|
177
|
* REQUEST_SCHEME |
|
178
|
* REQUEST_URI |
|
179
|
* SCRIPT_DIRECTORY |
|
180
|
* SCRIPT_FILENAME |
|
181
|
* SCRIPT_NAME |
|
182
|
* SERVER_NAME |
|
183
|
* SERVER_PORT |
|
184
|
* SERVER_PROTOCOL |
|
185
|
* SERVER_SOFTWARE |
|
186
|
|
|
187
|
Do a web search for |
|
188
|
"[https://duckduckgo.com/?q=cgi+environment_variables|cgi environment variables]" |
|
189
|
to find more detail about what each of the above variables mean and how |
|
190
|
they are used. |
|
191
|
Live listings of the values of some or all of these environment variables |
|
192
|
can be found at links like these: |
|
193
|
|
|
194
|
* [https://fossil-scm.org/home/test-env] |
|
195
|
* [https://sqlite.org/src/ext/checklist/top/env] |
|
196
|
|
|
197
|
In addition to the standard CGI environment variables listed above, |
|
198
|
Fossil adds the following: |
|
199
|
|
|
200
|
* FOSSIL_CAPABILITIES |
|
201
|
* FOSSIL_NONCE |
|
202
|
* FOSSIL_REPOSITORY |
|
203
|
* FOSSIL_URI |
|
204
|
* FOSSIL_USER |
|
205
|
|
|
206
|
The FOSSIL_USER string is the name of the logged-in user. This variable |
|
207
|
is missing or is an empty string if the user is not logged in. The |
|
208
|
FOSSIL_CAPABILITIES string is a list of |
|
209
|
[./caps/ref.html|Fossil capabilities] that |
|
210
|
indicate what permissions the user has on the Fossil repository. |
|
211
|
The FOSSIL_REPOSITORY environment variable gives the filename of the |
|
212
|
Fossil repository that is running. The FOSSIL_URI variable shows the |
|
213
|
prefix of the REQUEST_URI that is the Fossil CGI script, or is an |
|
214
|
empty string if Fossil is being run by some method other than CGI. |
|
215
|
|
|
216
|
The [https://sqlite.org/src/ext/checklist|checklist application] uses the |
|
217
|
FOSSIL_USER environment variable to determine the name of the user and |
|
218
|
the FOSSIL_CAPABILITIES variable to determine if the user is allowed to |
|
219
|
mark off changes to the checklist. Only users with check-in permission |
|
220
|
to the Fossil repository are allowed to mark off checklist items. That |
|
221
|
means that the FOSSIL_CAPABILITIES string must contain the letter "i". |
|
222
|
Search for "FOSSIL_CAPABILITIES" in the |
|
223
|
[https://sqlite.org/src/ext/checklist/top/self|source listing] to see how |
|
224
|
this happens. |
|
225
|
|
|
226
|
If the CGI output is one of the forms for which Fossil inserts its own |
|
227
|
header and footer, then the inserted header will include a |
|
228
|
Content Security Policy (CSP) restriction on the use of javascript within |
|
229
|
the webpage. Any <script>...</script> elements within the |
|
230
|
CGI output must include a nonce or else they will be suppressed by the |
|
231
|
web browser. The FOSSIL_NONCE variable contains the value of that nonce. |
|
232
|
So, in other words, to get javascript to work, it must be enclosed in: |
|
233
|
|
|
234
|
<verbatim> |
|
235
|
<script nonce='$FOSSIL_NONCE'>...</script> |
|
236
|
</verbatim> |
|
237
|
|
|
238
|
Except, of course, the $FOSSIL_NONCE is replaced by the value of the |
|
239
|
FOSSIL_NONCE environment variable. |
|
240
|
|
|
241
|
<h3>3.1 Input Content</h3> |
|
242
|
|
|
243
|
If the HTTP request includes content (for example if this is a POST request) |
|
244
|
then the CONTENT_LENGTH value will be positive and the data for the content |
|
245
|
will be readable on standard input. |
|
246
|
|
|
247
|
|
|
248
|
<h2>4.0 CGI Outputs</h2> |
|
249
|
|
|
250
|
CGI programs construct a reply by writing to standard output. The first |
|
251
|
few lines of output are parameters intended for the web server that invoked |
|
252
|
the CGI. These are followed by a blank line and then the content. |
|
253
|
|
|
254
|
Typical parameter output looks like this: |
|
255
|
|
|
256
|
<verbatim> |
|
257
|
Status: 200 OK |
|
258
|
Content-Type: text/html |
|
259
|
</verbatim> |
|
260
|
|
|
261
|
CGI programs can return any content type they want - they are not restricted |
|
262
|
to text replies. It is OK for a CGI program to return (for example) |
|
263
|
image/png. |
|
264
|
|
|
265
|
The fields of the CGI response header can be any valid HTTP header fields. |
|
266
|
Those that Fossil does not understand are simply relayed back to up the |
|
267
|
line to the requester. |
|
268
|
|
|
269
|
Fossil takes special action with some content types. If the Content-Type |
|
270
|
is "text/x-fossil-wiki" or "text/x-markdown" then Fossil |
|
271
|
converts the content from [/wiki_rules|Fossil-Wiki] or |
|
272
|
[/md_rules|Markdown] into HTML, adding its |
|
273
|
own header and footer text according to the repository skin. Content |
|
274
|
of type "text/html" is normally passed straight through |
|
275
|
unchanged. However, if the text/html content is of the form: |
|
276
|
|
|
277
|
<verbatim> |
|
278
|
<div class='fossil-doc' data-title='DOCUMENT TITLE'> |
|
279
|
... HTML content there ... |
|
280
|
</div> |
|
281
|
</verbatim> |
|
282
|
|
|
283
|
In other words, if the outer-most markup of the HTML is a <div> |
|
284
|
element with a single class of "fossil-doc", |
|
285
|
then Fossil will adds its own header and footer to the HTML. The |
|
286
|
page title contained in the added header will be extracted from the |
|
287
|
"data-title" attribute. |
|
288
|
|
|
289
|
Except for the three cases noted above, Fossil makes no changes or |
|
290
|
additions to the CGI-generated content. Fossil just passes the verbatim |
|
291
|
content back up the stack towards the requester. |
|
292
|
|
|
293
|
<h3>4.1 <tt>GATEWAY_INTERFACE</tt> and Recursive Calls to fossil</h3> |
|
294
|
|
|
295
|
Like many CGI-aware applications, if fossil sees the environment |
|
296
|
variable <tt>GATEWAY_INTERFACE</tt> when it starts up, it assumes it |
|
297
|
is running in a CGI environment and behaves differently than when it |
|
298
|
is run in a non-CGI interactive session. If you intend to run fossil |
|
299
|
itself from within an extension CGI script, e.g. to run a query |
|
300
|
against the repository or simply fetch the fossil binary version, make |
|
301
|
sure to <em>unset</em> the <tt>GATEWAY_INTERFACE</tt> environment |
|
302
|
variable before doing so, otherwise the invocation will behave as if |
|
303
|
it's being run in CGI mode. |
|
304
|
|
|
305
|
<h2>5.0 Filename Restrictions</h2> |
|
306
|
|
|
307
|
For security reasons, Fossil places restrictions on the names of files |
|
308
|
in the extroot directory that can participate in the extension CGI |
|
309
|
mechanism: |
|
310
|
|
|
311
|
1. Filenames must consist of only ASCII alphanumeric characters, |
|
312
|
".", "_", and "-", and of course "/" as the file separator. |
|
313
|
Files with names that includes spaces or |
|
314
|
other punctuation or special characters are ignored. |
|
315
|
|
|
316
|
2. No element of the pathname can begin with "." or "-". Files or |
|
317
|
directories whose names begin with "." or "-" are ignored. |
|
318
|
|
|
319
|
If a CGI program requires separate data files, it is safe to put those |
|
320
|
files in the same directory as the CGI program itself as long as the names |
|
321
|
of the data files contain special characters that cause them to be ignored |
|
322
|
by Fossil. |
|
323
|
|
|
324
|
<h2>6.0 Access Permissions</h2> |
|
325
|
|
|
326
|
CGI extension files and programs are accessible to everyone. |
|
327
|
|
|
328
|
When CGI extensions have been enabled (using either "extroot:" in the |
|
329
|
CGI file or the --extroot option for other server methods) all files |
|
330
|
in the extension root directory hierarchy, except special filenames |
|
331
|
identified previously, are accessible to all users. Users do not |
|
332
|
have to have "Read" privilege, or any other privilege, in order to |
|
333
|
access the extensions. |
|
334
|
|
|
335
|
This is by design. The CGI extension mechanism is intended to operate |
|
336
|
in the same way as a traditional web-server. |
|
337
|
|
|
338
|
CGI programs that want to restrict access |
|
339
|
can examine the FOSSIL_CAPABILITIES and/or FOSSIL_USER environment variables. |
|
340
|
In other words, access control is the responsibility of the individual |
|
341
|
extension programs. |
|
342
|
|
|
343
|
<h3>6.1 Restricting Robot Access To Extensions</h3> |
|
344
|
|
|
345
|
If the "ext" tag is found in the [/help/robot-restrict|robot-restrict setting] |
|
346
|
then clients are tested to see if they are robots before granting |
|
347
|
access to any extension. If the "ext" tag is omitted but a tag |
|
348
|
of the form "ext/PATH" is found on the robot-restrict setting, then |
|
349
|
robots are restricted from the particular extension at PATH. |
|
350
|
|
|
351
|
<h2>7.0 Trouble-Shooting Hints</h2> |
|
352
|
|
|
353
|
Remember that the /ext will return any file in the extroot directory |
|
354
|
hierarchy as static content if the file is readable but not executable. |
|
355
|
When initially setting up the /ext mechanism, it is sometimes helpful |
|
356
|
to verify that you are able to receive static content prior to starting |
|
357
|
work on your CGIs. Also remember that CGIs must be |
|
358
|
executable files. |
|
359
|
|
|
360
|
Fossil likes to run inside a chroot jail, and will automatically put |
|
361
|
itself inside a chroot jail if it can. The sub-CGI program will also |
|
362
|
run inside this same chroot jail. Make sure all embedded pathnames |
|
363
|
have been adjusted accordingly and that all resources needed by the |
|
364
|
CGI program are available within the chroot jail. |
|
365
|
|
|
366
|
If anything goes wrong while trying to process an /ext page, Fossil |
|
367
|
returns a 404 Not Found error with no details. However, if the requester |
|
368
|
is logged in as a user that has <b>[./caps/ref.html#D | Debug]</b> capability |
|
369
|
then additional diagnostic information may be included in the output. |
|
370
|
|
|
371
|
If the /ext page has a "fossil-ext-debug=1" query parameter and if |
|
372
|
the requester is logged in as a user with Debug privilege, then the |
|
373
|
CGI output is returned verbatim, as text/plain and with the original |
|
374
|
header intact. This is useful for diagnosing problems with the |
|
375
|
CGI script. |
|
376
|
|