|
1
|
<title>The Fossil Build Process</title> |
|
2
|
|
|
3
|
<h1>1.0 Introduction</h1> |
|
4
|
|
|
5
|
The build process for Fossil is tricky in that the source code |
|
6
|
needs to be processed by three different preprocessor programs |
|
7
|
before it is compiled. Most users will download a |
|
8
|
[https://fossil-scm.org/home/uv/download.html | precompiled binary] |
|
9
|
so this is of no consequence to them, and even those who |
|
10
|
want to compile the code themselves can use one of the |
|
11
|
[./build.wiki | existing makefiles]. |
|
12
|
So most people do not need to be concerned with the |
|
13
|
build complexities of Fossil. But hard-core developers who desire |
|
14
|
a deep understanding of how Fossil is put together can benefit |
|
15
|
from reviewing this article. |
|
16
|
|
|
17
|
<h1 id="srctour">2.0 Source Code Tour</h1> |
|
18
|
|
|
19
|
The source code for Fossil is found in the |
|
20
|
[/dir?ci=trunk&name=src | src/] subdirectory of the |
|
21
|
source tree. The src/ subdirectory contains all code, including |
|
22
|
the code for the separate preprocessor programs. |
|
23
|
|
|
24
|
Each preprocessor program is a separate C program implemented in |
|
25
|
a single file of C source code. The three preprocessor programs |
|
26
|
are: |
|
27
|
|
|
28
|
1. [/file/tools/mkindex.c | mkindex.c] |
|
29
|
2. [/file/tools/translate.c | translate.c] |
|
30
|
3. [/file/tools/makeheaders.c | makeheaders.c] |
|
31
|
|
|
32
|
Fossil uses [http://www.sqlite.org/ | SQLite] for on-disk |
|
33
|
storage. The SQLite implementation is contained in three source |
|
34
|
code files that do not participate in the preprocessing steps. |
|
35
|
These three files that implement SQLite are: |
|
36
|
|
|
37
|
4. sqlite3.c |
|
38
|
5. sqlite3.h |
|
39
|
6. shell.c |
|
40
|
|
|
41
|
All three SQLite source files are byte-for-byte copies of files by |
|
42
|
the same name in the |
|
43
|
standard [http://www.sqlite.org/amalgamation.html | amalgamation]. |
|
44
|
The sqlite3.c file implements the database engine. The shell.c file |
|
45
|
implements the command-line shell, which is accessed in fossil using |
|
46
|
the [/help/sqlite3 | fossil sql] command. |
|
47
|
|
|
48
|
The shell.c command-line shell uses the [https://github.com/antirez/linenoise | |
|
49
|
linenoise] library to implement line editing. linenoise comprises two |
|
50
|
source files which were copied from the upstream repository with only |
|
51
|
very minor portability edits: |
|
52
|
|
|
53
|
7. linenoise.c |
|
54
|
8. linenoise.h |
|
55
|
|
|
56
|
The TH1 script engine is implemented using files: |
|
57
|
|
|
58
|
9. th.c |
|
59
|
10. th.h |
|
60
|
|
|
61
|
The preprocessing steps are omitted for all of these imported |
|
62
|
files. |
|
63
|
|
|
64
|
The VERSION.h header file is generated from other information sources |
|
65
|
using a small program called: |
|
66
|
|
|
67
|
11. [/file/tools/mkversion.c | mkversion.c] |
|
68
|
|
|
69
|
The builtin_data.h header file contains the definitions of C-language |
|
70
|
byte-array constants that contain various resources such as scripts and |
|
71
|
images. The builtin_data.h header file is generated from the original |
|
72
|
resource files using a small program called: |
|
73
|
|
|
74
|
12 [/file/tools/mkbuiltin.c | mkbuiltin.c] |
|
75
|
|
|
76
|
Examples of built-in resources include the [/file/src/diff.tcl | diff.tcl] |
|
77
|
script used to implement the --tk option to [/help/diff| fossil diff], |
|
78
|
the [/file/src/markdown.md | markdown documentation], and the various |
|
79
|
CSS scripts, headers, and footers used to implement built-in skins. New |
|
80
|
resources files are added to the "extra_files" variable in |
|
81
|
[/file/tools/makemake.tcl | makemake.tcl]. |
|
82
|
|
|
83
|
The src/ subdirectory also contains documentation about the |
|
84
|
makeheaders preprocessor program: |
|
85
|
|
|
86
|
13. [../tools/makeheaders.html | makeheaders.html] |
|
87
|
|
|
88
|
Click on the link to read this documentation. In addition there is |
|
89
|
a [http://www.tcl-lang.org/ | Tcl] script used to build the various makefiles: |
|
90
|
|
|
91
|
14. makemake.tcl |
|
92
|
|
|
93
|
Running this Tcl script will automatically regenerate all makefiles. |
|
94
|
In order to add a new source file to the Fossil implementation, simply |
|
95
|
edit makemake.tcl to add the new filename, then rerun the script, and |
|
96
|
all of the makefiles for all targets will be rebuilt. |
|
97
|
|
|
98
|
There is an optional code verification step implemented using |
|
99
|
|
|
100
|
15. [/file/tools/codecheck1.c | codecheck1.c] |
|
101
|
|
|
102
|
This file implements a small utility program ("codecheck1") |
|
103
|
that scans other Fossil source files looking for errors in printf-style |
|
104
|
format strings. |
|
105
|
The codecheck1 utility detects missing or surplus arguments on |
|
106
|
printf-like functions and dangerous uses of "%s" that might |
|
107
|
permit SQL injection or cross-site scripting attacks. This code |
|
108
|
check step is run automatically on each build of Fossil, and can |
|
109
|
also be run separately by typing "make codecheck". Note that the |
|
110
|
built-in printf format checking of GCC does not function for Fossil |
|
111
|
since Fossil implements its own printf (in the |
|
112
|
[/file/src/printf.c | printf.c] source file) that includes special |
|
113
|
features and formatting letters that are useful to Fossil. The |
|
114
|
codecheck1 utility can be seen as an enhanced application-specific |
|
115
|
replacement for the GCC printf format checker. |
|
116
|
|
|
117
|
Finally, there is one of the makefiles generated by makemake.tcl: |
|
118
|
|
|
119
|
16. main.mk |
|
120
|
|
|
121
|
The main.mk makefile is invoked from the Makefile in the top-level |
|
122
|
directory. The main.mk is generated by makemake.tcl and should not |
|
123
|
be hand edited. Other makefiles generated by makemake.tcl are in |
|
124
|
other subdirectories (currently all in the win/ subdirectory). |
|
125
|
|
|
126
|
All the other files in the src/ subdirectory (79 files at the time of |
|
127
|
this writing) are C source code files that are subject to the |
|
128
|
preprocessing steps described below. In the sequel, we will call these |
|
129
|
other files "src.c" in order to have a convenient name. The reader |
|
130
|
should understand that whenever "src.c" or "src.h" is used in the text |
|
131
|
that follows, we really mean all (79) other source files other than |
|
132
|
the exceptions described above. |
|
133
|
|
|
134
|
<h1>3.0 Automatically generated files</h1> |
|
135
|
|
|
136
|
The "VERSION.h" header file contains some C preprocessor macros that |
|
137
|
identify the version of Fossil that is to be built. The VERSION.h file is |
|
138
|
generated automatically from information extracted from the "manifest", |
|
139
|
"manifest.uuid", and "VERSION" source files in the root directory of the |
|
140
|
source tree. |
|
141
|
(The "manifest" and "manifest.uuid" files are automatically generated and |
|
142
|
updated by Fossil itself. See the [/help/setting | fossil set manifest] |
|
143
|
command for additional information.) |
|
144
|
|
|
145
|
The VERSION.h header file is generated by |
|
146
|
a C program: tools/mkversion.c. |
|
147
|
To run the VERSION.h generator, first compile the tools/mkversion.c |
|
148
|
source file into a command-line program (named "mkversion.exe") |
|
149
|
then run: |
|
150
|
|
|
151
|
<pre> |
|
152
|
mkversion.exe manifest.uuid manifest VERSION >VERSION.h |
|
153
|
</pre> |
|
154
|
|
|
155
|
The pathnames in the above command might need to be adjusted to get the |
|
156
|
directories right. The point is that the manifest.uuid, manifest, and |
|
157
|
VERSION files |
|
158
|
in the root of the source tree are the three arguments and |
|
159
|
the generated VERSION.h file appears on standard output. |
|
160
|
|
|
161
|
The builtin_data.h header file is generated by a C program: tools/mkbuiltin.c. |
|
162
|
The builtin_data.h file contains C-language byte-array definitions for |
|
163
|
the content of resource files used by Fossil. To generate the |
|
164
|
builtin_data.h file, first compile the mkbuiltin.c program, then run: |
|
165
|
|
|
166
|
<pre> |
|
167
|
mkbuiltin.exe diff.tcl <i>OtherFiles...</i> >builtin_data.h |
|
168
|
</pre> |
|
169
|
|
|
170
|
At the time of this writing, the "diff.tcl" script (a Tcl/Tk script used |
|
171
|
to generate implement --tk option on the diff command) is the only resource |
|
172
|
file processed using mkbuiltin.exe. However, new resources will likely be |
|
173
|
added using this facility in future versions of Fossil. |
|
174
|
|
|
175
|
<h1 id="preprocessing">4.0 Preprocessing</h1> |
|
176
|
|
|
177
|
There are three preprocessors for the Fossil sources. The mkindex |
|
178
|
and translate preprocessors can be run in any order. The makeheaders |
|
179
|
preprocessor must be run after translate. |
|
180
|
|
|
181
|
<h2>4.1 The mkindex preprocessor</h2> |
|
182
|
|
|
183
|
The mkindex program scans the "src.c" source files looking for special |
|
184
|
comments that identify routines that implement various Fossil commands, |
|
185
|
web interface methods, and help text comments. The mkindex program |
|
186
|
generates some C code that Fossil uses in order to dispatch commands and |
|
187
|
HTTP requests and to show on-line help. Compile the mkindex program |
|
188
|
from the mkindex.c source file. Then run: |
|
189
|
|
|
190
|
<pre> |
|
191
|
./mkindex src.c >page_index.h |
|
192
|
</pre> |
|
193
|
|
|
194
|
Note that "src.c" in the above is a stand-in for the (79) regular source |
|
195
|
files of Fossil - all source files except for the exceptions described in |
|
196
|
section 2.0 above. |
|
197
|
|
|
198
|
The output of the mkindex program is a header file that is #include-ed by |
|
199
|
the main.c source file during the final compilation step. |
|
200
|
|
|
201
|
<h2>4.2 The translate preprocessor</h2> |
|
202
|
|
|
203
|
The translate preprocessor looks for lines of source code that begin |
|
204
|
with "@" and converts those lines into string constants or (depending on |
|
205
|
context) into special "printf" operations for generating the output of |
|
206
|
an HTTP request. The translate preprocessor is a simple C program whose |
|
207
|
sources are in the translate.c source file. The translate preprocess |
|
208
|
is run on each of the other ordinary source files separately, like this: |
|
209
|
|
|
210
|
<pre> |
|
211
|
./translate src.c >src_.c |
|
212
|
</pre> |
|
213
|
|
|
214
|
In this case, the "src.c" file represents any single source file from the |
|
215
|
set of ordinary source files as described in section 2.0 above. Note that |
|
216
|
each source file is translated separately. By convention, the names of |
|
217
|
the translated source files are the names of the input sources with a |
|
218
|
single "_" character at the end. But a new makefile can use any naming |
|
219
|
convention it wants - the "_" is not critical to the build process. |
|
220
|
|
|
221
|
After being translated, the output files (the "src_.c" files) should be |
|
222
|
used for all subsequent preprocessing and compilation steps. |
|
223
|
|
|
224
|
<h2>4.3 The makeheaders preprocessor</h2> |
|
225
|
|
|
226
|
For each C source module "src.c", there is an automatically generated |
|
227
|
header module "src.h" that contains all of the datatype and procedure |
|
228
|
declarations needed by the source module. These header files are generated |
|
229
|
automatically by the makeheaders program. The sources to makeheaders |
|
230
|
are contained in a single file "makeheaders.c". Additional documentation |
|
231
|
on makeheaders can be found in [../tools/makeheaders.html | tools/makeheaders.html]. |
|
232
|
|
|
233
|
The makeheaders program is run once. It scans all inputs source files and |
|
234
|
generates header files for each one. Note that the sqlite3.c and shell.c |
|
235
|
source files are not scanned by makeheaders. Makeheaders only runs over |
|
236
|
"ordinary" source files, not the exceptional source files. However, |
|
237
|
makeheaders also uses some extra header files as input. The general format |
|
238
|
is like this: |
|
239
|
|
|
240
|
<pre> |
|
241
|
makeheaders src_.c:src.h sqlite3.h th.h VERSION.h |
|
242
|
</pre> |
|
243
|
|
|
244
|
In the example above the "src_.c" and "src.h" names represent all of the |
|
245
|
(79) ordinary C source files, each as a separate argument. |
|
246
|
|
|
247
|
<h1>5.0 Compilation</h1> |
|
248
|
|
|
249
|
After all generated files have been created and all ordinary source files |
|
250
|
have been preprocessed, the generated and preprocessed files can be |
|
251
|
combined into a single executable using a C compiler. This can be done |
|
252
|
all at once, or each preprocessed source file can be compiled into a |
|
253
|
separate object code file and the resulting object code files linked |
|
254
|
together in a final step. |
|
255
|
|
|
256
|
Some files require special C-preprocessor macro definitions. |
|
257
|
When compiling sqlite.c, the following macros are recommended: |
|
258
|
|
|
259
|
* -DSQLITE_OMIT_LOAD_EXTENSION=1 |
|
260
|
* -DSQLITE_ENABLE_DBSTAT_VTAB=1 |
|
261
|
* -DSQLITE_ENABLE_FTS4=1 |
|
262
|
* -DSQLITE_ENABLE_LOCKING_STYLE=0 |
|
263
|
* -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 |
|
264
|
* -DSQLITE_THREADSAFE=0 |
|
265
|
* -DSQLITE_DEFAULT_FILE_FORMAT=4 |
|
266
|
* -DSQLITE_ENABLE_EXPLAIN_COMMENTS=1 |
|
267
|
|
|
268
|
The first three symbol definitions above are required; the others are merely |
|
269
|
recommended. Extension loading is omitted as a security measure. The dbstat |
|
270
|
virtual table is needed for the [/help/www/repo-tabsize|/repo-tabsize] page. |
|
271
|
FTS4 is needed for the search feature. Fossil is single-threaded so mutexing |
|
272
|
is disabled in SQLite as a performance enhancement. The |
|
273
|
SQLITE_ENABLE_EXPLAIN_COMMENTS option makes the output of "EXPLAIN" queries |
|
274
|
in the "[/help/sqlite3|fossil sql]" command much more readable. |
|
275
|
|
|
276
|
When compiling the shell.c source file, these macros are required: |
|
277
|
|
|
278
|
* -Dmain=sqlite3_main |
|
279
|
* -DSQLITE_OMIT_LOAD_EXTENSION=1 |
|
280
|
|
|
281
|
The "main()" routine in the shell must be changed into sqlite3_main() |
|
282
|
to prevent it from colliding with the real main() in Fossil, and to give |
|
283
|
Fossil an entry point to jump to when the |
|
284
|
[/help/sqlite3 | fossil sql] command is invoked. |
|
285
|
|
|
286
|
All the other source code files can be compiled without any special |
|
287
|
options. |
|
288
|
|
|
289
|
<h1>6.0 Linkage</h1> |
|
290
|
|
|
291
|
Fossil needs to be linked against [http://www.zlib.net | zlib]. If |
|
292
|
the HTTPS option is enabled, then it will also need to link against |
|
293
|
the appropriate SSL implementation. And, of course, Fossil needs to |
|
294
|
link against the standard C library. No other libraries or external |
|
295
|
dependences are used. |
|
296
|
|
|
297
|
<h1>7.0 Debugging</h1> |
|
298
|
|
|
299
|
Debug mode is controlled via a FOSSIL_DEBUG preprocessor macro. This can be |
|
300
|
set explicitly with the make command for the target platform. |
|
301
|
|
|
302
|
However, in practice it is instead recommended to add a respective configure |
|
303
|
option for the target platform and then perform a clean build. This way the |
|
304
|
Debug flags are consistently applied across the whole build process. For |
|
305
|
example, use these Debug flags in addition to other flags passed to the |
|
306
|
configure scripts: |
|
307
|
|
|
308
|
On Linux, *NIX and similar platforms: |
|
309
|
<pre> |
|
310
|
./configure --fossil-debug |
|
311
|
</pre> |
|
312
|
|
|
313
|
On Windows: |
|
314
|
<pre> |
|
315
|
win\buildmsvc.bat FOSSIL_DEBUG=1 |
|
316
|
</pre> |
|
317
|
|
|
318
|
The resulting fossil binary can then be loaded into a platform-specific |
|
319
|
debugger. Source files displayed in the debugger correspond to the ones |
|
320
|
generated from the translation stage of the build process, as that is what |
|
321
|
was actually compiled into the respective object files that make up the |
|
322
|
fossil binary. |
|
323
|
|
|
324
|
<h1>8.0 See Also</h1> |
|
325
|
|
|
326
|
* [./tech_overview.wiki | A Technical Overview Of Fossil] |
|
327
|
* [./adding_code.wiki | How To Add Features To Fossil] |
|
328
|
|