|
1
|
# JSON API Introduction |
|
2
|
([⬑JSON API Index](index.md)) |
|
3
|
|
|
4
|
Jump to: |
|
5
|
|
|
6
|
* [Why?](#why) |
|
7
|
* [Building JSON Support](#builing) |
|
8
|
* [Goals & Non-goals](#goals) |
|
9
|
* [Potential Client-side Uses](#potential-uses) |
|
10
|
* [Technical Problems and Considerations](#considerations) |
|
11
|
|
|
12
|
--- |
|
13
|
|
|
14
|
<a id="why"></a> |
|
15
|
# Why? |
|
16
|
|
|
17
|
In September, 2011, Fossil contributor Stephan Beal had the great |
|
18
|
pleasure of meeting D. Richard Hipp, Fossil's author, for lunch in |
|
19
|
Munich, Germany. During the conversation Richard asked, "what does |
|
20
|
Fossil need next?" Stephan's first answer was, "refactoring into a |
|
21
|
library/client, as opposed to a monolithic app." We very quickly |
|
22
|
agreed that the effort required would be "herculean," and second |
|
23
|
choice was voiced, "a JSON API." They briefly discussed the idea and |
|
24
|
Richard gave his blessing. That night work began. |
|
25
|
|
|
26
|
Why a JSON API? Because it is the next best thing to the |
|
27
|
"librification" of Fossil, in that it makes Fossil's features |
|
28
|
available to near-arbitrary applications using a simple, globally |
|
29
|
available data format. |
|
30
|
|
|
31
|
<a id="building"></a> |
|
32
|
# Building JSON Support |
|
33
|
|
|
34
|
In environments supported by fossil's `configure` script, |
|
35
|
simply pass `--enable-json` to it: |
|
36
|
|
|
37
|
``` |
|
38
|
$ ./configure --prefix=$HOME --enable-json ... |
|
39
|
``` |
|
40
|
|
|
41
|
When built without that option, JSON support is disabled. **When |
|
42
|
reconfiguring the source tree**, ***always be sure to do a "make |
|
43
|
clean"*** (or equivalent for your platform) between builds (preferably |
|
44
|
*before* reconfiguring), to ensure that everything is rebuilt properly. |
|
45
|
If you fail to do that after enabling JSON on a tree which has already |
|
46
|
been built, most of the sources will not be rebuilt properly. The reason |
|
47
|
is that the JSON files are actually unconditionally compiled, but when |
|
48
|
built without `--enable-json` they compile to empty object files. Thus |
|
49
|
after a reconfigure the (empty) object files are still up-to-date |
|
50
|
vis-a-vis the sources, and won't be rebuilt. |
|
51
|
|
|
52
|
To build Fossil with JSON support on Windows using the Microsoft C |
|
53
|
compiler: |
|
54
|
|
|
55
|
``` |
|
56
|
cd win |
|
57
|
nmake -f Makefile.msc FOSSIL_ENABLE_JSON=1 |
|
58
|
``` |
|
59
|
|
|
60
|
It has been seen to compile in VC versions 6 and higher. |
|
61
|
|
|
62
|
<a id="goals"></a> |
|
63
|
# Goals & Non-goals |
|
64
|
|
|
65
|
The API described here is most certainly not |
|
66
|
[*REST*](http://en.wikipedia.org/wiki/Representational_state_transfer)-conformant, |
|
67
|
but is instead JSON over HTTP. The error reporting techniques of the |
|
68
|
REST conventions (using HTTP error codes) "does not mesh" with my ideas |
|
69
|
of separation of transport- vs. app-side errors. Additionally, REST |
|
70
|
requires HTTP methods which are not specified by CGI (namely PUT and |
|
71
|
DELETE), which means we can't possibly implement a REST-compatible |
|
72
|
interface on top of fossil (which uses CGI mode even for its built-in |
|
73
|
server). |
|
74
|
|
|
75
|
The **overall goals** of this effort include: |
|
76
|
|
|
77
|
- A JSON-based API off of which clients can build customized Fossil |
|
78
|
UIs and special-purpose applications. e.g. a desktop notification |
|
79
|
applet which polls for new timeline data. |
|
80
|
- Arbitrary JSON-using clients should be able to use it. Though JSON |
|
81
|
originates from JavaScript, it is truly a cross-platform data format |
|
82
|
with a very high adoption rate. (There’s even a JSON implementation |
|
83
|
for Oracle PL/SQL.) |
|
84
|
- Fossil’s CGI and Server modes are the main targets and should be |
|
85
|
supported equally. CLI JSON mode is of secondary concern (but is in |
|
86
|
practice easier to test, so it’s generally implemented first). |
|
87
|
|
|
88
|
The ***non-goals*** include: |
|
89
|
|
|
90
|
- We won’t be able to implement *every* feature of Fossil via a JSON |
|
91
|
interface, and we won’t try to. |
|
92
|
- Binary data (e.g. commits of binary files or downloading ZIP files) |
|
93
|
is not an initial goal, but "might be interesting" once the overall |
|
94
|
infrastructure is in place and working well. See below for more |
|
95
|
details about binary data in JSON. |
|
96
|
- A "pure REST" interface is seemingly not possible due to REST |
|
97
|
relying on HTTP methods not specified in the CGI standard (PUT and |
|
98
|
DELETE). Additionally, REST-style error reporting cannot be used by |
|
99
|
non-HTTP clients (which this code supports). |
|
100
|
|
|
101
|
Adding JSON support also gives us a framework off of which to |
|
102
|
build/enhance other features. Some examples include: |
|
103
|
|
|
104
|
- **Internationalization**. Errors are reported via standard codes and |
|
105
|
the raw artifact data is language-independent. |
|
106
|
- The ability to author **special-case clients**, e.g. a ticket |
|
107
|
poller. |
|
108
|
- Use **arbitrary HTTP-capable languages** to implement such tools. |
|
109
|
Programming languages which can execute programs and intercept their |
|
110
|
stdout output can use the JSON API via a local fossil binary. |
|
111
|
- **Automatable tests.** Many of fossil's test results currently have |
|
112
|
to be "visually reviewed" for correctness after changes (e.g. |
|
113
|
changes in the HTML interface). JSON structures can be |
|
114
|
programmatically checked for correctness. Artifacts are immutable, |
|
115
|
which allows us to be very specific in what data to expect as output |
|
116
|
(for artifact-specific requests the payload data will often (but not |
|
117
|
always) be the same across all requests and all time). |
|
118
|
|
|
119
|
<a id="potential-uses"></a> |
|
120
|
# Potential Client-side Uses |
|
121
|
|
|
122
|
Some of the potential client-side uses of this API include... |
|
123
|
|
|
124
|
- Custom apps/applets to fetch timeline/ticket/etc. information from |
|
125
|
arbitrary repositories. There are many possibilities here, including |
|
126
|
"dashboard" sites which monitor several repositories. |
|
127
|
- Custom post-commit triggers, by polling for changes and reacting to |
|
128
|
them (e.g. sending mails). |
|
129
|
- A custom wiki front-end which uses fossil as the back-end storage, |
|
130
|
inheriting its versioning and user access support while providing a |
|
131
|
completely custom wiki-centric UI. Such a wiki need not have, on the |
|
132
|
surface, anything to do with fossil or source control, as fossil |
|
133
|
would just become a glorified wiki back-end. This approach also |
|
134
|
allows clients to serve wiki pages in a format of their choice - |
|
135
|
since all rendering would be done client-side, they could use |
|
136
|
whatever format they like. |
|
137
|
|
|
138
|
|
|
139
|
<a id="considerations"></a> |
|
140
|
# Technical Problems and Considerations |
|
141
|
|
|
142
|
A random list of considerations which need to be made and potential |
|
143
|
problem areas... |
|
144
|
|
|
145
|
- **Binary data:** JSON is a text serialization method, and it takes |
|
146
|
up the “payload” area of each HTTP request, so there is no |
|
147
|
reasonable way to include binary data in the JSON message without |
|
148
|
some sort of codec like Base64, for which there is no provision in |
|
149
|
the current JSON API. You will therefore find no JSON API for |
|
150
|
committing changes to a file in the repository, for example. Other |
|
151
|
Fossil APIs such as [`/raw`](/help/www/raw) or |
|
152
|
[`/fileedit`](../fileedit-page.md) may serve you better. |
|
153
|
- **64-bit integers:** The JSON standard does not specify integer precision, |
|
154
|
because it targets many different platforms, and not all of |
|
155
|
them can support more than 32 bits. JavaScript (from which JSON |
|
156
|
derives) supports 53 bits of integer precision, which may affect how |
|
157
|
a given client-side JSON implementation sends large integers to Fossil’s JSON |
|
158
|
API. Our JSON parser can cope with integers larger than 32 bits on input, and it |
|
159
|
can emit them, but it requires platform support. If you’re running |
|
160
|
Fossil on a 64-bit host, you should not run into problems in |
|
161
|
this area, but if you’re on a legacy 32-bit only or a mixed 32/64-bit |
|
162
|
system, it’s possible that some integers in the API could be |
|
163
|
clipped. Realize however that this is a rare case: Fossil currently |
|
164
|
cannot store files large enough to exceed a 32-bit `size_t` value, |
|
165
|
and `time_t` won’t roll past 32-bit integers until 2038. We’re aware |
|
166
|
of no other uses of integers in this API that could even in |
|
167
|
principle exceed the range of a 32-bit integer. |
|
168
|
- **Timestamps:** For portability, this API uses UTC Unix epoch |
|
169
|
timestamps. (`time_t`) They are the most portable time representation out |
|
170
|
there, easily usable in most programming environments. (In |
|
171
|
hindsight, we might better have used a higher-precision time format, |
|
172
|
but changing that now would break API compatibility.) |
|
173
|
|