Fossil SCM

fossil-scm / www / json-api / intro.md
Source Blame History 172 lines
c95f11b… stephan 1 # JSON API Introduction
c95f11b… stephan 2 ([⬑JSON API Index](index.md))
c95f11b… stephan 3
c95f11b… stephan 4 Jump to:
c95f11b… stephan 5
c95f11b… stephan 6 * [Why?](#why)
c95f11b… stephan 7 * [Building JSON Support](#builing)
c95f11b… stephan 8 * [Goals & Non-goals](#goals)
c95f11b… stephan 9 * [Potential Client-side Uses](#potential-uses)
c95f11b… stephan 10 * [Technical Problems and Considerations](#considerations)
c95f11b… stephan 11
c95f11b… stephan 12 ---
c95f11b… stephan 13
c95f11b… stephan 14 <a id="why"></a>
c95f11b… stephan 15 # Why?
c95f11b… stephan 16
c95f11b… stephan 17 In September, 2011, Fossil contributor Stephan Beal had the great
c95f11b… stephan 18 pleasure of meeting D. Richard Hipp, Fossil's author, for lunch in
c95f11b… stephan 19 Munich, Germany. During the conversation Richard asked, "what does
c95f11b… stephan 20 Fossil need next?" Stephan's first answer was, "refactoring into a
c95f11b… stephan 21 library/client, as opposed to a monolithic app." We very quickly
c95f11b… stephan 22 agreed that the effort required would be "herculean," and second
c95f11b… stephan 23 choice was voiced, "a JSON API." They briefly discussed the idea and
c95f11b… stephan 24 Richard gave his blessing. That night work began.
c95f11b… stephan 25
c95f11b… stephan 26 Why a JSON API? Because it is the next best thing to the
c95f11b… stephan 27 "librification" of Fossil, in that it makes Fossil's features
c95f11b… stephan 28 available to near-arbitrary applications using a simple, globally
c95f11b… stephan 29 available data format.
c95f11b… stephan 30
c95f11b… stephan 31 <a id="building"></a>
c95f11b… stephan 32 # Building JSON Support
c95f11b… stephan 33
c95f11b… stephan 34 In environments supported by fossil's `configure` script,
c95f11b… stephan 35 simply pass `--enable-json` to it:
c95f11b… stephan 36
c95f11b… stephan 37 ```
c95f11b… stephan 38 $ ./configure --prefix=$HOME --enable-json ...
c95f11b… stephan 39 ```
c95f11b… stephan 40
c95f11b… stephan 41 When built without that option, JSON support is disabled. **When
c95f11b… stephan 42 reconfiguring the source tree**, ***always be sure to do a "make
c95f11b… stephan 43 clean"*** (or equivalent for your platform) between builds (preferably
c95f11b… stephan 44 *before* reconfiguring), to ensure that everything is rebuilt properly.
c95f11b… stephan 45 If you fail to do that after enabling JSON on a tree which has already
c95f11b… stephan 46 been built, most of the sources will not be rebuilt properly. The reason
c95f11b… stephan 47 is that the JSON files are actually unconditionally compiled, but when
c95f11b… stephan 48 built without `--enable-json` they compile to empty object files. Thus
c95f11b… stephan 49 after a reconfigure the (empty) object files are still up-to-date
c95f11b… stephan 50 vis-a-vis the sources, and won't be rebuilt.
c95f11b… stephan 51
c95f11b… stephan 52 To build Fossil with JSON support on Windows using the Microsoft C
c95f11b… stephan 53 compiler:
c95f11b… stephan 54
c95f11b… stephan 55 ```
c95f11b… stephan 56 cd win
c95f11b… stephan 57 nmake -f Makefile.msc FOSSIL_ENABLE_JSON=1
c95f11b… stephan 58 ```
c95f11b… stephan 59
c95f11b… stephan 60 It has been seen to compile in VC versions 6 and higher.
c95f11b… stephan 61
c95f11b… stephan 62 <a id="goals"></a>
c95f11b… stephan 63 # Goals & Non-goals
c95f11b… stephan 64
c95f11b… stephan 65 The API described here is most certainly not
c95f11b… stephan 66 [*REST*](http://en.wikipedia.org/wiki/Representational_state_transfer)-conformant,
c95f11b… stephan 67 but is instead JSON over HTTP. The error reporting techniques of the
c95f11b… stephan 68 REST conventions (using HTTP error codes) "does not mesh" with my ideas
c95f11b… stephan 69 of separation of transport- vs. app-side errors. Additionally, REST
c95f11b… stephan 70 requires HTTP methods which are not specified by CGI (namely PUT and
c95f11b… stephan 71 DELETE), which means we can't possibly implement a REST-compatible
c95f11b… stephan 72 interface on top of fossil (which uses CGI mode even for its built-in
c95f11b… stephan 73 server).
c95f11b… stephan 74
c95f11b… stephan 75 The **overall goals** of this effort include:
c95f11b… stephan 76
c95f11b… stephan 77 - A JSON-based API off of which clients can build customized Fossil
c95f11b… stephan 78 UIs and special-purpose applications. e.g. a desktop notification
c95f11b… stephan 79 applet which polls for new timeline data.
c95f11b… stephan 80 - Arbitrary JSON-using clients should be able to use it. Though JSON
c95f11b… stephan 81 originates from JavaScript, it is truly a cross-platform data format
c95f11b… stephan 82 with a very high adoption rate. (There’s even a JSON implementation
c95f11b… stephan 83 for Oracle PL/SQL.)
c95f11b… stephan 84 - Fossil’s CGI and Server modes are the main targets and should be
c95f11b… stephan 85 supported equally. CLI JSON mode is of secondary concern (but is in
c95f11b… stephan 86 practice easier to test, so it’s generally implemented first).
c95f11b… stephan 87
c95f11b… stephan 88 The ***non-goals*** include:
c95f11b… stephan 89
c95f11b… stephan 90 - We won’t be able to implement *every* feature of Fossil via a JSON
c95f11b… stephan 91 interface, and we won’t try to.
c95f11b… stephan 92 - Binary data (e.g. commits of binary files or downloading ZIP files)
c95f11b… stephan 93 is not an initial goal, but "might be interesting" once the overall
c95f11b… stephan 94 infrastructure is in place and working well. See below for more
c95f11b… stephan 95 details about binary data in JSON.
c95f11b… stephan 96 - A "pure REST" interface is seemingly not possible due to REST
c95f11b… stephan 97 relying on HTTP methods not specified in the CGI standard (PUT and
c95f11b… stephan 98 DELETE). Additionally, REST-style error reporting cannot be used by
c95f11b… stephan 99 non-HTTP clients (which this code supports).
c95f11b… stephan 100
c95f11b… stephan 101 Adding JSON support also gives us a framework off of which to
c95f11b… stephan 102 build/enhance other features. Some examples include:
c95f11b… stephan 103
c95f11b… stephan 104 - **Internationalization**. Errors are reported via standard codes and
c95f11b… stephan 105 the raw artifact data is language-independent.
c95f11b… stephan 106 - The ability to author **special-case clients**, e.g. a ticket
c95f11b… stephan 107 poller.
c95f11b… stephan 108 - Use **arbitrary HTTP-capable languages** to implement such tools.
c95f11b… stephan 109 Programming languages which can execute programs and intercept their
c95f11b… stephan 110 stdout output can use the JSON API via a local fossil binary.
c95f11b… stephan 111 - **Automatable tests.** Many of fossil's test results currently have
c95f11b… stephan 112 to be "visually reviewed" for correctness after changes (e.g.
c95f11b… stephan 113 changes in the HTML interface). JSON structures can be
c95f11b… stephan 114 programmatically checked for correctness. Artifacts are immutable,
c95f11b… stephan 115 which allows us to be very specific in what data to expect as output
c95f11b… stephan 116 (for artifact-specific requests the payload data will often (but not
c95f11b… stephan 117 always) be the same across all requests and all time).
c95f11b… stephan 118
c95f11b… stephan 119 <a id="potential-uses"></a>
c95f11b… stephan 120 # Potential Client-side Uses
c95f11b… stephan 121
c95f11b… stephan 122 Some of the potential client-side uses of this API include...
c95f11b… stephan 123
c95f11b… stephan 124 - Custom apps/applets to fetch timeline/ticket/etc. information from
c95f11b… stephan 125 arbitrary repositories. There are many possibilities here, including
c95f11b… stephan 126 "dashboard" sites which monitor several repositories.
c95f11b… stephan 127 - Custom post-commit triggers, by polling for changes and reacting to
c95f11b… stephan 128 them (e.g. sending mails).
c95f11b… stephan 129 - A custom wiki front-end which uses fossil as the back-end storage,
c95f11b… stephan 130 inheriting its versioning and user access support while providing a
c95f11b… stephan 131 completely custom wiki-centric UI. Such a wiki need not have, on the
c95f11b… stephan 132 surface, anything to do with fossil or source control, as fossil
c95f11b… stephan 133 would just become a glorified wiki back-end. This approach also
c95f11b… stephan 134 allows clients to serve wiki pages in a format of their choice -
c95f11b… stephan 135 since all rendering would be done client-side, they could use
c95f11b… stephan 136 whatever format they like.
c95f11b… stephan 137
c95f11b… stephan 138
c95f11b… stephan 139 <a id="considerations"></a>
c95f11b… stephan 140 # Technical Problems and Considerations
c95f11b… stephan 141
c95f11b… stephan 142 A random list of considerations which need to be made and potential
c95f11b… stephan 143 problem areas...
c95f11b… stephan 144
0f20b53… wyoung 145 - **Binary data:** JSON is a text serialization method, and it takes
0f20b53… wyoung 146 up the “payload” area of each HTTP request, so there is no
0f20b53… wyoung 147 reasonable way to include binary data in the JSON message without
0f20b53… wyoung 148 some sort of codec like Base64, for which there is no provision in
0f20b53… wyoung 149 the current JSON API. You will therefore find no JSON API for
0f20b53… wyoung 150 committing changes to a file in the repository, for example. Other
c64f28d… drh 151 Fossil APIs such as [`/raw`](/help/www/raw) or
0f20b53… wyoung 152 [`/fileedit`](../fileedit-page.md) may serve you better.
0f20b53… wyoung 153 - **64-bit integers:** The JSON standard does not specify integer precision,
0f20b53… wyoung 154 because it targets many different platforms, and not all of
0f20b53… wyoung 155 them can support more than 32 bits. JavaScript (from which JSON
0f20b53… wyoung 156 derives) supports 53 bits of integer precision, which may affect how
0f20b53… wyoung 157 a given client-side JSON implementation sends large integers to Fossil’s JSON
0f20b53… wyoung 158 API. Our JSON parser can cope with integers larger than 32 bits on input, and it
0f20b53… wyoung 159 can emit them, but it requires platform support. If you’re running
0f20b53… wyoung 160 Fossil on a 64-bit host, you should not run into problems in
0f20b53… wyoung 161 this area, but if you’re on a legacy 32-bit only or a mixed 32/64-bit
0f20b53… wyoung 162 system, it’s possible that some integers in the API could be
0f20b53… wyoung 163 clipped. Realize however that this is a rare case: Fossil currently
0f20b53… wyoung 164 cannot store files large enough to exceed a 32-bit `size_t` value,
0f20b53… wyoung 165 and `time_t` won’t roll past 32-bit integers until 2038. We’re aware
0f20b53… wyoung 166 of no other uses of integers in this API that could even in
0f20b53… wyoung 167 principle exceed the range of a 32-bit integer.
0f20b53… wyoung 168 - **Timestamps:** For portability, this API uses UTC Unix epoch
0f20b53… wyoung 169 timestamps. (`time_t`) They are the most portable time representation out
0f20b53… wyoung 170 there, easily usable in most programming environments. (In
0f20b53… wyoung 171 hindsight, we might better have used a higher-precision time format,
0f20b53… wyoung 172 but changing that now would break API compatibility.)

Keyboard Shortcuts

Open search /
Next entry (timeline) j
Previous entry (timeline) k
Open focused entry Enter
Show this help ?
Toggle theme Top nav button