|
1
|
# JSON API: Hacker's Guide |
|
2
|
([⬑JSON API Index](index.md)) |
|
3
|
|
|
4
|
Jump to: |
|
5
|
|
|
6
|
* [Before Committing Changes](#before-committing) |
|
7
|
* [JSON C API](#json-c-api) |
|
8
|
* [Reporting Errors](#reporting-errors) |
|
9
|
* [Getting Command Arguments](#command-args) |
|
10
|
* [Creating JSON Data](#creating-json) |
|
11
|
* [Creating JSON Values](#creating-json-values) |
|
12
|
* [Converting SQL Query Results to JSON](#query-to-json) |
|
13
|
|
|
14
|
This section will only be of interest to those wanting to work on the |
|
15
|
Fossil/JSON code. That said... |
|
16
|
|
|
17
|
If you happen to hack on the code and find something worth noting here |
|
18
|
for others, please feel free to expand this section. It will only |
|
19
|
improve via feedback from those working on the code. |
|
20
|
|
|
21
|
--- |
|
22
|
|
|
23
|
<a id="before-committing"></a> |
|
24
|
# Before Committing Changes... |
|
25
|
|
|
26
|
Because this code lives in the trunk, there are certain |
|
27
|
guidelines which must be followed before committing any changes: |
|
28
|
|
|
29
|
1. Read the [checkin preparation list](/doc/trunk/www/checkin.wiki). |
|
30
|
2. Changes to the files `src/json_*.*`, and its related support code |
|
31
|
(e.g. `ajax/*.*`), may be made freely without affecting mainline |
|
32
|
users. Changes to other files, unless they are trivial or made for |
|
33
|
purposes outside the JSON API (e.g. an unrelated bug fix), must be |
|
34
|
reviewed carefully before committing. When in doubt, create a branch |
|
35
|
and post a request for a review. |
|
36
|
3. The Golden Rule is: *do not break the trunk build*. |
|
37
|
|
|
38
|
|
|
39
|
<a id="json-c-api"></a> |
|
40
|
# JSON C API |
|
41
|
|
|
42
|
libcson, the underlying JSON API, is a separate project, included in |
|
43
|
fossil in "amalgamation" form: see `extsrc/cson_amalgamation.[ch]`. It has |
|
44
|
thorough API docs and a good deal of information is in its wiki: |
|
45
|
|
|
46
|
[](https://fossil.wanderinghorse.net/wikis/cson/) |
|
47
|
|
|
48
|
In particular: |
|
49
|
|
|
50
|
[](https://fossil.wanderinghorse.net/wikis/cson/?page=CsonArchitecture) |
|
51
|
|
|
52
|
gives an overview of its architecture. Occasionally new versions of it |
|
53
|
are pulled into the Fossil tree, but other developers generally need not |
|
54
|
concern themselves with that. |
|
55
|
|
|
56
|
(Trivia: the cson wiki's back-end is fossil using this very JSON API, |
|
57
|
living on top of a custom JavaScript+HTML5 application.) |
|
58
|
|
|
59
|
Only a small handful of low-level fossil routines actually input or |
|
60
|
output JSON text (only for reading in POST data and sending the |
|
61
|
response). In the C code we work with the higher-level JSON value |
|
62
|
abstractions provided by cson (conceptually similar to an XML DOM). All |
|
63
|
of the JSON-defined data types are supported, and we can construct JSON |
|
64
|
output of near arbitrary complexity with the caveat that *cyclic data |
|
65
|
structures are strictly forbidden*, and *will* cause memory corruption, |
|
66
|
crashes, double free()'s, or other undefined behaviour. Because JSON |
|
67
|
cannot, without client-specific semantic extensions to JSON, represent |
|
68
|
cyclic structures, it is not anticipated that this will be a |
|
69
|
problem/limitation when generating output for fossil. |
|
70
|
|
|
71
|
|
|
72
|
|
|
73
|
<a id="json-commands"></a> |
|
74
|
# Architecture of JSON Commands |
|
75
|
|
|
76
|
In order to consolidate CLI/HTTP modes for JSON handling, this code |
|
77
|
foregoes fossil's conventional command/path dispatching mechanism. Only |
|
78
|
the top-most "json" command/path is dispatched directly by fossil's |
|
79
|
core. The disadvantages of this are that we lose fossil's conventional |
|
80
|
help text mechanism (which is based on code comments in the |
|
81
|
command/path's dispatcher impl) and the ability to write abbreviated |
|
82
|
command names in CLI mode ("json" itself may be abbreviated, but not the |
|
83
|
subcommands). The advantages are that we can handle CLI/HTTP modes |
|
84
|
almost identically (there are a couple of minor differences) by unifying |
|
85
|
them under the same callback functions much more easily. |
|
86
|
|
|
87
|
The top-level "json" command/path uses its own dispatching mechanism |
|
88
|
which uses either the path (in HTTP mode) or CLI positional arguments to |
|
89
|
dispatch commands (stopping at the first "flag option" (e.g. -foo) in |
|
90
|
CLI mode). The command handlers are simply callback functions which |
|
91
|
return a cson\_value pointer (the C representation of an arbitrary JSON |
|
92
|
value), representing the "payload" of the response (or NULL - not all |
|
93
|
responses need a payload). On error these callbacks set the internal |
|
94
|
JSON error state (detailed in a subsection below) and return NULL. The |
|
95
|
top-level dispatcher then creates a response envelope and returns the |
|
96
|
"payload" from the command (if any) to the caller. If a callback sets |
|
97
|
the error state, the top-level dispatcher takes care to set the error |
|
98
|
information in the response envelope. In summary: |
|
99
|
|
|
100
|
- The top-level dispatchers (`json_page_top()` and `json_cmd_top()`) |
|
101
|
are called by fossil's core when the "json" command/path is called. |
|
102
|
They initialize the JSON-mode global state, dispatch the requested |
|
103
|
command, and handle the creation of the response envelope. They |
|
104
|
prepare all the basic things which the individual subcommands need |
|
105
|
in order to function. |
|
106
|
- The command handlers (most are named `json_page_something()`) |
|
107
|
implement the `fossil_json_f()` callback interface (see |
|
108
|
[`src/json_detail.h`](/finfo/src/json_detail.h)). They are |
|
109
|
responsible for permissions checking, setting any error state, and |
|
110
|
passing back a payload (if needed - not all commands return a |
|
111
|
payload). It is strictly forbidden for these callbacks to produce |
|
112
|
any output on stdout/stderr, and doing so effectively corrupts the |
|
113
|
out-bound JSON and HTTP headers. |
|
114
|
|
|
115
|
There is a wrench in all of that, however: the vast majority of fossil's |
|
116
|
commands "fail fast" - they will `exit()` if they encounter an error. To |
|
117
|
handle that, the fossil core error reporting routines have been |
|
118
|
refactored a small bit to operate differently when we are running in |
|
119
|
JSON mode. Instead of the conventional output, they generate a JSON |
|
120
|
error response. In HTTP mode they exit with code 0 to avoid causing an |
|
121
|
HTTP 500 error, whereas in CLI mode they will exit with a non-0 code. |
|
122
|
Those routines still `exit()`, as in the conventional CLI/HTTP modes, but |
|
123
|
they will exit differently. Because of this, it is perfectly fine for a |
|
124
|
command handler to exit via one of fossil's conventional mechanisms |
|
125
|
(e.g. `db_prepare()` can be fatal, and callbacks may call `fossil_panic()` |
|
126
|
if they really want to). One exception is `fossil_exit()`, which does |
|
127
|
_not_ generate any extra output and will `exit()` the app. In the JSON |
|
128
|
API, as a rule of thumb, `fossil_exit()` is only used when we *want* a |
|
129
|
failed request to cause an HTTP 500 error, and it is reserved for |
|
130
|
allocation errors and similar truly catastrophic failures. That said... |
|
131
|
libcson has been hacked to use `fossil_alloc()` and friends for memory |
|
132
|
management, and those routines exit on error, so alloc error handling in |
|
133
|
the JSON command handler code can afford to be a little lax (the |
|
134
|
majority of *potential* errors clients get from the cson API have |
|
135
|
allocation failure as their root cause). |
|
136
|
|
|
137
|
As a side-note: the vast majority (if not all) of the cson API calls are |
|
138
|
"NULL-safe", meaning that will return an error code (or be a no-op) if |
|
139
|
passed NULL arguments. e.g. the following chain of calls will not crash |
|
140
|
if the value we're looking for does not exist, is-not-a String (see |
|
141
|
`cson_value_get_string()` for important details), or if `myObj` is NULL: |
|
142
|
|
|
143
|
```c |
|
144
|
const char * str = |
|
145
|
cson_string_cstr( // get the C-string form of a cson_string |
|
146
|
cson_value_get_string( // get its cson_string form |
|
147
|
cson_object_get(myObj,"foo") // search for key in an Object |
|
148
|
) |
|
149
|
); |
|
150
|
``` |
|
151
|
|
|
152
|
If `"foo"` is not found in `myObj` (or if `myObj` is NULL) then v will be |
|
153
|
NULL, as opposed to stepping on a NULL pointer somewhere in that call |
|
154
|
chain. |
|
155
|
|
|
156
|
Note that all cson JSON values except Arrays and Objects are *immutable* |
|
157
|
- you cannot change a string's or number's value, for example. They also |
|
158
|
use reference counting to manage ownership, as documented and |
|
159
|
demonstrated on this page: |
|
160
|
|
|
161
|
[](https://fossil.wanderinghorse.net/wikis/cson/?page=TipsAndTricks) |
|
162
|
|
|
163
|
In short, after creating a new value you must eventually *either* add it |
|
164
|
to a container (Object or Array) to transfer ownership *or* call |
|
165
|
`cson_value_free()` to clean it up (exception: the Fossil/JSON command |
|
166
|
callbacks *return* a value to transfer ownership to the dispatcher). |
|
167
|
Sometimes it's more complex than that, but not normally. Any given value |
|
168
|
may legally be stored in any number of containers (or multiple times |
|
169
|
within one container), as long as *no cycles* are introduced (cycles |
|
170
|
*will* cause undefined behaviour). Ownership is shared using reference |
|
171
|
counting and the value will eventually be freed up when its last |
|
172
|
remaining reference is freed (e.g. when the last container holding it is |
|
173
|
cleaned up). For many examples of using cson in the context of fossil, |
|
174
|
see the existing `json_page_xxx()` functions in `json_*.c`. |
|
175
|
|
|
176
|
<a id="reporting-errors"></a> |
|
177
|
# Reporting Errors |
|
178
|
|
|
179
|
To report an error from a command callback, one abstractly needs to: |
|
180
|
|
|
181
|
- Set g.json.resultCode to one of the `FSL_JSON_E_xxx` values |
|
182
|
(defined in [`src/json_detail.h`](/finfo/src/json_detail.h)). |
|
183
|
- *Optionally* set `g.zErrMsg` to contain the (dynamically-allocated!) |
|
184
|
error string to be sent to the client. If no error string is set |
|
185
|
then a standard/generic string is used for the given error code. |
|
186
|
- Clean up any resources created so far by the handler. |
|
187
|
- Return NULL. If it returns non-NULL, the dispatcher will destroy the |
|
188
|
value and not include it in the error response. |
|
189
|
|
|
190
|
That normally looks something like this: |
|
191
|
|
|
192
|
``` |
|
193
|
if(!g.perm.Read){ |
|
194
|
json_set_err(FSL_JSON_E_DENIED, "Requires 'o' permissions."); |
|
195
|
return NULL; |
|
196
|
} |
|
197
|
``` |
|
198
|
|
|
199
|
`json_set_err()` is a variadic printf-like function, and can use the |
|
200
|
printf extensions supported by mprintf() and friends (e.g. `%Q` and `%q`) |
|
201
|
(but they are normally not needed in the context of JSON). If the error |
|
202
|
string is NULL or empty then `json_err_cstr(errorCode)` is used to fetch |
|
203
|
the standard/generic error string for the given code. |
|
204
|
|
|
205
|
When control returns to the top-level dispatching function it will check |
|
206
|
`g.json.resultCode` and, if it is not 0, create an error response using |
|
207
|
the `g.json.resultCode` and `g.zErrMsg` to construct the response's |
|
208
|
`resultCode` and `resultText` properties. |
|
209
|
|
|
210
|
If a function wants to output an error and exit by itself, as opposed |
|
211
|
to returning to the dispatcher, then it must behave slightly |
|
212
|
differently. See the docs for `json_err()` (in |
|
213
|
[`src/json.c`](/finfo/src/json.c)) for details, and search that file |
|
214
|
for various examples of its usage. It is also used by fossil's core |
|
215
|
error-reporting APIs, e.g. `fossil_panic()` (defined in [`src/main.c`](/finfo/src/main.c)). |
|
216
|
That said, it would be "highly unusual" for a callback to need to do |
|
217
|
this - it is *far* simpler (and more consistent/reliable) to set the |
|
218
|
error state and return to the dispatcher. |
|
219
|
|
|
220
|
<a id="command-args"></a> |
|
221
|
# Getting Command Arguments |
|
222
|
|
|
223
|
Positional parameters can be fetched usinig `json_command_arg(N)`, where |
|
224
|
N is the argument position, with position 0 being the "json" |
|
225
|
command/path. In CLI mode positional arguments have their obvious |
|
226
|
meaning. In HTTP mode the request path (or the "command" request |
|
227
|
property) is used to build up the "command path" instead. For example: |
|
228
|
|
|
229
|
CLI: `fossil json a b c` |
|
230
|
|
|
231
|
HTTP: `/json/a/b/c` |
|
232
|
|
|
233
|
HTTP POST or CLI with `--json-input`: /json with POSTed envelope |
|
234
|
`{"command": "a/b/c" …}` |
|
235
|
|
|
236
|
Those will have identical "command paths," and `json_command_path(2)` |
|
237
|
would return the "b" part. |
|
238
|
|
|
239
|
Caveat: a limitation of this support is that all CLI flags must come |
|
240
|
*after* all *non-flag* positional arguments (e.g. file names or |
|
241
|
subcommand names). Any argument starting with a dash ("-") is considered |
|
242
|
by this code to be a potential "flag" argument, and all arguments after |
|
243
|
it are ignored (because the generic handling cannot know if a flag |
|
244
|
requires an argument, which changes how the rest of the arguments need |
|
245
|
to be interpreted). |
|
246
|
|
|
247
|
To get named parameters, there are several approaches (plus some special |
|
248
|
cases). Named parameters can normally come from any of the following |
|
249
|
sources: |
|
250
|
|
|
251
|
- CLI arguments, e.g. `--foo bar` |
|
252
|
- GET parameters: `/json/...?foo=bar` |
|
253
|
- Properties of the POST envelope |
|
254
|
- Properties of the `POST.payload` object (if any). |
|
255
|
|
|
256
|
To try to simplify the guessing process the API has a number of |
|
257
|
functions which behave ever so slightly differently. A summary: |
|
258
|
|
|
259
|
- `json_getenv()` and `json_getenv_TYPE()` search the so-called "JSON |
|
260
|
environment," which is a superset of the GET/POST/`POST.payload` (if |
|
261
|
`POST.payload` is-a Object). |
|
262
|
- `json_find_option_TYPE()`: searches the CLI args (only when in CLI |
|
263
|
mode) and the JSON environment. |
|
264
|
- The use of fossil's `P()` and `PD()` macros is discouraged in JSON |
|
265
|
callbacks because they can only handle String data from the CLI or |
|
266
|
GET parameters (not POST/`POST.payload`). (Note that `P()` and `PD()` |
|
267
|
*normally* also handle POSTed keys, but they only "see" values |
|
268
|
posted as form-urlencoded fields, and not JSON format.) |
|
269
|
- `find_option()` (from `src/main.c`) "should" also be avoided in |
|
270
|
JSON API handlers because it removes flag from the g.argv |
|
271
|
arguments list. That said, the JSON API does use `find_option()` in |
|
272
|
several of its option-finding convenience wrappers. |
|
273
|
|
|
274
|
For example code: the existing command callbacks demonstrate all kinds |
|
275
|
of uses and the various styles of parameter/option inspection. Check out |
|
276
|
any of the functions named `json_page_SOMETHING()`. |
|
277
|
|
|
278
|
<a href="creating-json"></a> |
|
279
|
# Creating JSON Data |
|
280
|
|
|
281
|
<a href="creating-json-values"></a> |
|
282
|
## Creating JSON Values |
|
283
|
|
|
284
|
cson has a fairly rich API for creating and manipulating the various |
|
285
|
JSON-defined value types. For a detailed overview and demonstration I |
|
286
|
recommend reading: |
|
287
|
|
|
288
|
[](https://fossil.wanderinghorse.net/wikis/cson/?page=HowTo) |
|
289
|
|
|
290
|
That said, the Fossil/JSON API has several convenience wrappers to save |
|
291
|
a few bytes of typing: |
|
292
|
|
|
293
|
- `json_new_string("foo")` is easier to use than |
|
294
|
`cson_value_new_string("foo", 3)`, and |
|
295
|
`json_new_string_f("%s","foo")` is more flexible. |
|
296
|
- `json_new_int()` is easier to type than `cson_value_new_integer()`. |
|
297
|
- `cson_output_Blob()` and `cson_parse_Blob()` can write/read JSON |
|
298
|
to/from fossil `Blob`-type objects. |
|
299
|
|
|
300
|
It also provides several lower-level JSON features which aren't of |
|
301
|
general utility but provide necessary functionality for some of the |
|
302
|
framework-level code (e.g. `cson_data_dest_cgi()`), which is only used |
|
303
|
by the deepest of the JSON internals). |
|
304
|
|
|
305
|
|
|
306
|
<a href="query-to-json"></a> |
|
307
|
## Converting SQL Query Results to JSON |
|
308
|
|
|
309
|
The `cson_sqlite3_xxx()` family of functions convert `sqlite3_stmt` rows |
|
310
|
to Arrays or Objects, or convert single columns to a JSON-compatible |
|
311
|
form. See `json_stmt_to_array_of_obj()`, |
|
312
|
`json_stmt_to_array_of_array()` (both in `src/json.c`), and |
|
313
|
`cson_sqlite3_column_to_value()` and friends (in |
|
314
|
`extsrc/cson_amalgamation.h`). They work in an intuitive way for numeric |
|
315
|
types, but they optimistically/naively *assume* that any fields of type |
|
316
|
TEXT or BLOB are actually UTF8 data, and treat them as such. cson's |
|
317
|
string class only handles UTF8 data and it is semantically illegal to |
|
318
|
feed them anything but UTF8. Violating this will likely result in |
|
319
|
down-stream errors (e.g. when emitting the JSON string output). **The |
|
320
|
moral of this story is:** *do not use these APIs to fetch binary data*. |
|
321
|
JSON doesn't do binary and the `cson_string` class does not |
|
322
|
protect itself against clients feeding it non-UTF8 data. |
|
323
|
|
|
324
|
Here's a basic example of using these features: |
|
325
|
|
|
326
|
```c |
|
327
|
Stmt q = empty_Stmt; |
|
328
|
cson_value * rows = NULL; |
|
329
|
db_prepare(&q, "SELECT a AS a, b AS b, c AS c FROM foo"); |
|
330
|
rows = json_stmt_to_array_of_obj( &sql, NULL ); |
|
331
|
db_finalize(&q); |
|
332
|
// side note: if db_prepare()/finalize() fail (==they exit()) |
|
333
|
// then a JSON-format error reponse will be generated. |
|
334
|
``` |
|
335
|
|
|
336
|
On success (and if there were results), `rows` is now an Array value, |
|
337
|
each entry of which contains an Object containing the fields (key/value |
|
338
|
pairs) of each row. `json_stmt_to_array_of_array()` returns each row |
|
339
|
as an Array containing the column values (with no column name |
|
340
|
information). |
|
341
|
|
|
342
|
**Note the seemingly superfluous use of the "AS" clause in the above |
|
343
|
SQL.** Having them is actually significant! If a query does *not* use AS |
|
344
|
clauses, the row names returned by the db driver *might* be different |
|
345
|
than they appear in the query (this is documented behaviour of sqlite3). |
|
346
|
Because the JSON API needs to return stable field names, we need to use |
|
347
|
AS clauses to be guaranteed that the db driver will return the column |
|
348
|
names we want. Note that the AS clause is often used to translate column |
|
349
|
names into something more JSON-conventional or user-friendly, e.g. |
|
350
|
"SELECT cap AS capabilities...". Alternately, we can convert the |
|
351
|
individual `sqlite3_stmt` column values to JSON using |
|
352
|
`cson_sqlite3_column_to_value()`, without referring directly to the |
|
353
|
db-reported column name. |
|
354
|
|