pyroid.json_repair: drop-in json_repair¶

pyroid.json_repair is a Rust port of json_repair 0.64.5 by Stefano Baccianella (MIT). json_repair turns broken JSON from LLMs into valid JSON: missing quotes, single quotes, comments, trailing commas, truncation, markdown fences, prose around it, Python literals. Its parser is ported line by line, so pyroid returns the same strings, objects and repair logs as json_repair 0.64.5 (parity). It also stays linear on 13 input families where json_repair's lookaheads are quadratic, and deep nesting never overflows the stack.

Change one import¶

-import json_repair
+import pyroid.json_repair as json_repair

 json_repair.repair_json("{'a': 1, b: [1, 2,}")    # '{"a": 1, "b": [1, 2]}'
 json_repair.loads('{"title": "Notes on the Analytical Eng')
 # {'title': 'Notes on the Analytical Eng'}

Repairing and decoding¶

repair_json, loads, load and from_file take json_repair's arguments:

Valid JSON comes back as json.dumps(json.loads(text)). Anything else goes through json_repair's heuristics.

import pyroid.json_repair as json_repair

reply = 'Sure! Here is the JSON:\n```json\n{"name": "Ada", "langs": ["en", "fr"\n```'
json_repair.repair_json(reply)  # '{"name": "Ada", "langs": ["en", "fr"]}'
json_repair.loads("{'ok': True, 'value': None, // comment\n 'n': 1,}")
# {'ok': True, 'value': None, 'n': 1}

json_repair.repair_json("[1, 2", logging=True)
# ([1, 2], [{'text': 'While parsing an array we missed the closing ], ignoring it',
#            'context': '[1, 2'}])
json_repair.repair_json("{'city': 'Zürich'", ensure_ascii=False)  # '{"city": "Zürich"}'

The options are json_repair's:

• return_objects=True returns the value instead of a string (what loads does).
• logging=True returns (value, repair_log): a list of {"text": ..., "context": ...} dicts, empty when nothing was repaired.
• skip_json_loads=True skips the json.loads validity check, so valid JSON goes through the repair parser too.
• stream_stable=True keeps the repairs of a growing, streamed prefix stable.
• strict=True raises ValueError on structural problems instead of repairing them, with json_repair's messages ("Missing ':' after key in strict mode while parsing object.").
• json_fd= reads the text from an open file (prefer load). chunk_length is accepted and ignored, because pyroid reads the file at once.
• Other keyword arguments go to json.dumps for the string result. ensure_ascii, indent, separators, sort_keys and allow_nan are reproduced in Rust; any others call json.dumps.

Lone surrogates are handled as json_repair handles them ("['a\ud800" gives '["a\\ud800"]'). Unicode-dependent checks (str.isspace, isalpha, isdigit, isalnum) follow the running interpreter's Unicode version, 14.0 to 17.0. A newer Python gets a RuntimeWarning and the 17.0 tables.

Many documents: repair_json_batch¶

repair_json_batch is pyroid's addition. It runs repair_json over a batch on pyroid's thread pool, without the GIL. Each result equals json_repair.repair_json(doc, ...) with the same skip_json_loads, stream_stable, strict and json.dumps arguments.

import pyarrow as pa
import pyroid.json_repair as json_repair

docs = ["{'a': 1", '{"b": 2}', "[1, 2,"]
fixed = json_repair.repair_json_batch(docs)  # ['{"a": 1}', '{"b": 2}', '[1, 2]']
fixed[1] is docs[1]  # True: a document that needs no change is the same object
json_repair.repair_json_batch(docs, threads=4, indent=2)  # at most 4 tasks at once

column = json_repair.repair_json_batch(pa.array(["{'a': 1", None, "[1, 2,"]))
column.to_pylist()  # ['{"a": 1}', None, '[1, 2]']

It takes a str, a list or tuple of str, or an Arrow string column (pyarrow, polars, datasets) and returns the same kind. For Arrow input that is a pyroid.Column, with nulls kept. In strict mode a failing document raises for the whole batch, and the exception's note names its index. json.dumps arguments other than ensure_ascii, indent, separators, sort_keys and allow_nan work for str, list and tuple input only (one repair_json call per document).

Deep nesting: NestingError¶

pyroid's parser follows up to MAX_DEPTH (1,000) levels of broken nesting. Deeper input raises NestingError, which is a pyroid.PyroidError, a ValueError and a RecursionError, so except clauses written for json_repair still catch it. json_repair itself raises ValueError past about 140 to 330 levels, depending on sys.getrecursionlimit() and the caller's stack.

import pyroid.json_repair as json_repair

json_repair.repair_json("[" * 1000)  # '[]' (json_repair raises ValueError)
try:
    json_repair.repair_json("[" * 1001)
except json_repair.NestingError as e:
    print(e)  # Input nesting exceeds the supported parser recursion depth.

Delegated features¶

These are handed to the real json_repair, which must then be installed. The first call emits a pyroid.FallbackWarning, and without json_repair the call raises pyroid.PyroidError:

• every schema= call, which needs jsonschema or pydantic as in json_repair (listed in pyroid.json_repair.DELEGATED);
• input that is not a str (for example bytes);
• the rare text holding a character in U+10F800..U+10FFFF, raw or as an escaped surrogate pair, because the port uses that range to store lone surrogates. In repair_json_batch, such documents in a list or tuple are delegated one by one; Arrow input raises PyroidError instead.

json_repair.repair_json(b"{'a': 1")  # json_repair's result, and:
# FallbackWarning: pyroid.json_repair.repair_json: a bytes input is not supported by the Rust
# port; delegating to json_repair (slow path). This warning is shown once.

json_repair's command line (python -m json_repair) and its internal modules are not provided.

Pathological input¶

json_repair has lookaheads and string copies that make some inputs quadratic. The port keeps every heuristic but memoizes those scans, and stays linear on all 13 super-linear input families found (json_repair's log-log slopes are 1.5 to 2.0). The table shows a few of them, at sizes doubling from 16 KiB to 1 MiB, with times at the largest size json_repair finished. CPython 3.12, on a Docker/OrbStack VM on Apple Silicon, indicative:

These families come from reading json_repair's code. Fuzzing found one more that is quadratic in both: a 62-character run of garbage, repeated, takes pyroid 8, 32, 127 and 511 ms at 7,936, 15,872, 31,744 and 63,488 characters, and json_repair 4.6 s at 15,872 and 18.4 s at 31,744 (Docker/OrbStack VM, indicative).

Differences from json_repair¶

From the module's documentation (pyroid/json_repair.pyi):

• json_repair raises ValueError past about 140-330 levels of broken input (it depends on sys.getrecursionlimit() and the caller's stack). pyroid repairs up to MAX_DEPTH levels, and deeper broken input raises NestingError, which is both a ValueError and a RecursionError.
• Valid JSON is decoded by json.loads for objects, as in json_repair, so its RecursionError past about 10,000 levels on 3.12 is the same. For string results pyroid decodes it at any depth (json_repair raises there too).
• load/from_file read the file once (fd.read()) instead of in chunk_length pieces. The result is the same, chunk_length is ignored and unseekable streams work.
• json_repair's command line (python -m json_repair) is not part of this module.

And by design, the delegated features need json_repair installed.

Parity¶

There were 0 differences in 184,403 comparisons on CPython 3.12 and 184,254 on free-threaded 3.15t, where the schema= replays are skipped for lack of jsonschema. The suite replays json_repair's own test suite as 672 recorded calls, 20,000 seeded broken LLM outputs under 8 option sets, Hypothesis grammar fuzzing with lone surrogates under 9 option sets, and load and from_file. Details: Compatibility.

Speed¶

These timings come from a Docker/OrbStack VM on Apple Silicon, indicative. Single documents, CPython 3.12, best time per call (3.15t is within ±10%):

On valid JSON there is little to gain. Both libraries start with CPython's json.loads, so loads of valid JSON costs the same. The speedup is on broken JSON, where json_repair's parser runs in Python.

For batches, 10,000 broken documents of about 1.2 KB on free-threaded CPython 3.15t (GIL off), on the same VM:

The 8-thread run is limited by the shared, loaded VM. See also Benchmarks.

API: pyroid.json_repair reference.