The Challenge of Maintaining Grammar Parsers for Open Models

Working with closed-source models is straightforward: you simply provide a list of functions to the API, and the model returns structured JSON without you needing to worry about the wire format. However, open models present a different challenge, as tool calling relies on a specific wire format that the engine must understand. If the engine doesn't support the model's format, the output can be a mess, with reasoning tokens in arguments, malformed JSON, and missing tool calls. This forces developers to either wait for support or create their own parsers.

Each model family encodes tool calls in its unique way, leading to incompatible wire formats with different token vocabularies and argument serialization schemes. To convert model output into a clean API response, developers often write custom parsers for each model, which is only part of the implementation challenge. For instance, Gemma 4's unique format led to issues like reasoning tokens being stripped before parsing and content leaking into tool-call arguments, requiring dedicated parser implementations.

Generic parsers, designed to handle all formats, struggle because wire formats are determined during training and are not bound to a shared convention. This open-ended nature means generic parsers can handle common cases but not the specific, hard-to-solve bugs that arise with each model's unique format. The same model-specific knowledge is needed both during generation and when parsing results, which is where grammar engines come into play.

Currently, when a new model is released, grammar engines and output parsers work independently to understand the model's format. They need the same knowledge about boundary tokens, argument serialization, and reasoning token behavior. However, this knowledge is reverse-engineered separately by different teams, leading to redundant efforts across the ecosystem.

The solution lies in creating a shared, declarative specification for wire formats that both grammar engines and parsers can use. This would eliminate the need for each team to independently reverse-engineer model formats, allowing them to update a shared spec instead. Such a separation would streamline the process and reduce the redundant work triggered by each new model release.