Tool Use & Function Calling

Tool use (also called function calling) is the mechanism by which a language model, instead of answering a query directly from its parameters, produces a structured description of an external operation it wants performed. That description — conventionally a function name and a set of arguments, usually serialized as JSON — is intercepted by a runtime, executed against real code (an API, a database query, a calculator, a search index, a Python interpreter), and the result is inserted back into the model's context so it can compose a final, grounded answer. The core object here is the tool schema: a specification of a callable's name, a natural-language description of what it does and when to use it, and a typed parameter list (names, types, and whether each is required). This is typically expressed in JSON Schema. The model is given the set of available schemas — either concatenated into the system prompt or passed through a dedicated API field — and is trained or instructed to choose zero or one (or, in parallel-call variants, several) tools per turn, filling in arguments consistent with the schema and the user's request. Two properties distinguish tool use from ordinary generation. First, its output must be machine-parseable: a human reading prose can tolerate ambiguity, but a JSON parser cannot. Second, its correctness is externally checkable: unlike a free-text answer, an arguments object can be validated against a schema (right types, required fields present) before execution even begins, giving an early, cheap error-detection point absent from ordinary text generation.

The mechanism has four moving parts: tool description, structured decoding, execution, and result grounding, orchestrated in a loop.

1. Tool description. Every available function is described with a name, a description string, and a parameter schema. This description is injected into the model's input, either as a special "tools" field (in APIs that support it natively) or serialized into the prompt text. The model's job during training or fine-tuning is to learn the mapping from "user intent that matches this description" to "emit a call to this name with these arguments."

2. Decision + generation. On each turn the model must decide: does answering this require a tool at all, and if so, which one, with what arguments? This is a joint decision made through ordinary autoregressive decoding, but the target format is constrained — typically a special token or tag (e.g., a `` marker, or a `finish_reason: "tool_calls"` API signal) followed by a JSON object.

3. Parsing and validation. The runtime extracts the JSON, validates it against the declared schema (required keys present, types match, enum values legal). If validation fails, the runtime can either reject and re-prompt the model with the error, or attempt a repair (re-ask with the parse error appended as context).

4. Execution and re-injection. The runtime calls the actual function with the validated arguments, captures its return value (or exception), and appends it to the conversation as a new message — usually with a special role like `tool` — so the model's next generation step is conditioned on the real result rather than a hallucinated guess.

5. Loop or terminate. If the model's task requires further tool calls (chained retrieval, multi-step computation), the loop repeats; otherwise the model emits a final natural-language answer, marked with a normal `assistant`/stop signal instead of a tool-call signal. ``` function TOOL_AUGMENTED_GENERATE(user_query, tool_registry, max_rounds):

# tool_registry: map from tool_name -> {description, json_schema, python_callable} messages = [ {role: "system", content: SERIALIZE_TOOLS(tool_registry)}, {role: "user", content: user_query} ] for round in 1 .. max_rounds: response = MODEL_DECODE(messages) # may emit prose OR a structured tool call if response.type == "final_answer": return response.text # base case: model is done, no tool needed if response.type == "tool_call": call = PARSE_JSON(response.raw_text) # may raise ParseError if call is ParseError: messages.append({role: "tool", content: "ERROR: malformed JSON: " + call.message}) continue # give model a chance to self-correct schema = tool_registry[call.name].json_schema valid, err = VALIDATE_AGAINST_SCHEMA(call.arguments, schema) if not valid: messages.append({role: "tool", content: "ERROR: " + err}) continue # re-prompt with the validation error try: result = tool_registry[call.name].python_callable(call.arguments) except Exception as e: result = "ERROR: execution failed: " + str(e) messages.append({role: "assistant", content: response.raw_text}) # the call itself messages.append({role: "tool", content: SERIALIZE(result)}) # the grounding fact return "ERROR: exceeded max_rounds without final answer" # edge case: runaway loop ``` Two engineering choices sharpen reliability. Constrained decoding restricts the token sampler at each step to only tokens that keep the output a valid instance of the JSON Schema (via a grammar or finite automaton over the schema), making malformed JSON structurally impossible rather than merely unlikely. Two-stage validation (schema check before execution, exception handling during execution) means failures are caught at the cheapest possible point instead of silently propagating a wrong answer.

Per-round cost. One round of the loop costs one model forward pass over the accumulated context of length $L$, which under a standard transformer is $O(L^2 \cdot d)$ time with self-attention (or $O(L \cdot d)$ amortized per generated token with KV-caching across $O(L)$ generated tokens, so $O(L^2 d)$ total for the round dominated by attention over growing context), plus $O(1)$ amortized cost for parsing and schema validation (JSON parsing is linear in the size of the call, which is small and bounded relative to $L$), plus the cost of the external tool call itself, which is outside the model's control and can range from microseconds (a calculator) to seconds (a network API).

Loop cost. If a task requires $k$ sequential tool calls (a chain — e.g., look up a city, then look up its weather), total cost is $\sum_{i=1}^{k} O(L_i^2 d)$ where $L_i$ grows by roughly the size of each call-and-result pair, since every prior round's messages remain in context. In the worst case, with no context truncation, this is $O(k \cdot L_{\max}^2 \cdot d)$ where $L_{\max}$ is the final context length — quadratic accumulation is the practical reason production agents cap `max_rounds` and periodically summarize or truncate old tool outputs.

Space. Context memory is $O(L)$ tokens (dominated by the growing message list plus the KV-cache, which is $O(L \cdot d \cdot \text{layers})$ for a standard transformer), which is the binding constraint on how many rounds a single conversation can sustain before context-window limits force truncation or summarization.

Correctness argument. The mechanism's correctness rests on a simple invariant maintained across the loop: > Invariant: at the start of every round, every fact currently in the model's context that resulted from a tool call is one the runtime actually executed and verified — never a fabricated or unexecuted call. This invariant is established by construction: the runtime only appends a `tool` message after it has (a) successfully parsed the call, (b) validated arguments against the schema, and (c) executed the callable and captured a real return value or a real exception. It is preserved across rounds because every subsequent round's context is built strictly by appending to a history that already satisfies the invariant — nothing is retroactively altered. The invariant does not guarantee the model's final answer is correct — that depends on the model correctly reasoning over true facts, which is a separate (harder, unverifiable) property of the generation policy, not of the tool-calling protocol itself. What the protocol does guarantee is that any factual claim attributable to a tool call is grounded in an actual execution, which eliminates one entire class of hallucination (fabricated tool outputs) while leaving another class open (misinterpreting correct tool outputs, or hallucinating a plausible-looking call the model was never actually able to execute correctly — e.g., calling `get_weather(city="Nairobi")` when the user meant a different Nairobi).

Honesty about tradeoffs. Constrained decoding guarantees syntactic validity (the invariant's precondition (b) is always satisfiable) but does not guarantee semantic validity — a syntactically perfect call can still have the wrong city, wrong units, or wrong function chosen entirely. Schema validation catches type errors cheaply but cannot catch a subtly wrong argument value that happens to have the right type (e.g., `"NY"` vs `"New York, NY"`). And the loop's termination is not automatically guaranteed — an unbounded model could call tools forever chasing a task it cannot complete, which is why `max_rounds` is a hard architectural safeguard, not merely a tuning parameter.

Consider the query: "Convert 100 USD to EUR and tell me if it's enough for a €85 hotel room." The system has two tools registered: `get_exchange_rate(base: string, target: string) -> float` and `calculator(expression: string) -> float`.

This trace is deliberately non-trivial: round 3 shows the validator catching a realistic model error (a typoed argument key) before any execution occurs, and the runtime feeding the precise error back rather than either crashing or silently guessing. Round 4 shows self-correction succeeding on the very next attempt because the error message pinpointed the exact schema violation. The final answer is grounded in two real numeric facts (`0.92`, `7.0`) that the model never computed itself — it composed language around numbers it did not generate, which is exactly the intended division of labor. Notice also what would happen without tool use: a model asked to do "100 * 0.92" and "92 - 85" purely by autoregressive arithmetic is prone to silent errors on multi-digit multiplication, especially with more decimal places or larger numbers — a well-documented weakness distinct from, but related to, why models struggle with precise multi-step symbolic tasks. The tool-calling protocol sidesteps this entirely by delegating the arithmetic to code, which is not merely "more reliable" but categorically exact.

Structured tool use in language models grew out of two converging threads: decades of research on semantic parsing and API grounding in classical NLP, and the sudden capability jump of large instruction-tuned transformers that could follow complex formatting instructions reliably enough for structured output to become practical at scale.

Production customer-support assistants use function calling to check order status, process refunds, and update account details by invoking internal company APIs rather than merely describing what a human agent would need to do. Coding assistants use it to run test suites, execute shell commands, and query a codebase's symbol index, turning a text model into something that can verify its own suggestions rather than merely emit plausible-looking code. Enterprise data assistants use it to translate natural-language business questions into database queries, letting non-technical users interact with structured data through conversation. Consumer assistants use it for real-time information — weather, stock prices, restaurant reservations, flight status — none of which can live inside frozen model weights. Search-augmented chat products use a tool call to a search index as their primary grounding mechanism against hallucination, feeding retrieved documents back as tool results before composing an answer, which connects directly to Prompting & Prompt Engineering techniques for controlling how retrieved context is incorporated. Every major LLM API provider now exposes a dedicated tools/functions field precisely because this pattern is so load-bearing across products; it is the mechanism through which Instruction Tuning translates into deployable, action-taking systems, and the single-call loop described here is the atomic unit that, chained and given persistent memory and planning, becomes the subject of LLM Agents.

Ask the tutor: How does schema-constrained (grammar-based) decoding actually get implemented at the token level, and what does it cost in latency?