Function calling LLMs: a schema won't secure your agent

TL;DR: Function calling lets a model ask your code to act. Strict schemas check the shape of a call, not whether it's a good idea. Validate on the server, give each tool least access, gate big actions behind approval, fail closed and keep replayable audit logs.

A language model can't do anything on its own. It writes text. Function calling is how you let it ask for something to happen, like a database lookup, a calendar booking or a payment.

Here's the catch. The model only asks. Your code does the work. So the safety of the whole thing sits in what your code does with that ask.

Here's the short version. Strict schemas check the shape of a call. They don't check whether the call is a good idea. You need checks around it.

How the loop works

Most vendors follow the same contract.

1. You send the model a list of functions, each with a name, a description and a parameter schema.
2. The model replies with the function it wants and the arguments.
3. Your app checks the call and runs it.
4. You send the result back to the model.
5. The model writes its final answer using that result.

Most APIs add a few controls. You can let the model decide, force it to call something, or switch tools off. Strict modes force the arguments to match your schema. Parallel calls let the model ask for several independent things in one turn. Keep anything with dependencies in separate turns.

When to use it

Use function calling when the model needs to touch something outside itself: a database, a calendar, a payment API. Skip it for plain question answering, or for tiny tasks where a tool round trip costs more than it returns.

If all you need is a reply in a fixed format, with nothing to run, use structured outputs instead.

There are two common agent patterns, covered in the OWASP agentic security guide and a survey of agentic tool use:

• Think, act, repeat. The model reasons, calls a tool, reads the result and loops. It's flexible, and harder to keep inside limits.
• Plan, then execute. One step drafts a plan and workers carry it out. It's easier to audit and to gate.

Voice agents are a good fit, because they call tools in the middle of a live conversation. 42Voice builds voice agents, and latency and call approval show up fast when you test them.

Test the no-call cases too. Whether the model should call a tool at all matters as much as whether the call is well formed.

The controls that go before execution

Treat the model's output as a request, never a command. Microsoft's agent safety guidance points out that data crosses several trust boundaries, including the function tools themselves. Each one is a place to check.

Put these in place:

1. Validate every argument. Check type, range and allowed values. Use allow-lists, not block-lists.
2. Give each tool the least access it needs. Small, single-purpose tools beat open-ended ones.
3. Add a policy layer. For every call, it returns allow, deny or ask for approval. Microsoft's post on a control plane for agent tool execution explains why a tool-discovery protocol alone gives you no checkpoint.
4. Gate high-impact actions behind a person. Tie the approval to the exact call, give it an expiry, and use it once only. Our post on human in the loop covers when a person belongs there.
5. Clean the output before you show it or run it.
6. Fail closed. If a tool is unknown or the policy check errors, stop.
7. Keep audit logs you can replay, without personal data in them.

See also our guides on LLM guardrails and chatbot security.

A build checklist

• Declare JSON schemas and turn on strict mode where the API has it.
• Validate on the server anyway. Strict mode doesn't replace your checks.
• Use parameterised queries. Never build SQL or shell commands by joining strings.
• Scope credentials tightly. Use read-only access where reads are enough.
• Set rate limits, a maximum chain depth and a cost cap, so one runaway loop can't drain your budget.
• Reject sensitive names like passwords or tokens inside tool arguments. The OWASP AI agent security cheat sheet shows a validator for this.
• Check code at review time too. Veridical does evidence-based pull request review that can catch unsafe argument handling before merge.

For the wider build, read our step-by-step guide to building an AI agent. If you're comparing ways to connect tools, the group has a piece on MCP vs API, and a guide to AI agent guardrails.

Test before you launch

Agents fail in ways a demo never shows. Run these first:

1. Tool failures. Timeouts and partial results. Does the agent retry or stop, or does it make up an answer?
2. Attacks. Prompt injection and malicious tool output.
3. Approval races. Check each approval is used exactly once.
4. Runaway cost. Simulate a loop that keeps calling an expensive tool.
5. Replay. Can you rebuild a bad run from the logs?

Where teams slip

The popular advice is to switch on the strictest schema mode and call it secure. It isn't. Strict mode guarantees shape, not intent. A perfectly formed call to delete the wrong record is still a perfectly formed call.

The gap is usually the layer between "the model asked" and "the system did". Nobody owns it. So build the policy check, the approval path and the logging as part of the project, not as a clean-up later.

If you want a team to build this with you, our AI platform development work covers agents that connect to your systems. AI app development fits when you want function calling inside one product. Tell us what the agent needs to do and which systems it touches, and we'll map the controls first. Start at our services page.

Frequently asked questions

What is function calling in LLMs?

The model returns a structured request naming a function and its arguments. Your app decides whether to run it, runs it, and sends the result back. The model never executes anything itself.

What is the difference between function calling and structured outputs?

Use function calling when the model needs to reach functions in your application. Use structured outputs when you only need the reply in a fixed format and nothing needs to run.

Does strict mode make function calling safe?

No. Strict mode makes arguments match your schema, but says nothing about whether the call is appropriate. You still need server-side checks, least privilege and approval for high-impact actions.

Is function calling the same as an agent?

No. Function calling is the call mechanism. An agent adds a loop, planning and state around it. An agent can use function calling, but one function call isn't an agent.