Red Flags When Hiring an AI Agency

TL;DR: Most businesses picking an AI agency get burned by the same mistakes. If an agency can't show you working AI they built, give you a real cost estimate, or name the tech they use, walk away. These red flags are easy to spot before you sign anything.

Most AI agencies are consultants with a ChatGPT subscription and a good pitch deck. A few actually build. Knowing the difference before you sign a contract saves you months and serious money.

Here are the red flags to look for when you're hiring an AI agency.

Can they show you AI they actually built?

This is the first question to ask. Not case studies. Not slides. Working software.

Ask to see a live product, a demo, or a client they can put you in touch with. A real AI agency has a portfolio of shipped work. Devwiz has built 200+ apps since 2015, including AI platforms for clients like NSW Government, Briometrix, Vivid, and Huskee.

If the agency you're talking to responds with a vague "we've done similar projects" or sends you a PDF full of frameworks and process diagrams, that's your answer. They consult on AI. They don't build it.

Ask directly: what did you ship, who for, and what can I see?

Do they talk about tech or just outcomes?

Consultants talk about outcomes. Builders talk about tech.

A genuine AI agency should be able to tell you straight away what stack they'd use for your project. Are they building a custom model, fine-tuning an existing one, or connecting APIs? Are they using RAG, vector databases, or agentic workflows? Do they have a view on when to use OpenAI versus open-source alternatives?

If the answer is a lot of "it depends" and "we'd need to scope that," that can be fine early on. But if they can't give you a rough technical direction after a 30-minute call, they're figuring it out as they go.

You want a team with opinions. Opinions come from experience.

Are they being straight about cost?

AI projects have a reputation for blowing budgets. Some of that is unavoidable complexity. A lot of it is agencies quoting low to win the work, then expanding scope mid-project.

Before you sign anything, read our guide on what it costs to build an AI app in Australia so you go in with realistic numbers. If an agency quotes you something dramatically lower than the market, ask why. Either they're cutting something important, or they'll be back with change requests.

Watch for:

• Quotes with no line-item breakdown
• Vague milestones like "Phase 1: Discovery" with no deliverables attached
• Contracts that define success as hours billed, not outcomes delivered
• No clear ownership of IP at project end

A good agency will give you a honest number, explain what drives the cost, and tell you what they'd cut if you needed to reduce scope.

Do they have a team or is it just one person?

A lot of "AI agencies" are solo operators or small freelance collectives. That's not automatically bad. But it matters for your project.

Ask who specifically will be doing the work. Is there a product lead, a data engineer, a frontend developer? Or does one person handle everything? If the answer is a team, ask to meet them.

Single-person operations tend to hit a ceiling fast on complex AI builds. They're also a risk: if your key person gets sick or takes another project, your build stalls.

For businesses building AI platforms and programs, you need a team with genuine depth across the stack, not one person who knows a bit of everything.

Are they learning on your budget?

This is harder to spot, but worth probing. Ask how many AI projects they've shipped in the last 12 months. Ask what went wrong on one of them and how they handled it. Ask what AI they use internally in their own business.

An agency that can't answer those questions honestly is probably still figuring out AI. Which means you're paying for their education.

A team that's genuinely embedded in this space will have opinions on what's changed in the last six months, what tools they've dropped and why, and where they've made mistakes. They'll also stay current. The AI space moves fast. An agency that was sharp 18 months ago but hasn't kept up is a risk.

We use Njin internally to run client delivery programs, including AI builds. If an agency doesn't use AI in their own operations, that tells you something.

Are they building something custom or stitching together tools?

There's nothing wrong with using existing AI tools. Most good builds do. But you need to know what you're paying for.

If an agency is charging you enterprise rates to connect a few APIs and wrap them in a UI, that's not an AI build. That's integration work. It might still be worth doing, but the price point should reflect it.

A real AI development partner will be clear about what's custom versus what's off-the-shelf. They'll explain the trade-offs. And they won't charge you premium rates for commodity work.

Ask specifically: what are you building from scratch, what are you integrating, and why?

Are the contracts one-sided?

Contracts in agency work are often written to protect the agency. That's normal. But some are genuinely unfair.

Things to check:

• Who owns the IP? You should own everything you paid to build.
• What happens if you want to leave mid-project? Is there a clear handoff process?
• Are there penalties if they miss milestones?
• What does the support and maintenance commitment look like post-launch?

If the agency resists reasonable questions about IP ownership or gets vague about what happens if the relationship ends, that's a flag.

FAQ

How do I know if an AI agency actually builds or just consults?

Ask to see working software they've shipped. A real AI agency can show you live products or connect you with clients. If the response is slides, frameworks, or vague references to past projects without specifics, they're consultants. Ask: what did you build, who for, and can I see it?

What questions should I ask when hiring an AI agency?

Start with: what tech would you use for this project and why? What have you shipped in the last 12 months? Who specifically will do the work? What does the IP ownership clause look like? A genuine builder will answer these clearly. Vague answers to technical questions are a warning sign.

Why do AI agency projects go over budget?

Most budget blowouts come from vague scoping upfront, agencies quoting low to win work and adding scope later, or underestimating the complexity of data preparation and integration. Read the actual line-item breakdown before you sign, and agree on what success looks like in measurable terms, not just hours billed.

Is it okay to hire a small AI agency or solo operator?

Sometimes, depending on scope. Solo operators can be fast and cost-effective for contained builds. The risk is capacity and continuity. For anything complex, or where the AI is core to your product, you want a team with depth across product, engineering, and data. Ask who's doing what and what happens if your key person is unavailable.

How do I check if an AI agency is up to date with the current state of AI?

Ask them what's changed in their approach in the last six months and what tools they've dropped and why. Ask what AI they use internally. An agency that's genuinely current will have strong opinions and recent examples. One that's coasting on past knowledge will give you generic answers about LLMs and automation.

Frequently asked questions

How do I know if an AI agency actually builds or just consults?

Ask to see working software they've shipped. A real AI agency can show you live products or connect you with clients. If the response is slides, frameworks, or vague references to past projects without specifics, they're consultants. Ask: what did you build, who for, and can I see it?

What questions should I ask when hiring an AI agency?

Start with: what tech would you use for this project and why? What have you shipped in the last 12 months? Who specifically will do the work? What does the IP ownership clause look like? A genuine builder will answer these clearly. Vague answers to technical questions are a warning sign.

Why do AI agency projects go over budget?

Most budget blowouts come from vague scoping upfront, agencies quoting low to win work and adding scope later, or underestimating the complexity of data preparation and integration. Read the actual line-item breakdown before you sign, and agree on what success looks like in measurable terms, not just hours billed.

Is it okay to hire a small AI agency or solo operator?

Sometimes, depending on scope. Solo operators can be fast and cost-effective for contained builds. The risk is capacity and continuity. For anything complex, or where the AI is core to your product, you want a team with depth across product, engineering, and data. Ask who's doing what and what happens if your key person is unavailable.

How do I check if an AI agency is up to date with the current state of AI?

Ask them what's changed in their approach in the last six months and what tools they've dropped and why. Ask what AI they use internally. An agency that's genuinely current will have strong opinions and recent examples. One that's coasting on past knowledge will give you generic answers about LLMs and automation.