AI platform architecture: a layered guide for CTOs

TL;DR: An AI platform is three layers doing three jobs: data and knowledge, model and serving, orchestration and app. Three planes cut across all of them: watching, rules, and model lifecycle. Build the data layer first, because no model fixes bad data. Design governance in from the start, because retrofitting the Privacy Act is expensive.

Most AI platform diagrams are useless. Boxes, arrows, and nothing you can act on come Monday.

Here's the thing. An AI platform is three layers doing three jobs, with three things running across all of them. Get that shape right and the rest is detail. Get it wrong and you rebuild in year two.

This guide covers the layers, what to build first, what drives the cost, and the Australian rules you cannot skip.

What an AI platform actually looks like

Three runtime layers. Top to bottom.

Three planes cut across all three.

• Watching. Traces, speed, cost, and whether the answers are still good.
• Rules. Who sees what, where data sits, and what got logged.
• Model lifecycle. Which model is live, what replaced it, when it retires.

A request runs like this. It hits the gateway. The gateway checks who is asking and picks a model. The orchestrator pulls context from the data layer, calls the model, runs any tools, and sends back an answer. Every hop gets logged.

That is the whole shape. Microsoft's Well-Architected guidance for AI workloads splits it the same way: the app users touch, the training work behind it, and the platform services in the middle.

If you want the product-side view of the same problem, we wrote about AI-first product architecture separately.

The data layer decides how good your AI is

Teams spend months picking models. Then they find out the data was the problem all along.

No model fixes bad data. Not a bigger one, not a newer one. So build this layer first.

Land your data in three stages:

• Raw. Exactly as it arrived. Never edited. Keep the source and the timestamp.
• Clean. Deduped, checked against a schema, every change logged.
• Ready. The tables your models actually train and run on.

Then two more pieces sit on top.

A feature store. Something like Feast serves the same feature two ways: fast lookups when the model is running, and big batches when it is training. The trap is letting those two drift apart. Train on one version of a feature, serve another, and the model quietly gets worse. Nothing breaks. Nothing alerts. It just gets worse.

A vector index. This holds the documents your models read from. Stale index, wrong answers. Version it alongside your models so a rollback rolls back both.

One habit worth building early. Put a content hash and a source document ID on every chunk you store. When a document changes you can re-embed just that one, and you can prove where any answer came from.

How a model earns its way into production

A pile of notebooks is not a platform. The difference is a gate.

Every model passes the same checks before it serves a single real user:

1. Tests pass on the training code and the feature logic.
2. It beats the current model on a held-back test set.
3. Bias checks pass across the groups you defined.
4. No known security holes in the serving container.
5. Re-running the training gives you the same numbers, near enough.
6. The old model stays in the registry, ready to swap back.
7. Someone signed off, and the sign-off is recorded.

Track every run. Parameters, metrics, which dataset version, which container image. MLflow does this and gives you the promotion states to go with it.

Pin your image digests and dataset snapshots, not just their names. A name points somewhere different next month.

Orchestration is where AI becomes a product

This layer takes a job, breaks it up, and gets it done.

The pattern that holds up is one orchestrator and many workers. The orchestrator takes the task, splits it, and hands pieces to workers. Workers do one thing each and stay easy to test. The orchestrator deals with the mess when a step fails.

Memory works best in two tiers. Fast cache for what happened this session. Vector store for long-term knowledge. Never load the whole store into the prompt. Pull the top few matches, set a hard token budget, and your speed stays predictable.

Put a gateway in front of it. The gateway handles auth, rate limits, and which model gets the call based on cost, speed and where the data is allowed to live. Your app calls one endpoint and never has to care.

Most of the work here is not the AI. It is wiring the platform into the systems you already run. The AI Orchestrators team covers that ground well in their piece on AI system integration.

Google's guidance on choosing agentic architecture components makes a fair point: pick your runtime, memory and tools as you learn, not upfront. Locking in one framework on day one buys you a migration later.

We built exactly this shape for a multi-tenant white-label AI SaaS, where routing across several model providers keeps the client off any single vendor.

Three ways to serve a model

Most teams want the middle option and talk themselves into one of the others.

A few things that cut cost fast:

• Cache near-duplicate questions. Same question, same answer, no model call.
• Batch small requests together. Better GPU use for the same money.
• Expire cached answers when the source documents change. Otherwise you serve yesterday's answer with confidence.
• Set timeouts and retries at the gateway. One slow model should not take the whole platform down.
• Alert on p99, not average. The average always looks fine.

Cost creeps up here more than anywhere else. We broke down where the money actually goes in what is software infrastructure cost.

The Australian rules you cannot skip

Governance is not a phase at the end. It is a plane that touches every layer, and it carries legal weight here.

The Privacy Act 1988 sets the floor. For an AI platform that means:

• Collect less. Only the personal data the model genuinely needs, for a purpose you wrote down.
• Know where it goes. Sending personal data to a provider overseas triggers Australian Privacy Principle 8. Confirm the protections or get consent.
• Be able to find it. People can ask for their data. You need a way to pull it from every store you own.
• Report breaches. Serious ones go to the OAIC and to the people affected.
• Treat outputs as data too. If a model writes something that identifies a person, that output is personal information.

Practical controls that make the above possible:

• Classify data at the moment it lands, not later.
• Mask or tokenise personal data the model does not need in the raw.
• Log every access with who, when and why.
• Log the model version, the prompt, the retrieved context and the answer for every call.
• Read the data processing agreement of every provider in your stack.

Both major Australian cloud regions sit in Sydney and Melbourne. Not every service lands in both at the same time, so check before you commit to a design.

What to watch once it is live

Server uptime tells you nothing about whether the model still works.

Watch these:

Set two alerts that matter. One when inputs drift past your threshold. One when quality drops below the bar the model passed at promotion. Both should kick off a retrain, and the new model goes through the same gate as everything else.

Give every request one correlation ID and carry it through every layer. When something breaks you want to trace it, not guess.

Picking the right shape for your team

The best architecture is the one that fits your real constraints.

• Need answers under 100ms? Containers on GPU nodes with caching. Serverless will not get there.
• Data has to stay in Australia? Australian region hosting, and residency written into every provider contract.
• Board wants a fixed monthly number? Reserved pricing on containers beats pay-per-token at unknown volume.
• Small team, light on ops? Provider APIs behind a managed gateway. Self-hosting can wait.
• Regulated sector? Compliance drives the tool choice. Not the other way around.

Off-the-shelf works fine for standard jobs. Build custom when you need your own data pipelines, your own fine-tuning, real tenant isolation, or deep hooks into systems you already run. Our friends at AI-Led wrote a good breakdown of build versus buy for AI if you are still weighing it up.

Worth asking any provider before you sign:

• What is your uptime SLA in Australian regions, and what happens when you miss it?
• How much notice do you give before retiring a model?
• Can you show data lineage from source to output for an auditor?
• What telemetry can we export into our own monitoring?

A build order that works

1. Weeks 1 to 4. Scope it. Map the data you have, pick the use cases, write down the architecture decision. Output: a data readiness check.
2. Weeks 5 to 10. Prove it small. Raw and clean data tiers, a basic feature store, one model endpoint. Gate: is the data good enough for the job?
3. Weeks 11 to 20. Pilot. Add orchestration and monitoring. Limited user group. Gate: do speed, quality and drift meet the numbers you set?
4. Weeks 21 to 36. Roll out. Full user base in stages. Remaining controls, audit logging, sign-offs. Gate: legal and security are happy.
5. Weeks 37 to 52. Tune. Caching, smaller models, batching, storage lifecycle. Output: a real cost per answer.

Then it never stops. Quarterly reviews. Models approaching retirement. Dependencies with holes. Rules that changed.

The big cost drivers, in order: GPU hours, data engineering time, storage growth, log ingestion, and provider tokens. Most teams budget for the first and get surprised by the rest.

What we see go wrong

Three patterns, over and over.

The data layer is underbuilt. Teams argue about models for months, then find training and serving features do not match, or the vector index is weeks stale. The model cannot fix that.

Governance gets left to the end. The platform demos beautifully and cannot ship, because audit logs, residency and access control were never designed in. Retrofitting that is expensive.

Orchestration looks simpler than it is. A prototype with three happy-path steps turns brittle the moment a tool call fails or a model returns a shape you did not expect. Explicit error handling is not over-building. It is the thing that makes agents safe to put in front of real users.

Now here's the important bit. Before you commit to any design, audit your data quality and your logging. Architecture can be changed later. Bad data and missing audit trails are much harder to undo.

Where to start

Three checks you can run this week:

1. Does your feature store serve the same values at training and at inference?
2. Does a model need to pass anything before it reaches production?
3. Does your gateway route by cost, speed and data residency, or does everything hit one endpoint?

If any of those is a no, that is your next piece of work.

Devwiz has shipped 200+ apps and platforms. That includes work for the NSW Government, Briometrix, Vivid and Huskee, plus CARED, a national allied health platform built on the Microsoft stack with four mobile apps and three web platforms behind it.

If you are designing a platform and want it built properly, talk to us about AI app development or an AI program. CTOs weighing up the architecture call can start on tech for CTOs. If you are earlier than that and still picking tools, read how to pick your AI stack first.

Get the layers right before you get clever. Clever is cheap. Rebuilding is not.

Frequently asked questions

What is AI platform architecture?

It is the layered design of the systems that run AI in production. Three runtime layers do the work: a data and knowledge layer holding features, documents and memory; a model and serving layer running and versioning the models; and an orchestration layer that turns a request into steps, tool calls and an answer. Three planes cut across all three: observability, governance, and model lifecycle.

What should you build first in an AI platform?

The data layer. Teams routinely spend months choosing models and then find the real problem was inconsistent features or a stale vector index. No model compensates for bad data. Land your data in three stages (raw, clean, ready), get a feature store serving the same values at training and at inference, and version your vector index alongside your models.

Should we build a custom AI platform or use an off-the-shelf one?

Off-the-shelf suits standard use cases, limited customisation and small teams. Build custom when you need your own data pipelines, proprietary fine-tuning, real multi-tenant isolation, or deep integration with systems you already run such as a CRM, ERP or data warehouse.

Where does AI platform data have to be stored in Australia?

The Privacy Act 1988 sets the baseline. Collect only the personal data the model needs, and if training or inference sends personal data to a provider outside Australia, Australian Privacy Principle 8 applies: confirm comparable protections or get consent. Model outputs that identify a person are personal information too, and carry the same obligations.

How long does it take to build a production AI platform?

A realistic phased build runs about a year. Roughly four weeks to scope, six to prove it small, ten to pilot with real users, sixteen to roll out with full governance, and the rest to tune cost and speed. Each phase ends in a go or no-go gate, so you can stop before committing to full-scale infrastructure.