Multimodal AI apps: how they work and how to ship one

TL;DR: Multimodal AI apps take text, images, audio and video and give one answer. The model is the easy part. Below: how the models work, three ways to build the pipeline, how to route by task, what to test before launch, and the Australian privacy rules to plan for.

A multimodal AI app takes more than text. Photos. Voice. Video. It reads them together and gives you one answer.

Here's the short version. The model is the easy part. Speed, cost and privacy are the hard parts. Get the design wrong and the app is slow. Or it costs too much to run. Or a customer's face ends up in a log file.

Three calls matter most:

1. Pick your pipeline. Turn each image into text first, or search the images as images. Each choice has a price.
2. Route by task. Send easy jobs to a small model. Save the big one for hard jobs.
3. Plan for privacy early. A photo or a voice clip can be personal information. Treat it that way from day one.

What multimodal AI is

A plain LLM takes text in and gives text out. A multimodal model takes more than one kind of input.

Australia's digital regulators call them multimodal foundation models. Their working paper describes a type of generative AI that can process and output more than one data type, such as text, images and audio. It lists DALL-E 3 and GPT-4o among the examples.

The inputs you'll work with most are text, images, audio and video. Some builds add 3D scans from sensors too.

What does that look like in a real app?

• A tech points a phone at a broken machine and gets talked through the fix.
• A finance team snaps an invoice and the fields fill in.
• A shopper takes a photo of a chair and sees others like it.

One system. Two or more kinds of input. One answer. That's the test.

Only need voice? Start with AI voice agents for business.

How the models work

Two ideas cover most of what you'll ship.

Vision-language models. A VLM links images and text. It reads a picture and writes about it. Most document readers and visual search tools sit on one.

Embeddings. An embedding turns a piece of content into a list of numbers. Things that mean the same sit close together. So the words "red leather armchair" land near a photo of one. That's how a text search finds a picture. We cover the basics in embeddings explained simply.

Put the two together and you get multimodal RAG. The app finds the right images and text first. Then the model writes its answer from them. New to RAG? Read what RAG is, then come back.

There's one more choice. When do the inputs get mixed?

If you call a hosted model through an API, this choice is made for you. It matters when you build or tune your own.

Three ways to build the pipeline

Lenovo ran a benchmark of three multimodal RAG designs. It's a handy map of the trade.

The paper doesn't crown a winner. Its goal is to make the trade clear. So match the design to what your app can't get wrong.

A support bot can live with summaries. A tool that reads charts or checks parts needs the real image when it answers.

One catch on the first two. The summary step is the slow part. The paper says it pushes total time up unless you cache the summaries and reuse them across many queries. So cache them.

Where do the embeddings live? In a vector database. See vector databases explained.

Route by task

Don't send every request to your biggest model. Send the easy ones to a small, fast model. Keep the big one for the hard ones.

Does that hold up for image tasks? VL-RouterBench put it to the test. The benchmark covers 17 vision-language models, 14 datasets and 30,540 samples. The paper tried 10 routing methods and baselines. It reports a significant gain from routing. It also found the best routers still sit well short of a perfect one.

So routing pays. And there's room to do it better.

We built model routing into a white-label AI SaaS. OpenRouter routes between Claude, GPT and other models. The team can pick a model per tool, per tenant and per task. The case study shows how it fits together.

Routing is also your main cost lever. Our guides on token cost optimisation and AI model selection criteria go deeper.

Live voice and video

Live apps are harder. The user talks. The camera rolls. The answer has to land while they're still mid-task.

Google Cloud has published a design for live two-way multimodal streaming. The main parts:

• A persistent WebSocket. It holds the line open between the app and the server.
• A dispatcher agent. It spots a request for help and sends the stream to the live model.
• A streaming buffer. It carries the answer back to the user.
• A safety monitor. It watches the video for hazards and triggers a warning.

Their example is a good one. A user points a camera at a control panel and asks what a flashing red light means. If a background vision check spots a cable in the wrong port, the system cuts in with the fix.

Digiocial has a plain take on what live voice AI can do now.

Can the model run on the phone? Sometimes. Our on-device AI guide covers when that pays off.

Measure the slow tail

Average speed hides the pain. Measure P99. That's the time 99 out of 100 requests beat. The slowest one in 100 is the one your user remembers.

Split the clock too. The Lenovo paper times indexing and answering as two jobs. It also tracks time to first token, which is how quick the app feels. Do the same.

Four tests to run before launch:

1. P99 under load. Many users at once, on real networks.
2. Dropped streams. Cut the line mid-answer. Does the app recover clean?
3. Grounding. Does the answer match the image, or did the model make it up? Our hallucination reduction guide covers the checks.
4. Agreement. Do the text and the picture tell the same story?

After launch, watch for drift. Real photos and real voices won't look like your test set for long. See AI model drift.

Tip: build the dropped-stream test first. An app that handles a cut line well tends to handle the rest.

Privacy in Australia

A photo of a face can be personal information. So can a voice clip. If the Privacy Act covers your business, it covers these inputs too. It doesn't matter whose model sits underneath.

The OAIC has two guides worth a read. One covers developing and training generative AI models. The other covers using commercially available AI products.

One date to know. From 10 December 2026, APP 1.7 kicks in. Your privacy policy must say so if a computer program makes a decision, or does something substantially and directly related to making one. It applies when the decision could significantly affect a person's rights or interests and uses their personal information. The OAIC's APP 1 guidelines set out the detail.

What to do now:

1. List the decision points. Write down each place the app decides something about a person.
2. Run a privacy impact assessment early. Not the week before launch.
3. Map the data flow for each input. Photos and voice often need different consent and retention rules.
4. Strip what you don't need. Our PII redaction guide covers the basics.
5. Know where the data sits. See data residency in Australia.

Want a frame for all of it? Standards Australia has adopted ISO/IEC 42001, the standard for AI management systems. It gives you a structure to show how you govern your AI.

In a regulated industry and running AI agents? WithinBounds builds a control layer that enforces policy when an agent acts and records each decision.

For the wider rules, read our guide to AI regulation in Australia.

Build or buy?

Most teams overbuild. They wire up a custom pipeline before they know anyone wants the feature.

Start narrow. One flow. One off-the-shelf VLM through an API. A visual search or a document reader is a good first pick. Prove people use it. Then spend.

Build your own when:

• the data is too sensitive to send out
• the feature is the heart of your product
• off-the-shelf models can't hit your speed or accuracy mark

Bringing in a partner? Ask to see systems under real load, not a demo. Get it in writing who owns the data pipeline and any tuned model.

AI Orchestrators has a build vs buy scorecard for the business side. AI-Led breaks down what an AI app costs to build.

Where Devwiz fits

A multimodal demo is quick to make. An app that holds up under real use takes real engineering.

We build AI apps and turn programs into AI platforms. The AI goes in from the start, not bolted on at the end.

Devwiz has shipped 200+ apps, including work for NSW Government (Justice and Corrective Services), Briometrix, Vivid and Huskee.

Planning a multimodal feature? A discovery phase maps your inputs, your speed budget and your privacy risk before any code gets written. Worth a chat.

Frequently asked questions

What is a multimodal AI app?

It's an app that takes more than one kind of input, such as text, images, audio or video, and gives one answer. A field tool that watches a camera feed and talks a tech through a fix is one example.

What are examples of multimodal AI apps?

Visual search that matches a photo to a product. Document readers that pull fields from a scanned invoice. Field support tools that pair a live camera feed with spoken help. Call tools that turn speech into text.

Is ChatGPT multimodal?

Yes. Australia's Digital Platform Regulators Forum notes that chatbots that began as text-only now offer multimodal functions. Its working paper names OpenAI's GPT-4o as a multimodal foundation model.

Which multimodal RAG design is fastest?

It depends which clock you watch. In Lenovo's benchmark, searching the images direct was 10 to 100 times faster to index. Summarising images to text first gave the fastest first word of the answer. No design won on both.

How do multimodal AI apps handle privacy in Australia?

Photos, video and voice can be personal information under the Privacy Act. Map the data flow for each input, run a privacy impact assessment early, and read the OAIC's AI guidance. From 10 December 2026, APP 1.7 also needs your privacy policy to cover certain decisions made by computer programs.