AI

Data Lake Governance: A Practical Guide for AI Builders

By James KillickNovember 16, 2026

TL;DR: Data lake governance means every dataset has an owner, a catalogue entry, an access rule, a quality check and a retention rule. Build it into the pipelines from the first load, because AI features inherit whatever state the lake is in.

Data lake governance is how you keep a data lake useful. Every dataset gets an owner, a description, an access rule, a quality check and an end date. Without that, the lake fills with files nobody trusts. If you are building AI on top of it, the AI inherits the mess.

This is for founders and CTOs building an AI product on a lake. It covers what to govern, how to roll it out and what Australian rules mean for you. We build data warehouses, automation and AI platforms together at Devwiz, so this comes from the build side.

What data lake governance covers

A data lake is one big store for data in its original form. Database exports, app events, documents, logs and third-party feeds all sit in it. That freedom is the point. It is also the risk. Nothing forces anyone to say what a file holds, who owns it or if it is still right.

Governance fills the gap. Here's the thing: it comes down to six questions per dataset.

  1. What is it? A plain definition and a place in a catalogue.
  2. Where did it come from? The source system, how it was collected and how often it updates.
  3. Who owns it? One person who answers for its quality and use.
  4. Who can use it, and why? Access rules tied to roles and purpose.
  5. Can we trust it? Quality rules that are checked, not assumed.
  6. When does it go? Retention and disposal rules.

The research agrees. Khatri and Brown's framework in Communications of the ACM names five linked decision areas: data principles, data quality, metadata, data access and data lifecycle. Principles come first. What you plan to do with the data sets the quality bar, and the bar shapes how you describe the data and who can read it.

Why it matters more when you build AI

A lake with no governance is called a data swamp. Data goes in with no oversight and no documentation. TechTarget lists the damage: quality problems, analytics errors, poor decisions and exposed sensitive data. A well-governed lake does the reverse. Data scientists find what they need faster and spend less time preparing it.

AI raises the stakes in three ways:

  • Models learn from what you feed them. Stale, duplicated or mislabelled data gives you bad output, and it is hard to see why.
  • Agents act on data. If an agent can read a table, it can repeat what is in it. Sensitive fields can end up in an answer.
  • You may have to explain it. When a customer or regulator asks where an answer came from, you need lineage. That is the path from source to result.

The Australian Government's AI Technical Standard asks agencies to set standard operating procedures for data orchestration. It also asks them to watch for changes in data access and completeness, check whether sensitivity changes after processing, and validate data trust and authenticity. It is written for government, but it works as a checklist if you ship AI to Australian customers.

We cover the wider picture in our five-pillar model for data governance for AI. This post stays on the lake.

The parts of a governed lake

The Queensland Government's data governance guideline is a clear Australian reference. It treats quality as completeness, consistency, conformity, accuracy, integrity and timeliness. It also covers the lifecycle from planning through to disposal.

Here is that turned into a build checklist:

PartWhat it means in the lakeWho owns it
OwnershipEvery dataset has an accountable owner and a day-to-day stewardBusiness lead for the domain
Catalogue and metadataDefinition, source, update rate, sensitivity, certified or notData team
Access controlRole-based permissions at every layer, with a reviewed exception processEngineering and security
Quality rulesAutomatic checks before data moves up a layerData engineering
LineageA record of how data moves and changesData engineering
LifecycleRetention periods, archiving and safe disposalData owner and legal
OversightA regular review of issues, exceptions and standardsFounder, CTO or steering group

Metadata needs extra care. The Australian Government Data Governance Framework asks agencies to make data currency clear: the timestamp, the source, how it was collected and how often it updates. Those four fields are a good minimum for any lake. They answer the first thing a new engineer or an AI pipeline needs to know: is this still good?

Practices that hold up in production

Govern every layer

Most lakes have layers. A raw landing zone, a cleaned layer and a curated layer that feeds reports and AI. If you only protect the curated layer, you miss the point. Raw data often holds the most sensitive fields. Catalogue from the first load, set access rules on the raw layer too, and check quality before anything moves up. Estuary makes the same case across all layers in its data lake architecture guide.

Set the quality bar by use

Not every dataset needs the same bar. The Australian Government framework says quality should match how the data is used and how much it matters. Data behind audit, enforcement and compliance needs high accuracy and scrutiny. In a product, a billing table or the data behind a customer-facing AI answer gets strict checks and alerts. An exploratory table does not.

Name owners before you automate

The ANAO's Audit Lessons: Governance of Data pushes for clear data leadership and roles, and for data processes mapped end to end. That is a solid order of work. Automation on top of unclear ownership just spreads the confusion faster.

Make access specific

"Everyone in engineering can read the lake" is not a policy. A real rule says who can read which data, under what conditions and for what purpose. It also has a way to grant temporary access that records who approved it and when it ends. The same goes for AI. Decide which datasets an agent or retrieval pipeline may read, and enforce that in the platform, not in a prompt.

Plan the retirement

Lakes only grow. Data nobody needs still costs storage and still carries privacy risk. Set retention and disposal rules when you create the dataset.

A rollout that works for a product team

You do not need a committee of twelve. You need the same logic in a smaller form.

  1. Inventory the lake. List the source systems, the datasets and who uses them (reports, apps, models). This is where unowned and duplicate data shows up.
  2. Classify by sensitivity and by how much each dataset matters. Mark personal information, money data and anything a customer-facing AI feature relies on.
  3. Name owners and stewards. One accountable person per domain. Without that, quality rules have nobody to escalate to.
  4. Stand up the catalogue. Definition, owner, source, update rate, sensitivity and certified status is enough to start.
  5. Put quality gates in the pipelines. Checks run between layers and failures block promotion. Our note on ETL versus ELT shows how the pipeline pattern changes where those checks sit.
  6. Apply role-based access and an exception process. Review access on a schedule, not only when something breaks.
  7. Add lineage and audit logs. Record who accessed what, including models and agents.
  8. Set retention rules and review the whole thing often. The ANAO says to review regularly because AI keeps changing.

One problem though. Teams go wrong when they do steps 4 to 7 by hand. A spreadsheet catalogue goes stale in weeks. Controls built into the pipeline code and platform stay current, because data cannot move without passing through them. When we build a warehouse or lake as part of an AI platform, governance sits in the same design as the pipelines. That is why we raise it in discovery, not after launch.

If you are also deciding how the lake connects to your AI features, see our guides to AI platform architecture and to building a RAG pipeline. Retrieval is only as reliable as the documented data under it. AI-Led has a plain explainer on what a vector database is if that part is new.

What Australian rules mean for your lake

Two things matter most if you build on personal data.

First, the Privacy Act reforms. The Australian Institute of Company Directors says the first set, legislated in November 2024, did three things. It introduced a statutory tort for serious invasions of privacy. It added new transparency rules for automated decision-making. And it widened the OAIC's monitoring and investigation powers and added new civil and criminal penalties. The OAIC is the Office of the Australian Information Commissioner, the privacy regulator.

In plain terms for a lake: know where personal information sits, limit who and what can read it, and be able to show where the data behind automated decisions came from. Classification, lineage and audit logs are the controls that make that possible.

Second, public-sector expectations. If you sell to government, or to firms that copy government practice, the NSW Data Governance Toolkit is worth reading. It says to ask if a data platform is open source, scalable, interoperable and easy to fit into existing processes. It also says to check it meets government standards on data sovereignty, privacy and cyber security. Sovereignty means where the data is stored and which laws apply to it. You decide that at design time. Our post on data residency in Australia goes deeper, and our AI software compliance guide covers the wider obligations.

This is general information, not legal advice. For how the Privacy Act applies to your data and customers, talk to a privacy lawyer.

Governing data for AI features

The ANAO says AI should sit inside your information governance framework, with specific care for the data that goes into and comes out of the AI. Three habits follow.

  • Certify the datasets AI may use. Keep a short list of approved, documented, quality-checked datasets for training and retrieval. Everything else is off limits until reviewed.
  • Treat model outputs as data. Generated content that lands back in the lake needs an owner, a label, a retention rule and a flag that a machine wrote it.
  • Track versions. When a model, prompt or dataset changes, you should be able to say which version produced which output. Our guide to AI model versioning in SaaS covers the model side.

Tools can enforce the rules. People decide what counts as sensitive, who owns a dataset and what good enough means for each use. The team at AI Orchestrators lays out a five-step data governance plan for data teams if you want a second view.

Where it goes wrong

  • Governance as a document. A policy PDF with no enforcement in the pipelines is ignored within a quarter.
  • No owner for the raw layer. Raw data feels temporary, so nobody claims it. It holds the most sensitive fields.
  • One quality bar for everything. Too strict blocks experiments. Too loose lets bad data into reports.
  • AI added later. Bolt an agent onto an ungoverned lake and it exposes whatever the lake holds.
  • No retirement rules. Storage cost and privacy risk grow quietly.

Where Devwiz fits

Devwiz builds AI agents, AI-first platforms and custom apps, along with the data warehouses and automation that run underneath them. We have shipped 200+ apps, including work for the NSW Government, Briometrix, Vivid and Huskee.

If you need a governed data foundation for an AI product, start with our AI platform development service. We design the data layer, the pipelines and the AI features together, so ownership, access and quality rules are part of the build.

Already running a SaaS or mobile app and want AI on top of your data? Same approach. We look at what is in your stores, what the AI needs to read and what needs controlling before anything ships. Worth a chat? Book a call with Devwiz.

Frequently asked questions

What are the four pillars of data governance?

Sources group them differently. A common set is ownership, data quality, access and security, and lifecycle, with metadata tying them together. Khatri and Brown's academic framework uses five areas: principles, quality, metadata, access and lifecycle. Pick the grouping your team will use and give each one an owner.

What is the difference between data lake and data warehouse governance?

The principles are the same. A lake takes more effort because it accepts data in its original form with little structure. A warehouse enforces a schema on the way in, so some quality control comes built in. In a lake you add catalogues, quality gates and access rules on purpose, layer by layer.

When should a startup start governing its data lake?

At the first load. Cataloguing and ownership are easy with five datasets and painful with five hundred. Start with minimum metadata, named owners and role-based access. Add lineage and retention as the platform grows.

Will AI replace data governance?

No. AI can help classify data, spot odd values and draft documentation. People still decide ownership, sensitivity and acceptable use. The ANAO says to bring AI into your governance framework, not swap the framework for AI, and to review it regularly.

About James Killick

10+ years building digital products · 200+ apps shipped since 2015

James is a co-founder of Devwiz and an AI product specialist. Since 2015 he has helped ship 200+ apps for founders, businesses and government, including work for NSW Government, Briometrix and Huskee. He builds AI-first platforms and writes about turning a proven program into software. He also hosts the Up in the AI podcast.

More articles by James · James's personal site · LinkedIn · AI Orchestrators

Tags: Data Lake Governance, Data Governance, AI Platforms, Data Quality

Browse all Devwiz articles·See our case studies