Your AI Agents Need a Metadata Layer First

Two client conversations from the last quarter keep coming back to me. I want to talk about them because they’re the same conversation from opposite ends, and together they explain why every AI-agent initiative I’ve watched inside a Fortune 1000 in the last twelve months has gotten stuck at the same layer.

Two clients, two failure modes of the same problem

The first is a life & annuity carrier — mid-to-large US book, respected name, disciplined data team. They’ve been paying six-to-seven figures a year on Alation for years. Real spend, publicly committed to as their enterprise catalog. When we ran a metadata audit last quarter, we found less than 10% of tables had meaningful business context. Technical schema was there — every column, every data type, every join key that the crawlers could infer. Ownership, business definitions, PII flags, certification status, glossary linkage — almost nothing. They were paying enterprise SaaS pricing for what functionally amounted to search over a schema dictionary. The tool worked exactly as designed. Nobody was hired to fill it in.

The second is a national fire & security field-services operator — installs and monitors systems across thousands of commercial sites. No catalog at all. Every department maintains its own KPIs and definitions in its own spreadsheets. “Active install base” means one thing to the operations team, a different thing to finance, a third thing to the sales VP. Weekly leadership meetings routinely stall on which number is real. Nobody’s fault, exactly — the org grew fast, the systems layered on top of each other, and the metadata + semantic layer that would have kept everyone speaking the same language never got built.

Two failure modes of the same underlying problem. One paid a vendor for a tool nobody populated. The other never bought a tool and let the layer sprawl across the org. Neither is unusual. I could give you five more of each shape from just this year.

Why this pain suddenly matters much more

For years, this was tolerable. Not good — but tolerable. A human analyst who knows the estate can work around a bad catalog. If the sales VP calls asking about “active install base,” the finance analyst just picks up the phone. If the underwriter needs to know which column carries the current premium, they ask the DBA. Institutional memory carried the load that the metadata layer was supposed to carry.

That workaround does not survive contact with AI agents.

An AI agent cannot call the analyst who knows. It has to read what’s in the catalog. If the catalog says nothing, the agent guesses. And an LLM-driven agent guesses in ways that are confident, syntactically plausible, and wrong. A text-to-SQL agent will pick revenue_final_v2_use_this because it’s the most-recently-modified table matching the word “revenue,” and produce a report that reconciles to nothing. A claims-processing agent will treat two different codes as the same status, or the same code as two different statuses, depending on which subquery it wrote. The agent doesn’t know it’s wrong. The user doesn’t know it’s wrong until the number hits a board deck.

The reframe I’d offer: metadata is no longer documentation. It’s an input to the AI system. Its quality is now measurable in agent output quality. That changes the ROI calculation of finally building the layer from “nice governance win, hard to justify” to “does the AI initiative we just approved actually work or not.”

The three layers people conflate into one purchase

Part of why enterprises get stuck here is that the vocabulary is sloppy. “Catalog,” “semantic layer,” “governance tool” — buyers use them interchangeably in RFPs, and vendors don’t correct them because the ambiguity sells software. There are three distinct layers doing three different jobs:

• Technical catalog — AWS Glue Data Catalog, Microsoft Purview, Databricks Unity Catalog. Schema, lineage, columns, refresh timestamps. Mostly machine-populated by crawlers. Cheap, always partial, and worth roughly zero to a business user on its own.
• Semantic layer — dbt Semantic, Cube, LookML, Fabric semantic models, Snowflake Cortex Semantic. Business definitions, metrics, joins, hierarchies. Human-defined. This is where “revenue,” “active install base,” and “in-force premium” get their canonical definitions.
• Governance / discovery layer — Alation, Collibra, Atlan. Ownership, certification, glossary, search, stewardship workflows. Steward-populated. This is where a business user goes to ask “who owns the customer table?” and get a real answer.

The life & annuity client bought the governance layer at premium pricing and never staffed the stewards to populate it. The fire & security client didn’t buy anything and had no framework to build even the technical layer. Enterprises assume that buying one tool solves all three jobs. They don’t, and the tool spend continues while the layer never gets built.

What AI agents can do when this layer is right — and what they can’t when it isn’t

Here’s where the story turns from diagnosis to what’s actually possible. Every one of these examples is an agent shape I’ve either shipped or scoped in the last year. The “with vs without” contrast is not theoretical.

Life & annuity

• New-business underwriting-assist agent. With certified policy_type, annuity_product_code, funding_source, and medical-underwriting-class definitions: the agent reads the application packet, cross-checks against the certified product catalog, pre-fills the underwriter’s worksheet, and flags anything requiring reinsurance treaty review. Underwriter throughput lifts materially; time-to-issue drops. Without: product codes drift between the illustration system, the admin platform, and the reinsurance feed; the agent’s recommendations reference codes the underwriter has to translate manually — negating the whole point of automation.
• Annuity servicing / withdrawal-request agent. With one canonical definition of contract_value, surrender_charge_schedule, and guaranteed_minimum_withdrawal_benefit: the agent handles routine withdrawal requests end-to-end, escalates only exceptions, and generates the client-facing confirmation. Without: three systems disagree on remaining surrender period; agent output can’t be trusted for anything customer-facing, so it gets confined to internal notes that nobody reads.
• Beneficiary-change / claims-processing agent. With a certified beneficiary-hierarchy model and canonical date_of_death and proof_of_death fields: the agent walks the case through validation and payout calculation, flags anti-fraud signals. Without: every corner case hits a definitional gap and 90% of work gets handed back to a human — which is the same throughput as no agent at all.
• Actuarial-data-prep agent. With certified in-force block, mortality, and lapse metrics: the agent assembles the quarterly experience study extract, reconciles to the GL, and flags exceptions. Actuaries stop spending their first two weeks of every quarter on data plumbing. Without: every quarter is a manual reconciliation exercise, and nobody trusts the first-pass numbers.

Manufacturing

• Predictive-maintenance triage agent. With a clean asset hierarchy, OEE definitions, and a downtime_reason_code glossary: the agent correlates vibration and temperature signals to specific line assets and generates work orders that maintenance can actually act on. Without: agent recommendations reference asset IDs that don’t match CMMS records, and technicians ignore the queue within a week.
• Quality-inspection agent. With a defect taxonomy defined once and referenced by both vision system and MES: the agent flags out-of-spec parts by canonical defect code, and trending across shifts, lines, and plants actually works. Without: the same defect gets logged under three names, and trending is impossible — so continuous improvement stalls on the data step before it even begins.

Retail

• Merchandising agent. With a certified SKU hierarchy, canonical net_sales metric, and consistent brand/banner taxonomy: the agent auto-generates weekly assortment recommendations by store cluster. Category managers actually use them. Without: recommendations reference brand codes from one system and store codes from another; managers don’t trust them and revert to the spreadsheet they always used.
• Pricing agent. With one definition of “margin” across POS, ERP, and marketplace feeds: the agent proposes price moves with defensible math the CFO can audit. Without: three definitions of margin produce three recommendations, and zero adoption.

Cross-industry — the ones every enterprise needs

• Text-to-SQL / analyst-copilot agent. With: only certified metrics are exposed; every answer cites the definition. Analysts get a real force-multiplier. Without: the agent picks the most-recently-modified table matching the keyword, hallucinates joins, and produces confidently wrong numbers. This is the single most common way I see AI agent programs fail — the demo works on a curated dataset, and the production rollout falls apart on the messy 80%.
• Data-quality agent. With: the agent knows which columns are business-critical and pages the owner when drift hits a certified field. Without: it flags every schema change equally, gets muted within a week, and eventually gets turned off.
• Catalog-search agent. With: “where does customer churn live?” returns the certified dataset, the owner, and the freshness. Without: it returns three tables from 2019 and nobody clicks on them.

The pattern across all of these: the ceiling on agent accuracy isn’t the LLM. It’s the metadata layer underneath. Change the model from Claude to GPT-5 to whatever’s next, and the ceiling doesn’t move — because the constraint is what the agent can read about the data, not how well it can reason.

The stack that actually works

Here’s the pattern I’d stand up if I were starting from scratch today, and increasingly what we’re deploying under our S.C.A.L.E. foundation on real engagements:

Foundation: a warehouse or lakehouse — Snowflake, Databricks, Fabric, BigQuery. Pick by your platform strategy, not by this decision. The foundation doesn’t care about AI agents; it just needs to be stable, cost-controlled, and queryable at the concurrency the semantic layer will demand.

Semantic layer: always-on contract. Every metric defined once. Every join encoded. Every hierarchy modeled. This is where the AI agent reads business meaning — not where a human occasionally goes to look up a definition. If the agent can’t reach a definition programmatically, it’s not really there.

Metadata / catalog layer: technical plus business plus steward-curated. Discovery, ownership, certification, PII flags, refresh cadence, quality signals. This is where the AI agent reads discovery and trust from. “Is this dataset certified? Who owns it? Is it fresh?” — the agent has to be able to answer those questions before it acts.

AI agents on top: not as a parallel initiative, not as a separate program with its own budget line. Every agent — Arcastra’s or anyone else’s — inherits the layer’s discipline. If the layer is real, agents get accuracy for free. If it isn’t, agents get stuck, and no amount of prompt engineering fixes it.

Governance: emerges from this stack rather than being a separate program with a separate committee. Ownership is metadata. Certification is metadata. PII handling is metadata. When the layer is real, governance is a byproduct — and the compliance conversation stops being “prove our controls” and starts being “here’s the certified view, filtered to what you’re entitled to see.” That’s the reframe every regulated-industry CDO I talk to has been waiting years for. If governance is a program you’re actively running on top of the layer, the layer isn’t real yet.

We package this as a data governance and management engagement — but the deliverable is the layer, not the tool.

The uncomfortable truth about metadata tool spend

Alation, Collibra, Atlan, Purview — all fine tools. The tool isn’t the problem. Nobody was hired to fill it in.

Steward-hours are the invisible line item. Enterprises don’t budget for them, so the tool ROI never materializes, and then the tool gets blamed. I’ve watched three separate clients rip out one governance tool and buy the next one over, when the actual problem was that neither tool had steward FTEs assigned to it.

Blunt take: if you’re two years into a metadata tool and the populate rate is under 50%, you don’t have a metadata problem — you have a stewardship problem. Adding another tool won’t help. If you’re at 10%, like the life & annuity client above, you have an acute stewardship problem, and the first productive conversation is with HR, not with a vendor.

The fire & security case is the mirror image. Don’t buy a tool first. Define who owns each domain first — active install base is finance’s number and it means X; installed asset count is operations’ number and it means Y; contract value is sales’ number and it means Z. Get the definitions written down and owned before you decide where to store them. The tool is the last decision, not the first.

What to do in the next 90 days

If you’re mid-way into an AI agent initiative and this diagnosis sounds uncomfortably familiar, here’s the sequence I’d run:

1. Audit the current metadata layer honestly. Populate rate, ownership rate, freshness. Report the number, even if it embarrasses somebody.
2. Pick five “gold” datasets — the ones every executive report and every planned AI agent depends on. Not 50. Not 500. Five.
3. Assign human owners to those five. Named individuals with time carved out. Not “the data team.”
4. Get their metadata to ~95% populated — business context, PII flags, certification, refresh cadence, canonical metric definitions, quality signals. Use whatever tool you already own; don’t buy a new one for this.
5. Deploy your first AI agent on top of those five datasets only. Measure accuracy vs a baseline. Iterate on the layer, not the model.
6. Expand from there — the next five, then the next twenty. The population curve compounds because the stewards develop the muscle memory.

This is data management done as engineering, not as a compliance checkbox. It’s less exciting than the AI-agent demo, and it’s the only reason the AI-agent demo works in production.

The closing thought

The AI-agent conversation and the metadata-layer conversation are the same conversation now. Every enterprise I talk to wants to know which agent to build first, which model to license, which platform to standardize on. Those are all reasonable questions. They’re just not the first question anymore.

The first question is: is the layer underneath ready to be read by something that can’t call the analyst? If it isn’t, the rest of the roadmap is theatre. If it is, the roadmap you already have will work better than you expect.

Neither of the two clients I opened with is a lost cause. The life & annuity carrier has the tool, the budget, and the appetite — they need the stewards. The fire & security operator has the appetite and the ownership clarity waiting to be surfaced — they need the framework, not the tool. Both are now on paths where the AI agent conversation is the layer conversation. Both will be materially further along a year from now than the peers who buy another model license instead.

If you’re evaluating where your own estate sits on this curve, that’s the conversation I’d want to have first.