Why Access Control Cannot Be an Afterthought

It is tempting to treat access control as a finishing step: build the memory layer, add the agent, then tell the model “only use information this user is allowed to see.” That approach fails in predictable ways.

Models do not reliably enforce policy. Retrieval indexes can return items the user should not receive. Even correct chunks become a leak if they are assembled into an answer without an upstream gate.

Authorization must resolve before retrieval

The question to answer at query time is not “what is semantically similar?” but “what is this user allowed to access right now, and among that, what is relevant?”

In the architecture I am exploring, PostgreSQL resolves allowed projects (and eventually finer scopes) from roles and assignments. Only then does the system query hybrid graph or vector indexes, and only within permitted boundaries.

Graphiti and similar tools may store access metadata on episodes and chunks, but they are not the authority for rights. They are indexes that must be filtered against a trusted authorization store.

Access envelopes belong on retrievable items

Every element that could surface in retrieval—graph facts, distilled passages, structured objects—should carry an access envelope at write time: tenant, project, visibility scope, classification, and policy references.

This metadata is attached during ingestion, not inferred at answer time. If an item lacks a valid envelope, it should not be retrievable.

Verification is a second line of defense

Even with scoped indexes, the retrieval orchestrator should verify graph hits against stored memory records and source references in PostgreSQL before they enter the agent context. A hit that cannot be matched to an allowed project and a known source is discarded.

That pattern is slower than trusting the index blindly. It is also how you avoid training users to treat probabilistic retrieval as a security boundary.

Prompts are instructions, not policy engines

System prompts can describe behavior: cite sources, express uncertainty, separate facts from interpretation. They cannot reliably substitute for deterministic checks.

Sensitive categories—HR matters, client-confidential material, legal privilege—need rules evaluated outside the model. The agent should receive only pre-authorized context.

Individual agents still need shared governance

User-specific agents are useful: a project manager and a finance lead should not see the same slice of memory. Individualization is a product feature. Governance is a platform requirement.

The same ingestion pipeline and policy resolver should serve all agents. Otherwise each team ships a different notion of “safe enough,” and audits become impossible.

What this costs

Building access control first feels slower than shipping chat on top of an unscoped index. It is slower. It is also the difference between a demo and something a business could trial without immediately creating a data-handling incident.

Practical takeaway

If you are designing enterprise AI memory, draw the boundary early: authorize, then retrieve, then reason, then answer. Anything that reverses the first two steps is borrowing trust you have not earned.