A reference architecture for LLM features inside enterprise SaaS.

Most LLM integration guides assume you are building a consumer product or an internal tool with a small user base. The architecture that works for a chatbot demo does not work inside a regulated enterprise product with 50,000 seats, a compliance team, and a security review process.

This is a reference architecture for the second kind of product. It is not the only way to do it. It is the pattern I have found works across financial services, healthcare-adjacent, and media companies — three sectors where I have shipped AI features at scale.

The five non-negotiable concerns

Before any architecture decision, there are five questions every LLM feature in a regulated enterprise product has to answer. Not eventually — on day one, before the feature ships to any user.

1. Identity

Who is calling the model, and does the caller have permission to make that call?

In a consumer product, you authenticate the user once at login and assume all subsequent actions are theirs. In an enterprise product, that assumption is not enough. Different users have different roles. Some roles should not be able to trigger certain model behaviors. Some content should not be visible to certain users even if they can technically reach the endpoint.

The architecture implication: the LLM API call must be downstream of your authorization layer, not parallel to it. The model should never see content or instructions that the calling user is not authorized to see. This sounds obvious. In practice, I see it violated regularly when teams add AI features to existing products without revisiting the authorization model.

2. Audit

Can you reconstruct exactly what happened in any given model interaction, after the fact?

Regulated industries require this. Healthcare needs it for HIPAA. Finance needs it for SOX and various banking regulations. Even in industries without hard requirements, enterprise customers will ask for it in procurement. "Can we audit what your AI said to our users" is a question you want to be able to answer with yes.

The minimum viable audit log contains: timestamp, user identifier, the full prompt (including system prompt), the full response, the model and model version, and the latency. Store it somewhere that is not the same database as your production data. Keep it for at least 90 days, longer if your compliance team specifies.

3. Fallback

What happens when the model is unavailable or returns an error?

LLM APIs have higher variance in availability and latency than most infrastructure your product depends on. If your feature degrades gracefully when the model is unavailable, it is a temporary inconvenience. If it breaks the workflow entirely, it is an incident.

Design the fallback before you design the happy path. In most cases, the right fallback is the pre-AI version of the feature: the manual workflow, the rule-based system, the static content. If there was no pre-AI version, the fallback is a clear message to the user that the feature is temporarily unavailable, with no broken state left behind.

4. Evaluation

How do you know the model is performing correctly in production?

This is the concern most teams defer and then regret. See my earlier piece on eval-first development for the full argument. In the context of enterprise architecture, the practical requirement is: you need a way to sample production interactions, score them against your quality criteria, and detect degradation before users report it.

At minimum, this means logging a sample of interactions with enough metadata to reconstruct what happened. Ideally, it means a lightweight automated scorer that runs on that sample on a schedule. The scorer does not need to be perfect. It needs to be consistent enough that a significant change in the score correlates with a real change in quality.

5. Cost

What does this feature cost to operate at your projected usage level, and does that cost make sense relative to the value it creates?

Enterprise procurement teams will ask about this. Your finance team will ask about it after the first month of production traffic. It is better to have the answer ready.

Cost modeling for LLM features has three components: token cost (input + output, at current model pricing), the cost trajectory as usage scales, and the cost of model calls in failed or retried interactions. The last component is consistently underestimated. If your retry logic is aggressive and your error rate is 2%, the retries on a high-traffic feature add up.

The architecture pattern

Given those five concerns, the pattern I use is a thin orchestration layer between your application and the model API. The orchestration layer is responsible for:

• Enforcing authorization before constructing the prompt
• Constructing the prompt from templates, not user-supplied strings
• Calling the model API with a timeout and retry policy
• Writing the audit log entry
• Returning a structured response to the application layer, including a fallback signal

The application layer is responsible for the fallback behavior. It should not be the orchestration layer's job to decide what the UI does when the model is unavailable.

This separation keeps the model integration testable in isolation and makes it easier to swap model providers without touching application code. When a provider releases a new model, you test it against your eval set in the orchestration layer. The application layer does not need to change.

What this pattern does not solve

This architecture handles the infrastructure concerns. It does not handle prompt quality, which is a product and evaluation problem. It does not handle the governance question of what behaviors the model should and should not have, which is a policy problem. And it does not handle the change management question of how you get your organization comfortable with AI-generated content in a regulated context, which is a leadership problem.

Those three problems are harder than the architecture. The architecture just ensures that when you have solved them, you have a system that is auditable, resilient, and cost-accountable.