A Real Model, Not a Mock

The series finale. In the first article we laid out the problem, and in the second we turned the contract into code. In this one we measure that same contract behind real Azure services.

My concern in the first article was this: in regulated industries, an answer a model gives without a source isn't just a "wrong answer"; it's a real risk in production. In the Air Canada case and in Mata v. Avianca, it came down to the same thing: the model didn't stop where it didn't know. It kept talking even though it wasn't sure.

In the second article I brought that problem down to code. I treated the refusal and citation contract as a mechanism that can genuinely stop the system on the .NET side. But that article had a deliberate limit: the system still ran on mock services. MockKnowledgeSearchService searched over a handful of in-memory chunks, and MockChatClient returned deterministic answers. So at the end of the second article, one question was left open:

Will the same contract still hold behind a real model and a real retrieval service?

This article is the answer.

I moved the system behind Azure AI Search and Azure OpenAI. Then I didn't just check whether it "worked". I tried to measure this: does it refuse when there's no source, does it let invalid citations out, does it ever show the model a document the user isn't authorized to see, and when it does answer, is the answer actually grounded in the context?

In short, this article takes the architectural claim one step further. We're no longer just saying "this is how I designed it". We're measuring it.

What's in this article?#

I'll show three things.

First, I took the system off mocks and connected it to real Azure resources. While doing that, I deliberately didn't touch the Application and Domain layers, because that was the series' main claim: if the contract lives in the right place, the business rules shouldn't change when the provider does.

Second, I measured the system at two levels. The first was contract behavior: did it return a refusal when there was no source, did the role filter block unauthorized documents, does the system stop the model if it returns a made-up citation? These are yes/no questions.

Third, when the system did answer, I looked at the quality of the answer, using evaluation metrics like groundedness, relevance and completeness. I didn't stop at "an answer came back"; I measured how well-grounded that answer was.

Let me be clear from the start: the results in this article come from tests run against real Azure resources. Anyone who opens the repository can try the same approach with their own resources. That's what I've been arguing since the beginning of the series: a contract you can't prove isn't a contract.

Part 1From mock to Azure: a controlled reality check#

When I moved from mocks to real Azure resources, my aim wasn't to rebuild a production environment one-to-one. If I had, the discussion would have quickly drifted into cost, networking, deployment and operations. What I wanted to measure was narrower: do the refusal and citation contracts hold behind real retrieval and a real model?

So I deliberately built a small but sufficient setup:

I used Azure AI Search for retrieval, a gpt-4.1-mini deployment on Azure OpenAI for answer generation, and Azure SQL Database for the audit side. The system still ran on my local machine, but the two critical pieces behind it were no longer mocks: the search was real, and the model was real.

On the search side I didn't get into vector search at this stage. My goal wasn't to design an embedding pipeline, chunk vectorization or hybrid search, so I took a simpler route with the semantic ranker. One distinction is worth noting: the semantic ranker's free plan gives you a monthly quota, and if you want the paid standard plan you need a Basic or higher Search service. I used Azure AI Search Basic here, and the semantic ranker's free monthly quota was enough for these evaluation runs.

I didn't create an App Service at all. That was deliberate: I didn't need to host the web application on Azure to prove this article's claim. Running the system locally and connecting it to real Azure AI Search and Azure OpenAI services was enough to see how the contract behaves behind real services. That also kept fixed hosting costs completely out of the picture.

On cost, the resource that needed the most attention was Azure AI Search, because you can't "pause" a Search service to stop the bill. Once the service is created, its resources are allocated and it keeps costing money even while idle. So my strategy was simple: create the resources, run the tests, collect the results, then delete what's no longer needed. I set up what was needed for a controlled few-day study, not an environment that stays up for months.

On security I drew a clear line from the start: no API keys anywhere. No Search key, no OpenAI key, no SQL password. For local development I authenticated with Microsoft Entra ID through DefaultAzureCredential, and granted my own user the data-plane roles needed for Search and OpenAI. The SQL connection didn't carry a password either; it went through Entra ID.

I also kept the real endpoints, deployment names and connection details out of the repository. For local development they lived in .NET user-secrets, and the appsettings.json in the repo carries placeholders rather than real values. In production these values would move to Key Vault; I deliberately left Key Vault out of scope for this pilot.

Of course, the Azure integration didn't go perfectly smoothly, and I'm describing this part for anyone who wants to try it. The first serious problem was identity: my Search calls were getting 401s. The problem wasn't that the code was calling Search incorrectly. The DefaultAzureCredential chain was trying to get a token from a different tenant than the one I expected.

That small-looking error cost me a few hours. But it was useful. In local development, DefaultAzureCredential often gives you a comfortable "well, it works" feeling. In the real world you need to know which credential gets picked, which tenant the token comes from, and where your Azure CLI identity is attached. My fix was to explicitly point the credential chain at the right tenant and the right Azure CLI session.

By the end of this part I had a local API backed by real Azure AI Search, real Azure OpenAI, and Azure SQL for audit. Still no App Service, no API keys, no secrets baked into the repo. In other words, not all of production, but the smallest real foundation needed to test the contract. (As of the day this article was published, the whole study had cost me around $2.)

Part 2The real question: did the contract survive the move to Azure?#

For me, the most important question in moving to Azure was: when I swap the mock services for real Azure services, will I have to touch the layers where the contract lives?

The answer: no. Throughout the evaluation and Azure integration work, I didn't change a single line in the AgentAssist.Application and AgentAssist.Domain projects. I can show it with the commit diff:

git diff <before-eval>..HEAD --stat -- src/AgentAssist.Application src/AgentAssist.Domain
# empty output

That's not a small detail to me. It's exactly what I've argued since the start of the series: a safety contract shouldn't be buried in the provider. It shouldn't depend on Azure, OpenAI, Search or a mock service; it should live as application behavior.

What changed during the move was predictable: real adapters and configuration were added in Infrastructure, the necessary wiring was done in the API host, and the evaluation tests were built out in the test layer. But AssistantAnswer, CitationValidator, the refusal flow and the orchestrator's decision points stayed the same.

What made that possible was the interface separation we set up in the second article. The Application layer doesn't know whether the search service is a mock or Azure AI Search, or whether answers come from a deterministic test client or a real model behind Azure OpenAI. All it knows is the contract: IKnowledgeSearchService and IChatClient.

The real implementations behind those two interfaces can change. The orchestrator's rule doesn't: no source, no answer; invalid citation, no answer; high risk, no direct answer. Choosing the provider comes down to a single condition in the API host:

if (agentAssistMode is AgentAssistMode.DevCloud)
{
    builder.Services.AddDevCloudInfrastructure(builder.Configuration);
}
else
{
    builder.Services.AddMockInfrastructure();
}

From the application's point of view, that's the entire move to Azure: a different infrastructure package gets plugged in. Because the contract sits above that branch, which provider is chosen doesn't change the contract.

There's also a small real-life detail I want to share here. I originally wanted to use gpt-4o-mini, but I couldn't get quota for that model in my region. So I went with gpt-4.1-mini and left the deployment name as I'd first created it: agentassist-chat-gpt-4o-mini. The deployment name said "4o-mini", but the model behind it was gpt-4.1-mini.

That naming isn't great, and in production you'd fix it. But I deliberately didn't hide it in this study, because it showed me something: the application layer wasn't tied to a model family. The business rules didn't behave one way for gpt-4o-mini and another way for gpt-4.1-mini. The deployment in the config changed, the model behind it changed, and the contract stayed the same. If the business rules don't change when the provider does, the architecture is split in the right place.

That's the main claim of this article. I removed the mock services, real Azure AI Search and real Azure OpenAI came in, and the refusal and citation contract didn't move an inch.

Part 3Test layer 1: does the system know where to stop?#

After moving to Azure, the first thing I looked at wasn't answer quality. I started with a more basic question: does the system stop where it shouldn't answer?

Because that's what this series is really about. The model doesn't have to answer every question; for some questions it must not answer at all. If there's no source, if a citation isn't valid, if the user's role isn't enough, or if the topic is high-risk, the safe behavior isn't to produce an answer, it's to stop in a controlled way.

To measure that, I built an evaluation harness in the repository with six categories:

• answerable_with_citation: when an answer is given, does it have citations?
• no_source_refusal: when there's no source, does the system refuse?
• high_risk_escalation: do high-risk questions get escalated?
• role_restricted: if the user's role isn't enough, does the document leak anyway?
• adversarial_prompt_injection: can the model be made to return a made-up citation or the system prompt?
• inactive_filter: does an expired or inactive document come back as an active result?

For the first time, I ran these tests against real Azure AI Search and Azure OpenAI instead of mocks.

The result: 19 of 20 cases behaved as expected. Eight of them produced answers with sources and citations. Eleven were refused as the contract requires: some because no source was found, some because the user's role wasn't enough, some because the document was inactive, and some because they hit the adversarial checks. One case landed right on the threshold.

The most valuable part for me wasn't just being able to say "the tests passed". I recorded a trace of every step: the message sent to the model, the model's output, or the fact that the model was never called. I wanted to see exactly where the system stopped, and why.

The first example was the no_source scenario. I asked a question that had no counterpart in a clinic's knowledge base (something like "what's the latest on the stock market index?"). Semantic search didn't return a single chunk above the 0.7 threshold. The key point: the system never called the model. Because retrieval came back empty, the orchestrator stopped with "not enough sources were found" before it ever reached the generation step. The system didn't spend a single model call on a question it couldn't answer; the safe behavior kicked in at the earliest possible point. The run's own transcript confirms it: llmInvoked: false.

In fact, all 12 refusals in this run happened at the retrieval/orchestrator gate; the model wasn't called for any of them. The model's own refusal (model_self_refusal), the malformed-response defense and the invalid-citation defense are proven by unit tests in the repository, but the retrieval gate worked so well that the model never faced a borderline case. That's exactly what layered defense means: before one layer even has to act, the previous one has already closed the door.

The second example was the role_restricted and adversarial scenario. The index contained a chunk open only to the supervisor role: SECRET-CHK. When a user with the agent role tried to reach that information, the system never showed it to the model. The filter sent to Azure AI Search already started with isActive = true and was narrowed down to the roles the user is authorized for. So the secret chunk never made it into the retrieval results.

I want to stress this point: I didn't leave security to the model here. I don't rely on the model's good intentions, as in "please don't reveal the secret information". The secret information never enters the context at all. A model can't leak what it never sees.

There's a second layer on the citation side: CitationValidator. The model may only return, as citations, the chunk IDs it was given. If it returns an ID outside that whitelist, the handler rejects the answer and turns it into a refusal with the reason model_returned_invalid_citation.

To be honest, I couldn't trigger this layer end to end with the real model, because the role filter worked so early that the secret chunk never reached the model. In practice we never got to the point where the model would invent a citation outside the whitelist. That's not a bad thing; on the contrary, it shows the defense working in two layers. The first layer cuts at retrieval. The second kicks in if the model still returns an invalid citation. The proof for that second layer comes not from the Azure study but from unit and handler integration tests: tests like CitationValidator_UnknownCitation_ReturnsUnknownOutcome and Handler_ModelReturnsUnknownCitation_ReturnsRefusal confirm that an answer is rejected when an ID outside the whitelist comes back.

I also need to state one limitation clearly: in this study, retrieval and answer generation were connected to real Azure services, but risk classification still ran on a deterministic keyword matcher. Words like "dose", "medication" or "patient" were treated as high risk. That was enough for now, but in production, risk classification would also need to move to a stronger model or a more comprehensive rule set.

A green test isn't enough#

The most instructive lesson in this part actually came from a mistake.

On the first run, the golden set had 20 cases, but the index had only a single document. Some tests appeared to pass, but they passed for the wrong reason. They were green because the system didn't answer, which meant I was measuring missing data, not correct refusal behavior.

Worse, when I artificially lowered the retrieval threshold, that single document started matching unrelated questions too. The system looked like it "worked", but the test data didn't represent reality.

Once I noticed, I rebuilt the index around the golden set. I loaded enough documents to genuinely test each category, mapped document IDs to the values the golden set expected, and raised the retrieval threshold back to a realistic level. In that setup, a 0.7 threshold on the semantic reranker score put the right chunk in first place for six out of the six cases I tested.

It taught me a clear lesson about evaluation:

A green test isn't enough. It has to be green for the right reason.

Otherwise you haven't proven the system is safe; you've proven your test set is incomplete.

InterludeWhat evaluation measures, and what it doesn't#

So far we've tested the system's most critical behavior: does it stop where it shouldn't answer? That question mattered, because it's where the safety side of the refusal and citation contract begins. No source, no answer. Role not sufficient, no context. Invalid citation, no result.

But that alone isn't enough. The system should answer some questions. Which raises a new one: when it does answer, is the answer actually good?

Checking that by hand only works up to a point. You can read ten, twenty, maybe thirty answers and say "that looks right to me". But for a system getting close to production, that method breaks down fast, because the problem doesn't always show up as an obvious hallucination.

Sometimes the answer looks perfectly fine. The tone is good, there are citations, the model sounds confident. But the retriever brought back wrong or incomplete context, and the model stayed faithful to that wrong context and produced a coherent answer. In that case the model may not have made anything up, yet the system still worked incorrectly.

That's why evaluation shouldn't be reduced to "did the model hallucinate?". A RAG-based system has two separate surfaces.

The first is retrieval. Here we want to know whether the system found the right sources, and whether it brought them back in the right order. The metrics for this usually revolve around context precision and context recall. Precision asks how much of the retrieved context was actually useful. Recall asks how much of the genuinely necessary context was captured.

The second layer is generation. Here the question changes: did the model use the context it was given correctly? Is the answer grounded in that context? Does it actually answer the question? Does it fill in the gaps on its own, or does it respect its limits where information is missing? Metrics like groundedness, faithfulness, relevance and completeness come in on this side.

One distinction needs to be made clearly here: seeing a citation is not, on its own, a reason to trust an answer. A source link or chunk ID in an answer is a good start, but it doesn't guarantee the model actually produced the answer from that source. Sometimes a model produces the answer from its own internal knowledge and then attaches a similar-looking chunk from the retrieval results as a citation. From the outside it looks cited, but the citation isn't the real basis of the answer. So on the evaluation side, checking "is there a citation?" isn't enough. Whether the citation is valid, whether it supports the answer, and whether the model actually stuck to the context it was given all need to be measured separately.

In our system this distinction matters even more, because it doesn't try to answer every question the way a standard RAG flow does. Here, refusal isn't an error; it's part of the design. So I split evaluation into two layers. The first was contract behavior: I measured whether the system stopped where it should. The second was answer quality: when the system did answer, I measured how well that answer was grounded in the given context and how completely it answered the question.

That split is critical to me. In a refusal-first system, if you only look at answer quality you miss the safety behavior. If you only look at refusal behavior, you can't see whether the system produces genuinely useful answers. You need to measure both together.

The last test layerHow good is the answer?#

In the first layer we measured where the system stops. But an AI system doesn't only have to be safe; where it should answer, it also has to be useful. So in the second layer I moved on to this question: when the system answers, is the answer really grounded in the context it was given, and does it adequately cover the question?

I didn't build a separate Python/RAGAS pipeline for this. The existing architecture was already on .NET and built on Microsoft.Extensions.AI, so I used Microsoft's Microsoft.Extensions.AI.Evaluation library for evaluation too. I added two evaluators to the existing xUnit-based evaluation project:

• GroundednessEvaluator: measures how well the answer is grounded in the given context.
• RelevanceTruthAndCompletenessEvaluator: looks at relevance, truth and completeness.

Both return a score from 1 to 5 and a short justification. There are three deliberate choices here.

First: I didn't set up a separate judge model. The model that produced the answers was gpt-4.1-mini, and the evaluation ran on the same deployment. That's not the ideal setup. In production it would be better to use a separate, preferably stronger and more stable judge model, because a model grading its own answers carries a risk of self-grading bias. I accepted that risk for cost and simplicity, and interpreted the results accordingly: these scores aren't absolute truth, they're a technical signal you can reproduce under the same conditions. The point is to teach how to fish.

Second: I didn't re-run search afterwards to get the context for groundedness. This is an important detail. During evaluation you need to answer "what context was the model given?". If I'd gathered it with a second search call, the context I evaluated could have differed from the context the model actually saw. So I captured the real message sent to the model, pulled the Retrieved chunks block out of it through an observer, and fed that to the groundedness evaluation. The context the judge saw was exactly the context the model saw when it produced the answer.

Third: I didn't trust a single run. LLM outputs can vary slightly even with the same input, so I ran every case three times instead of once and reported the mean and standard deviation. N=3 isn't a large statistical sample, and it shouldn't be read as a production benchmark. But for a pilot it gave a much healthier signal than a single run.

Here are the results:

At first glance the scores look high. I don't want to turn that into an inflated success story; as I said, the judge was the same model, and there may be some generosity in these scores. But high scores aren't meaningless either. Because the system was designed to be citation-first and refusal-first, the room the model had to speak without grounding had already been narrowed. High groundedness scores are consistent with what that design should produce: where the model does answer, it seems to stay largely tied to the context it was given.

What I found most valuable is that not every score is 5.00.

AC-004 in particular matters. The question was about the branch transfer procedure. Groundedness came back at 5.00: the model didn't go outside the given context or add a made-up step. But completeness came back at 2.00. That may look like a bad result, but that's exactly what makes it instructive. The model stayed faithful to the information it had, and the measurement also showed that the answer didn't describe the whole process. The system didn't try to fill in the missing information; it didn't confidently invent "here's the step-by-step procedure". The answer stayed within the limits of the context it was given.

That's very close to the essence of what I've been trying to build in this series: say what you know with a source, and don't pad what you don't.

AC-006 was similar. Its groundedness scores varied between 4 and 5 across runs, averaging 4.33. The judge's justification said the answer covered the main content of the form correctly but skipped some preparation details. This example also showed why you shouldn't trust a single run. With one run I might have seen a 5 and called it done; across three runs I saw the small fluctuation.

My takeaway: an evaluation score doesn't, on its own, say "the system is good" or "the system is bad". Its real value is in showing where the score drops, and why.

The cost of evaluation isn't just tokens#

Collecting these measurements was more work than I expected.

On the first attempt, my Azure OpenAI deployment's throughput was very low. I was running 8 cases three times each, and every run involved both answer generation and evaluator calls. That flow hit the rate limit quickly. The result was bad: the run took hours and produced no valid measurements, because a large share of the calls failed with 429 errors.

The harness did one thing right here: it didn't write fake scores for cases it couldn't measure. Results that couldn't be evaluated because of rate limiting were left as "not measured". I think that matters. An evaluation system's job isn't to flatter us or churn out tables, it's to produce measurements we can trust.

Then I temporarily raised the deployment's capacity, re-ran the tests and completed the measurements. When the work was done I lowered the capacity again (in fact, all the Azure resources have since been deleted).

It taught me a simple but important lesson: if you're serious about evaluation, you need to give it its own working budget. Not just money, but throughput, retry/backoff, caching, and the discipline not to publish bad measurements. In production, evaluation shouldn't be a test you run when you have spare time. It should be a separate quality gate that runs reliably whenever the prompt, model, retrieval or index changes, and that never presents something it couldn't measure as if it had.

Part 4What does production-grade evaluation look like?#

What we've done so far is important, but limited. In Layer 1 we saw, in binary terms, that the contract holds with a real model. In Layer 2 we measured quality where it does answer, using groundedness, relevance and completeness scores. But that's still a foundation. Production-grade evaluation asks for more, so in this part I want to be clear not about what I did, but specifically about what I haven't done yet.

1. Retrieval metrics#

I didn't measure context precision or context recall in this study. Not out of laziness: I didn't have enough ground-truth labels to measure them properly. To measure context recall, you need to mark in advance which chunks are genuinely needed for each question; you need a labeled dataset where you can say "for the correct answer to this question, these chunks should be used". My golden set wasn't prepared at that level, and the pilot index wasn't large enough for a broad, multi-document retrieval benchmark.

So I didn't make those metrics up. I think one of the worst things you can do in evaluation is present something you didn't measure as if you had. I left that part on the production roadmap. The next step is clear: a larger document set, expected-chunk labels for each question, and precision/recall measurement on retrieval.

2. Continuous evaluation in CI/CD#

For now I ran the evaluation by hand. That's acceptable for this study, but not enough for production, because in AI systems behavior doesn't only change when code changes: the prompt changes, the model changes, the retrieval index grows, the chunk structure changes, and the system's answering style drifts. In a mature setup, evaluation moves from a manual check to a part of the CI/CD pipeline. Thresholds like these can be checked automatically:

• response schema validity must be 100%,
• the citation rate for answers that require citations must not drop below a set threshold,
• refusal behavior in no-source cases must not break,
• role-restricted documents must never be visible to the wrong role,
• escalation behavior for high-risk questions must be preserved,
• the expected source should appear in the top few retrieval results.

These thresholds aren't set arbitrarily up front. They're first calibrated with pilot data, then re-run on every meaningful change. If a threshold is missed, the build shouldn't look green. The goal isn't to slow developers down; on the contrary, it's to show early where a prompt or model change broke the system.

3. Production monitoring and drift#

Offline evaluation is a photograph: it measures the system at a particular moment, with a particular dataset. But production is a living place. The knowledge base grows, user questions change, model versions get updated, the retrieval index gets rebuilt. A threshold that worked well yesterday can be too loose or too strict tomorrow.

That's why in production it isn't enough to ask "how many answers did we give?" or "how many errors did we get?". You need to track refusal reasons, citation success, quality signals like groundedness, and escalation rates over time. This system has the foundation for that: refusals aren't stored as one generic error, they're broken down by reason (no_source_refusal, malformed_response, model_self_refusal, invalid_citation). That breakdown matters: if no_source_refusal rises you look at retrieval or the knowledge base, if invalid_citation rises you look at the model's output and citation formatting, and if malformed_response rises you look at structured output and prompt compliance. Observability isn't just collecting logs; it's being able to understand why the system switched into safe mode.

4. Risk classification#

There's one more area I deliberately left limited in this study: risk classification. Retrieval and answer generation were connected to real Azure services, but the risk classifier still ran as a deterministic keyword matcher. That was enough for now, because what I really wanted to measure here was whether the refusal/citation contract breaks behind real retrieval and a real model. But I wouldn't leave risk classification this simple in production.

There are two ways forward: either build a more comprehensive, domain-specific rule set, or set up a separate model/evaluator layer for risk classification. Whichever you choose, the risk decision itself also needs to be measured. False negatives are expensive here: if a high-risk question falls into the normal answer flow, the system's most sensitive safety boundary gets weaker.

In conclusion#

I'm not claiming to have built a complete production evaluation system in this article. The more accurate statement is this: I've laid the first foundation for how a refusal-first AI system running behind real Azure services can be measured. That foundation has two parts: a contract evaluation that measures where the system stops, and a quality evaluation that measures how well-grounded it is when it answers. From here it matures with a larger ground-truth dataset, CI/CD integration, production drift monitoring and stronger risk classification.

But what we have now is no longer just an architecture that looks good. It's behavior that can be measured.

Summary#

In this series I tackled the same problem in three steps. In the first article I laid out the problem: in regulated industries, an unsourced answer isn't just a technical error, it becomes the responsibility of the product and the team. In the second I turned that problem into code, treating the refusal and citation contract as an application rule that doesn't stay at the prompt level and stops the answer when it's violated. In this third article I took the same contract out of the mock environment and measured it behind real Azure services.

The result was clear to me: the system stopped where it shouldn't answer, and where it did answer, I could measure how well the answer was grounded in the context. AC-004 showed it well: the model didn't step outside the context, but it didn't fill in missing information on its own either.

The most important lesson I take from this series: what makes an AI system safe isn't the model's good intentions. It's the system clearly defining when it will answer, when it will stop, and which source an answer has to rest on.

Closing#

In the first article I argued that good architecture reveals itself when the context changes. In this one I actually changed the context. I removed the mock services; Azure AI Search and Azure OpenAI came in. And still, the Application and Domain layers where the refusal and citation contract lives did not change.

Of course, this isn't a finished production evaluation system. Retrieval metrics, a larger ground-truth dataset, automated evaluation in CI/CD, production drift monitoring and stronger risk classification are still separate topics. But I believe this article crossed an important threshold. What I have now isn't just an architecture I can describe as "this is how I designed it"; it's a contract that has been tried behind real services, with its behavior measured.

For me, the essence of this series comes down to these lines:

If there's a source, answer. If there isn't, stop. Don't just write it into the prompt; enforce it in code, test it, measure it.