We built a retrieval assistant that answers governance questions inside Microsoft Teams, grounded strictly in the source documents and cited back to the paragraph it came from. Here is the architecture, the design decisions, and the engineering that makes an answer trustworthy enough to rely on.
Every organisation with a compliance function owns the same inefficiency. The policies, standards and procedures are thorough and well written, and almost nobody reads them end to end. Instead, staff ask the same handful of questions on repeat, and those questions land on the desk of whoever owns the documents rather than being answered from the documents themselves. The expert becomes a lookup service, and the knowledge that should be self serve stays locked in a person’s calendar.
The fix is not another portal or a better search box. It is an assistant that reads the source material for you and answers in plain language, in the tool your people already live in. We built exactly that: a compliance assistant in Microsoft Teams that responds to a question in seconds, draws its answer only from the approved documents, and shows the citation so the reader can verify it. This article is the account of how it works and why each decision was made the way it was, written to be useful whether you sign the budget or write the code.
What it does
Ask a question in Teams. The assistant retrieves the most relevant passages from your governance library, composes an answer grounded strictly in that text, and cites the source document. If the answer is not in the corpus, it says so rather than inventing one.
A general purpose chatbot is a confident generalist. It will answer a policy question with something that sounds right, assembled from the open internet and its training data. In compliance, that is precisely the wrong behaviour. An answer that is plausible but not traceable to your actual policy is worse than no answer, because someone will act on it.
Retrieval augmented generation, or RAG, changes the contract. Rather than answering from memory, the model is handed the specific passages from your documents that relate to the question, and instructed to answer only from those passages. The intelligence is in the retrieval and the discipline, not in the model’s opinions. The result is an assistant whose answers you can audit, because every one points back to the paragraph it came from.
The system has two halves that rarely run at the same time. Ingestion is the one time job of turning documents into a searchable knowledge base, repeated only when a document changes. The query path is what runs on every question, and it is where speed, access control and grounding all have to hold together.
On ingestion, source documents are loaded from blob storage and split into overlapping passages of roughly 500 characters, with 50 characters of overlap so a sentence is never cut cleanly in two across a boundary. Each passage is converted into a numerical vector that captures its meaning, a 1536-dimension embedding produced by OpenAI’s text-embedding-ada-002 model, and stored in a Chroma vector database tagged with its source document and a reference for citation. The overlap and the tagging are not incidental details. Overlap keeps an answer that straddles two passages intact, and the tag is what lets the assistant tell the reader where an answer came from.
On every question, the flow is deliberate: Teams passes the message to the bot service, which routes it to the API, which first checks whether the user is allowed to ask. Only then does the system embed the question, retrieve the closest passages, and hand them to the model to compose a grounded, cited answer. The ordering matters, and it is the next thing worth dwelling on.
The most common way to bolt authorisation onto a RAG system is to filter results after retrieval, or to trim the answer before it is returned. Both are weaker than they look, because by then the query has already touched the knowledge base and the model. We put the access gate at the front of the API layer, before a single passage is retrieved. If a request is not authorised, it never reaches the vector store and never reaches the model.
This is cleaner, cheaper and easier to audit. Cleaner because the sensitive data path has one entrance and one guard. Cheaper because rejected requests cost nothing in retrieval or model calls. More auditable because the decision to allow or refuse is a single logged event at a single point, rather than an emergent property of several downstream filters. When a security review asks how you prevent unauthorised access, the answer is a location on a diagram, not a paragraph of caveats.
Grounding is enforced by the instructions the model receives with every question. The prompt is deliberately strict, and two of its rules are load bearing. The rest of the prompt is housekeeping, but these two are the difference between a tool a compliance team will trust and one they will stop using.
Use context even if not labelled exactly as asked. Without this, the model refuses a perfectly relevant passage because its wording does not match the question word for word. Cross document questions, where the answer lives under different terminology from the one the user typed, are the first casualty. This rule keeps the assistant useful across a real corpus rather than only on questions phrased the way the documents happen to be written.
Do not add examples not explicitly stated in the context. This is the hallucination guard. Left unchecked, a model will helpfully invent a plausible policy detail that appears in none of your documents, and it will read exactly like a correct answer. That is what makes it dangerous. This single instruction closes the most damaging failure mode a compliance assistant has.
One more setting underpins all of it. The model’s temperature is fixed at zero, so the same question returns the same answer every time rather than a different phrasing on each run. In a compliance context, reproducibility is not a nicety. It is what lets you stand behind an answer a second time.
The principle
A compliance assistant should be measured on whether it can be trusted, not on whether it is clever. Every design choice here trades a little breadth for a lot of reliability, and in this domain that is the correct trade.
An assistant that cannot be measured cannot be improved or defended. We evaluate against a fixed set of test questions using six metrics, and we optimise for the one that reflects trust rather than the one that flatters the demo. Manual spot checking will miss the failures that matter; an automated evaluation harness surfaces them systematically, on every change.
Faithfulness, at 0.98, is the metric we optimise for. It measures whether the answer is fully supported by the retrieved context, which is another way of asking whether the assistant is making anything up. In compliance, an answer that invents nothing is worth more than an answer that retrieves every possible relevant sentence. Mean reciprocal rank matters too, because it rewards the most relevant passage landing near the top of the context rather than buried in the middle, where models reliably pay it less attention.
Recall at five, at 0.41 against a target of 0.70, looks low, and that is honest, not broken. Recall at five measures how many of the relevant passages appear in the top five retrieved. When a single question can have a dozen or more relevant passages across the corpus, perfect recall at five is arithmetically impossible. Measuring at a larger k to make the number look better would flatter the metric without improving the product, because five is the depth the assistant actually reads from. We set the target against the k we use in production and report it straight. A team that inflates recall is optimising the slide, not the assistant.
Moving a RAG system from a working prototype to a reliable Teams deployment on Azure surfaces a small set of failure modes that are easy to miss and quick to fix once you recognise them. We instrument for all four before first deploy, so a broken deployment identifies itself in under a minute rather than after an afternoon of guessing.
The vector store folder is excluded by .gitignore, so the pipeline ships a working application with no retrieval data and only fallback answers. Assert the passage count on startup; a count of zero locates the fault at once.
The vector store needs SQLite 3.35 or newer, but the Linux App Service runtime ships an older build, so startup fails. Pin pysqlite3-binary with a Linux-only, platform-conditional override.
The tenant is not inferred automatically, so the Bot Service will not connect. Set channel_auth_tenant explicitly on the adapter before first deploy.
With no signed-in user, a delegated permission returns a 403 on the group check. Use an application permission with admin consent instead.
The most insidious of the four is the empty knowledge base. If the vector store folder is excluded by version control, the pipeline deploys a working application with no retrieval data. The bot runs, returns its fallback answer, and raises no obvious error, which is the worst kind of failure because everything looks healthy. Logging the passage count on startup turns a silent failure into a one line diagnosis: a count of zero tells you instantly whether the fault is in retrieval or generation. The same discipline applies to the other three. Instrument the startup sequence, not just the query path, and the system tells you where it hurts.
The principles below shaped this build and will shape the next one. They are the short version of everything above.
An answer that invents a policy detail is worse than no answer. Get faithfulness right, then improve recall. The order is not negotiable in compliance.
Refusing an unauthorised request at the API layer, before any query reaches the store or the model, is cleaner, cheaper and more auditable than any downstream filter.
Logging the passage count, the authentication tenant and the Graph API response on startup locates a broken deployment in under a minute.
A local vector store instead of a fully managed search service saves budget at launch without touching the retrieval logic. Design the store as a swappable component and the upgrade is a single change when the budget arrives.
Question and answer is version one, not the destination. The same retrieval foundation that answers on demand can push proactively: upcoming audit reminders, non conformity tracking, and alerts on approaching review dates, surfaced in Teams without anyone having to ask. The retrieval layer built today is the groundwork for a compliance function that reaches out rather than waiting to be queried. That is the change worth planning for, and it starts with getting the boring, trustworthy foundations right.
We design and build assistants that your teams can actually rely on, grounded in your own material and measured against the metrics that matter. If you have a document heavy process that runs on people answering the same questions, there is a version of this for you.
“Your digital partner bridging strategy, technology, and human experience”
We gather information about your needs and objectives of your apps. Unsure about the app you need? We will carefully assess your top challenges and provide expert guidance on the perfect solution tailored to your success.
We create wireframes and an interactive prototype to visualise the app flow and make changes as per your feedback.
Estimation of the project deliverables including the resources, time, and costs involved.
Showcasing POC to relevant stakeholders illustrating the functionalities and potential of the app to meet business objectives.
