RAG for Audit: Building Knowledge Bases of SAs, CARO, and Your Firm Methodology
The most common LLM failure mode in audit is hallucination — confidently citing "SA 530 paragraph 12" when the actual paragraph says something different. The fix isn't a smarter model. It's an architecture where the model retrieves the actual source text before answering — Retrieval-Augmented Generation (RAG).
This post is the practitioner's introduction to RAG for Indian audit work. What RAG is, how to build a basic system, what to put in your knowledge base, when RAG beats fine-tuning, and the production-grade considerations.
If you've followed the series — Multi-agent frameworks covered orchestration of multiple agents; RAG is the foundation that makes those agents factually grounded.
What RAG actually is
A naive LLM workflow:
User: "What does SA 530 say about sample size for tests of details?"
LLM: [generates from training] "SA 530 says..."
The LLM responds from its training data. It might be right. It might hallucinate. There's no way to know without verifying.
A RAG workflow:
User: "What does SA 530 say about sample size for tests of details?"
↓
[Retrieval]: System searches the SA 530 text + ICAI Implementation Guide.
Finds the 3 most relevant paragraphs.
↓
[Augmentation]: Those paragraphs are added to the LLM's prompt.
↓
LLM: [generates from prompt with source text] "Per SA 530 paragraph 9,
the sample size shall be sufficient..." [cites paragraphs verbatim]
↓
User: gets a cited, grounded answer
Three components: a retriever (finds relevant text), an augmenter (adds it to the prompt), a generator (the LLM that produces the final answer).
The retriever is the substantive innovation. It typically uses vector embeddings — a numerical representation of meaning that lets you search by similarity, not just keywords. "Sample size for substantive procedures" finds passages about "sample size for tests of details" even though the exact words don't match.
A minimum-viable RAG for audit knowledge
Here's what a basic RAG system looks like for an Indian CA firm. Most of this is buildable in a weekend.
1. Document corpus
What goes into the knowledge base:
- All Standards on Auditing (SA 200 - SA 720) — full text + ICAI Implementation Guides
- CARO 2020 Order — full text + ICAI Guidance Note on CARO 2020
- Companies Act 2013 — key chapters auditors reference (XII for audit, XIII for KMP, XIV for prevention of oppression, etc.)
- Income Tax Act 1961 — sections most relevant to tax audit (44AB, 269ST, 40A(3), 269SS, 269T, 271DA, etc.)
- Ind AS Standards — major ones (115, 116, 109, 19, 12, 36, 1, 16, 38, 8)
- Form 3CD — current notification version
- GST Act + Rules — sections + rules relevant to GSTR-9C and GST audit
- Your firm's methodology — engagement manual, working paper templates, sample observations
- Prior audit findings — anonymised, with the conclusions reached
Typical corpus size for an Indian audit firm: 3,000-10,000 pages of source documents. That's around 5-15 million tokens.
2. Chunking strategy
You can't embed entire 50-page documents — too large for the model to retrieve precisely. Documents are split into chunks (usually 500-1500 tokens each).
For audit content:
- SAs: chunk by paragraph. Each paragraph becomes a chunk. Preserves the "SA 530 paragraph 9" structure.
- CARO clauses: chunk by clause. Each of the 21 clauses is a chunk.
- Form 3CD: chunk by clause / sub-clause.
- Companies Act / IT Act: chunk by section. Some sections are long enough to split further.
Smart chunking matters more than people think. A bad chunking strategy (splitting SAs in the middle of paragraphs) makes retrieval miss the actual relevant text.
3. Embedding model
The embedder turns each chunk into a numerical vector. Options:
- OpenAI text-embedding-3-large — 3072 dimensions, ~₹10 per million tokens, US-hosted (DPDPA risk for your firm's confidential methodology). Best general quality.
- Voyage AI / Cohere — strong commercial alternatives, US-hosted.
- BGE-M3 (multilingual, open-source) — runs on your own infrastructure. Cost: GPU time for embedding generation. India-hostable.
- Indian-language models — for Hindi / regional content, but most CA work is English so this is minor.
For a serious audit firm building RAG, open-source India-hosted embedders are the right answer for DPDPA + audit-trail reasons. The performance gap vs commercial embedders has narrowed substantially in 2024-2026.
4. Vector database
The store that holds embeddings + supports similarity search. Options:
- Pinecone — easy, managed, US-hosted (DPDPA concern for confidential content)
- Qdrant — open-source, self-hostable. India deployment possible.
- Weaviate — open-source + managed options
- ChromaDB — open-source, simple, in-process
- PostgreSQL + pgvector — your existing Postgres setup gets vector capability. Lowest friction for firms with existing data infrastructure.
For DPDPA + cost, Qdrant or pgvector self-hosted on India cloud is the smart choice. Pinecone is fast to start with but the US-hosting is a concern for client-data adjacent content.
5. Retrieval logic
Given a user question, you retrieve the K most-relevant chunks (typically K=5-10). Some refinements:
- Hybrid search — combine vector similarity + keyword (BM25) search. Catches exact-term matches (e.g., specific section numbers) that vector search might miss.
- Re-ranking — use a smaller model to score the top 20 results and pick the best 5.
- Metadata filtering — restrict search to specific document types (e.g., "only search SAs" or "only search firm methodology")
6. Generation
Pass the retrieved chunks + the user question to the LLM. Prompt template:
You are an audit professional. Answer the user's question based ONLY on
the following source text. If the source text doesn't contain the answer,
say so. Cite the specific source (SA number + paragraph, or section +
clause) for every claim.
Source text:
[retrieved chunks here]
User question: [user question]
Answer:
The model produces the answer constrained to the retrieved source. Hallucination probability drops dramatically.
Why RAG beats fine-tuning for most audit use cases
A common question: "Should I fine-tune an LLM on my firm's methodology, or use RAG?"
For most Indian CA firms, RAG beats fine-tuning in 3 ways:
1. Updatability
When CARO 2020 is amended, you update the document in the RAG corpus. The system immediately reflects the change.
A fine-tuned model needs retraining to absorb the change. Fine-tuning takes hours of GPU time and engineering effort.
2. Citability
RAG answers come with explicit source references. The auditor can verify the citation against the actual document.
A fine-tuned model's answer comes from absorbed knowledge — there's no source to cite, no way to verify other than checking the source manually.
For SA 230 documentation purposes, RAG's explicit-citation is critical. The working paper says "Per SA 530 paragraph 9 [verified]" not "the AI said the answer is X."
3. Cost
Fine-tuning a 70B-parameter model in India costs ₹2-5 lakh in GPU time. The fine-tuned weights need ongoing maintenance.
RAG infrastructure costs roughly ₹50K-2 lakh / year for a mid-size firm (vector DB + embedder + LLM API or self-hosted). Updates are cheap.
When fine-tuning beats RAG
Fine-tuning has its place — for style consistency. If you want the LLM to write CARO observations in your firm's exact tone, fine-tuning on 200 examples of your firm's prior observations can produce a model that consistently matches your style.
But for content accuracy (the substantive citations), RAG is the right answer.
The smart architecture: RAG for content + light fine-tuning for style.
Common RAG failure modes in audit
Six things go wrong when CA firms first build RAG systems:
1. Bad chunking
Chunking SAs in the middle of paragraphs makes retrieval miss obvious context. Fix: chunk by paragraph for SAs, by clause for CARO, by section for Acts.
2. Embedding model mismatch
Using a general-purpose embedder on dense regulatory text gives mediocre retrieval. Fix: test multiple embedding models on a held-out set of audit questions. Measure retrieval recall (does the relevant chunk appear in top 10?). Pick the best.
3. No re-ranking
Top-10 retrieved chunks may include irrelevant ones that crowd out the actual answer in the prompt. Fix: add a re-ranker — a small model that scores chunk relevance to the question and keeps only the top 3-5.
4. Stale corpus
If you build the RAG in March 2026 and never update it, by March 2027 the SAs may have been amended, the IT Act 2025 may be in force, CBDT may have revised Form 3CD. Fix: documented refresh process. Re-embed amended documents quarterly.
5. No source filtering
A question about CARO 2020 retrieves chunks from the CARO 2016 archive (which you also indexed) — and the LLM produces an obsolete answer. Fix: metadata filtering by effective date. Always restrict to current-version sources.
6. Treating RAG output as authoritative
Even with RAG, the LLM can misinterpret the retrieved text. The auditor must verify. Fix: the working paper records BOTH the question and the retrieved source text, not just the LLM's answer. Five years later, the auditor (or peer reviewer) can re-verify.
A practical RAG implementation outline (Python + open-source)
For a CA firm wanting to build a minimal RAG system without buying a vendor solution:
1. Document collection (manual or scraping)
- SAs from ICAI website (PDFs)
- CARO 2020 from MCA
- Acts from indiacode.nic.in
- Your firm methodology (internal docs)
2. Chunking
- Use LangChain's RecursiveCharacterTextSplitter
- 1000-token chunks with 100-token overlap
- Preserve SA paragraph / CARO clause boundaries via custom splitter
3. Embedding
- BGE-M3 (open-source, India-hostable) via Hugging Face
- Run on local GPU or Indian cloud GPU (E2E, Yotta, Azure South India)
- Embed all chunks; store vectors + chunk text + metadata
4. Vector DB
- Qdrant on Indian cloud, self-hosted
- Or PostgreSQL + pgvector if you have an existing DB
5. Retrieval
- Hybrid: vector similarity + BM25 keyword
- Top K = 10
- Re-rank to top 5 via cross-encoder (also BGE-based)
6. Generation
- LLM: Llama 3.3 70B or Qwen 2.5 72B (open-source, India-hosted)
- Or: Claude / GPT-4o if you accept US hosting for non-client-data queries
- Prompt template as above
7. Evaluation
- Hold-out set of 50-100 audit questions with known correct answers
- Measure retrieval recall + answer accuracy
- Iterate on chunking + embedding + retrieval until quality is high
Build time: 4-6 weekends for a competent Python engineer + an audit SME working together. Infrastructure cost: ₹15K-50K / month for self-hosted setup serving a mid-sized firm.
For most firms, the build cost (engineering time) exceeds the value of building it yourself. A vendor-provided RAG-on-audit-content is the better path — see CORAA's University which includes the SA library, CARO clause-by-clause, and the calculator stack all built on RAG.
RAG + multi-agent + open-source LLM = audit-grade AI stack
The full architecture for a serious Indian audit firm wanting AI-native operations:
[Open-source LLMs hosted in India]
↓
[RAG over: SAs + CARO + Acts + Firm methodology]
↓
[Multi-agent orchestration: Triage / Substantive / Reporting agents]
↓
[Audit-tech integration: working papers, evidence trail]
↓
[Partner review + sign-off with UDIN]
Each layer matters:
- Open-source LLMs for India hosting + DPDPA + cost predictability (see next post in this series)
- RAG for factual grounding + citability
- Multi-agent for workflow orchestration
- Audit-tech integration for the SA 230 audit trail
- Partner review for professional judgement and accountability
CORAA is one instantiation of this stack. Other audit-tech vendors are building similar architectures. The next 2-3 years will see this become standard for Indian audit firms — not just optional.
Bottom line
RAG transforms LLMs from "confidently guessing" to "answering with citations". For audit work — where every claim needs a verifiable source — RAG is foundational, not optional.
For Indian CA firms:
- Use RAG-enabled audit tools for any work that touches statutes, standards, or firm methodology. The citation requirement is non-negotiable for SA 230.
- Don't fine-tune for content accuracy — RAG is faster, cheaper, and more updatable. Fine-tune for style consistency only.
- Self-host the RAG stack if you have engineering capacity — better DPDPA posture, lower per-user cost at scale.
- Or use a vendor (CORAA-style) that has built the stack — faster to deploy, lower upfront cost, ongoing maintenance handled.
The next post in this series — Hosting Your Own Open-Source LLM for Audit: The India Cost / ROI Math — covers the infrastructure layer that makes self-hosted RAG feasible.
Try CORAA → RAG over Indian audit content, multi-agent orchestration, India-hosted, audit-trail-by-default. See pricing · Browse calculators · Trust Centre · AI Lab.