Enterprise Multi-Agent RAG Platform

Architecture and methodology are described at a high level; production code and internal data are proprietary.

Role: Technical Lead / Architect  ·  Period: 2025.11 – Present  ·  Stack: Python, LangChain, LangGraph, Azure OpenAI, Azure AI Search, FastAPI, React

I architected an enterprise, domain-specific multi-agent RAG platform end-to-end, taking it from a single-agent pilot to a company-wide initiative.

• Origin — grew out of the earlier data-standardization system, promoted from a single-agent plan into a multi-agent flagship.
• What it does — turns fragmented internal knowledge into a queryable, cited assistant over shared Azure infrastructure.
• Three cooperating sub-agents — a knowledge QnA assistant, a data-standardization assistant, and a code-analysis agent.
• Status — live with ~30 practitioners and expanding company-wide; two of the three agents shipped two months ahead of target.

The three agents

Platform foundation

• RAG pipeline — Parent-Child + contextual chunking, hybrid search (BM25 + vector), child→parent mapping, and reranking to suppress hallucination; a LangChain → LangGraph → Agentic 3-stage orchestration roadmap.
• Evaluation & MLOps — LLM-as-judge auto-scoring (factuality, reasoning, out-of-scope, multi-turn) + architecture A/B benchmarking + metric logging for operations, cutting estimated cloud operating cost by ~32%.

Architecture

The platform’s sub-agents share a common foundation — a hybrid (BM25 + vector) RAG index with parent-child + contextual chunking and reranking — and a common evaluation loop. The knowledge QnA assistant runs a 9 sub-agent Self-RAG / CRAG loop with token streaming and citation; the data-standardization and code-analysis agents reuse the same grounding and orchestration. An LLM-as-judge stage scores every interaction and feeds quality signals back to each agent.

flowchart TB
    DOCS[Internal knowledge] --> IDX[(Hybrid RAG index<br/>BM25 + vector, parent-child)]
    Q[User query] --> ORCH[Orchestration / control plane]
    IDX --> ORCH
    ORCH --> A1[Knowledge QnA assistant<br/>9 sub-agent Self-RAG / CRAG]
    ORCH --> A2[Data-standardization assistant<br/>Rule + classifier + RAG]
    ORCH --> A3[Code-analysis agent<br/>self-built orchestration]
    A1 --> OUT[Grounded, cited output]
    A2 --> OUT
    A3 --> OUT
    OUT --> EVAL[LLM-as-judge evaluation]
    EVAL -. quality feedback .-> ORCH

Orchestration follows a deliberate LangChain → LangGraph → Agentic roadmap, so the control plane grows in capability without locking into a single framework.

Per-agent architecture

The three agents share the RAG foundation but run different orchestration strategies: the QnA assistant is corrective + Self-RAG, the standardization assistant pins a deterministic rule engine ahead of RAG to minimize autonomous judgment, and the code-analysis agent stacks Graph RAG under Agentic RAG.

Knowledge QnA — corrective + Self-RAG

flowchart LR
    Q[User query] --> R[Hybrid retrieval<br/>BM25 + vector]
    R --> G{Relevance grade}
    G -- fail --> RW[Rewrite /<br/>re-retrieve]
    RW --> R
    G -- pass --> A[Generate + cite]
    A --> SC{Self-check}
    SC -- revise --> A
    SC -- ok --> OUT[Streamed, cited answer]

Data standardization — Rule + ALBERT + RAG hybrid

flowchart LR
    M[Metadata field] --> RULE[Rule engine<br/>300+ rules, abbrev dict]
    M --> CLF[ALBERT domain classifier]
    M --> RAG[RAG recommender]
    RULE --> MRG[Merge + confidence]
    CLF --> MRG
    RAG --> MRG
    MRG --> AUD{Domain auditor<br/>LangGraph corrective}
    AUD -- revise --> RAG
    AUD -- ok --> OUT[3 metadata recommendations]

Code analysis — 2-layer RAG, Graph then Agentic

flowchart LR
    CB[~400K-line codebase] --> IDX[40K AST facts<br/>code graph, 11.7K nodes]
    IDX --> L1[Layer 1: Graph RAG<br/>code relations]
    L1 --> L2[Layer 2: Agentic RAG<br/>autonomous exploration]
    L2 --> OUT[Grounded answer<br/>with provenance]

Case study — self-built orchestration vs. Claude Code

Problem. A general-purpose CLI agent (Claude Code) could already answer internal questions, but its cost-per-query profile did not scale to company-wide adoption, and “it feels better” is not an argument a platform decision can rest on.

What changed. I benchmarked three architectures head-to-head — raw Claude Code, Claude Code wrapped in a metadata/skill harness, and a dedicated self-built orchestration around the RAG pipeline that keeps the control plane in-house — across 9 variants (including the latest Claude Sonnet 5) on a 51-question eval set with a 6-metric composite, treating the comparison as a designed experiment rather than a demo.

Composite = weighted sum of accuracy, no-hallucination, completeness, structure-traceability, relevance, and citation. Latency and cost are per query; the response-collection channel is in parentheses — sub = subscription CLI, API = direct API, int = interactive (efficiency not measured). The self-built orchestration answers in a single grounded API call, so it is both the cheapest and the fastest while topping the composite.

Result. The self-built, determinism-first orchestration won overall — a grounded mini-tier model leading frontier CLI agents on answer quality at up to ~17× lower cost per query, with the gap established by paired t-test · McNemar · Cohen’s d · bootstrap CI rather than impression — the evidence that justified moving from a single-agent pilot to a company-wide rollout. The winning pipeline is now nearing production deployment.

Why it matters

A self-built harness keeps the control plane in-house — three payoffs:

• Vendor flexibility — models are borrowed, but the control code is ours, so we are not locked to one provider’s stack.
• Cost control — determinism-first grounding runs on a cheap mini-tier model.
• Knowledge as a durable asset — captured in-house rather than rented from a single provider.

I made this case beyond my own team, too: across two Microsoft workshops I led the technical discussion and persuaded a Microsoft architect and seven engineers of the self-built orchestration approach over a general-purpose Copilot CLI.