Feature: RAG

[sofiachavezb]

Summary

This PR implements the Retrieval-Augmented Generation (RAG) module in DashAI, enabling users to chat with their documents through a 4-stage pipeline: Document Loading, Chunking, Retrieval, and LLM Generation.

The RAG pipeline takes a user query, splits the user's documents into chunks, retrieves the top-k most relevant chunks for the query, injects them into a prompt template, and feeds the result to an LLM that generates a grounded response with cited sources.

Pipeline stages:

1. Document Loading — loads files from the database and hydrates them into in-memory document objects. Six file extensions are accepted, though only two have dedicated parsing:

   • txt: TxtDocument class, plain text parsing.
   • pdf: PDFDocument class, text extraction via textract (default) or PyPDF2.
   • md, rst, tex, csv: all mapped to TxtDocument and read as raw text with no structural parsing. CSV files are not parsed column-by-column; they are ingested as flat text.

2. Chunking — splits documents via CharacterChunkModel, TokenChunkModel, or RecursiveCharacterChunkModel. Chunks are persisted with a SHA-256 signature cache for reuse across sessions.

3. Retrieval — finds relevant chunks via Sparse (TF-IDF, BM25), Dense (embedding-based similarity), or Composite retrievers (Sequential, Parallel, MMR Reranker).

4. Generation — injects chunks into a multi-language prompt template (en/es/pt) and generates a response via an LLM.

Scope: ~25,400 lines added across 226 files (213 core code + docs + translations).

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Briefly list the important modified files and what was done.

Backend

• RAGPipeline: central orchestrator receiving dependencies via constructor injection. It does not construct factories, repositories, or loaders itself.
• RAGModelsFactory: Abstract Factory (GoF pattern) with 4 sub-factories, all following a lookup-or-create pattern: sort params by key, query DB for existing matching record, reuse if found, create if not.
  • PromptFactory: multi-language prompts (en/es/pt), supports custom templates.
  • ChunkingModelFactory: resolves the chunker, splits documents, persists chunks.
  • RetrieverFactory: handles sparse (TF-IDF, BM25), dense (via DenseEmbedding subclasses), and composite retrievers (recursive child creation).
  • LLMFactory: creates or looks up TextToTextGenerationTaskModel instances.
• Retriever hierarchy (Composite design pattern):
  • Unit retrievers: TFIDFRetriever, BM25Retriever, DenseEmbeddingRetriever.
  • Composite retrievers: SequentialRetriever (cascading fallback), ParallelRetriever (merge results), MMRRerankerRetriever.
  • Two-layer dense design: separates embedding (10 DenseEmbedding subclasses: BERT, DistilBERT, RoBERTa, E5, Gemma, Instructor, LaBSE, OpenAI, SentenceTransformer, HuggingFace) from retrieval (shared DenseEmbeddingRetriever class accepting any embedding via component_field).
• Retriever persistence: all retriever SQL isolated in RetrieverRepository (337 lines). The rag_retriever table acts as a bridge: every retriever gets one canonical identity record, with sub-tables for sparse/dense details and rag_retriever_child for composite links.
• Chunk set caching: get_or_create_chunk_set() computes a SHA-256 hash from (sorted_document_ids + sorted_chunking_config) for cross-session reuse.
• Job layer: GenerativeJob detects RAGTask via task_name at runtime and delegates to RAGJob.run(). RAGJob builds the full pipeline, runs inference, persists output.
• RAGTask: extends BaseGenerativeTask with USE_HISTORY = True for chat history folding into the message list.
• Typed dataclasses throughout: RAGPipelineConfig, ChunkReference, RAGGenerationOutput, *FactoryResult, SparsePersistence, DensePersistence.
• 15 new SQLAlchemy ORM models with UniqueConstraints and ondelete=CASCADE foreign keys.
• 4 Alembic migration files.
• ~30 RAG components registered in DashAI/back/initial_components.py (3 chunking models, 5 prompts, 7 retrievers, 10 dense embeddings, plus vectorizer and schema models).

Frontend

• Routing: SessionRouter checks session.task_name. If RAGTask, renders RAGSessionPage; otherwise renders the standard Generative view.
• RAGSessionSetup: 6-section wizard with collapsible accordions (Session Details, Documents, Chunking Strategy, Retriever Model, Prompt Template, Language Model).
• 3-panel layout: Documents panel (left), Setup/Chat view (center), Info bar/Params panel (right).
• Retriever presets: 3-card system — Keyword (BM25), Semantic (Dense), Hybrid (Sequential BM25 + Dense).
• Context window validation: chunk_size * top_k + prompt_tokens must not exceed the LLM's context window.
• Composite retriever builder: advanced modal for building Sequential, Parallel, and MMR Reranker configurations.
• Advanced configuration modals for chunking, retriever, generator, and custom prompts.
• 4 per-stage section components (ChunkingSection, RetrieverSection, GeneratorSection, PromptSection) and 17 shared components in components/generative/RAG/.
• TypeScript API layer (api/rag.ts, 295 lines) with full typing.

API Endpoints

No new endpoints were added. Existing generative session endpoints were extended with RAG-specific logic:

• POST /api/v1/generative-session: validates RAG sessions against RAGPipeline.SCHEMA, enforces non-empty documents list, persists with task_name=RAGTask and model_name=RAGPipeline.
• PUT /api/v1/generative-session/{id}: validates RAG session updates against RAGPipeline.SCHEMA.
• DELETE /api/v1/generative-session/{id}: cascades cleanup of dense/sparse retrievers, embedding matrices, chunking models, and on-disk files.
• POST /api/v1/generative-process: creates processes linked to RAG sessions.
• POST /api/v1/job: enqueues RAGJob via Huey async task queue.
• GET /api/v1/jobs/{id}: frontend polling for job completion.
• POST/GET/DELETE /api/v1/document: document upload, listing, and deletion.
• POST/GET/PUT /api/v1/prompt: RAG prompt creation, listing, and updates.

Database

15 new SQLAlchemy ORM models across 5 areas, all with UniqueConstraints and CASCADE deletes. Four Alembic migrations orchestrate the schema evolution.

Document & Chunk Storage

• document: uploaded files (txt, pdf, md, rst, tex, csv); extended with file_hash and relationships to chunks and embedding matrices.
• chunk: individual text fragments belonging to a chunk set and document.
• rag_chunk_set: canonical identity for a set of chunks, keyed by a SHA-256 signature (sorted_document_ids + sorted_chunking_params) so two sessions sharing the same documents and chunker reuse the same row.
• rag_chunk_set_document: many-to-many bridge between chunk sets and documents.

Pipeline Configuration

• rag_pipeline: central pipeline record, linked to a generative_session via session_id, with FKs to the configured chunking model, prompt, and generation model. Each pipeline has exactly one retriever.
• rag_prompt: prompt template deduplicated by (class_name, parameters).
• rag_chunking_model: chunker configuration deduplicated by (class_name, parameters).
• rag_generation_model: LLM configuration deduplicated by (class_name, parameters).

Retriever Hierarchy

• rag_retriever: canonical identity row for every retriever — unit (sparse/dense) or composite. Sub-tables reference this row via bridge_id. Composite children are stored in rag_retriever_child.
• rag_retriever_child: ordered child links for composite retrievers (Sequential, Parallel, MMR Reranker).
• rag_sparse_retriever: TF-IDF or BM25 details plus on-disk storage folder, deduplicated by (class_name, parameters, chunk_set_id).
• rag_dense_retriever: dense retriever details plus FK to rag_embedding_model, deduplicated by (class_name, parameters, chunk_set_id, embedding_model_id).
• rag_retriever_chunk_set: links a retriever to the chunk set it operates on.

Embedding Layer

• rag_embedding_model: embedding model config deduplicated by (class_name, parameters).
• rag_embedding_matrix: persisted embedding matrices on disk per (document, chunk_set, embedding_model), storing matrix_shape for loading at inference time.

Session-Document-Pipeline Links

• rag_document_pipeline_session_link: three-way bridge among documents, generative_sessions, and rag_pipelines, with three separate UniqueConstraints to prevent duplicates in any direction.

Documentation

7 new files in docs/rag/:

• README.md: documentation index.
• 01-overview.md: high-level overview, pipeline stages, architecture, component types, code locations.
• 02-backend-architecture.md: detailed backend — pipeline orchestration, factory pattern, retriever hierarchy, persistence, typed return values.
• 03-frontend-architecture.md: frontend routes, component tree, API endpoints, key features.
• 04-execution-flow.md: 12-step end-to-end flow.
• 05-known-limitations.md: performance constraints, concurrency caveats, intentional technical debt.
• 06-future-work.md: planned improvements — advanced retrieval paradigms, query transformation, configurable PDF parsing, improved documents library, tree-based retrievers, message-level document filtering, and multi-modal support.

---

Testing (optional)

Only add if there's something reviewers should verify.

---

Notes (optional)

Known Limitations

Documented in docs/rag/05-known-limitations.md. Key items:

• No streaming; the frontend waits for the full LLM response.
• No FAISS/HNSW indexing; dense retrieval uses O(n*dim) brute-force search.
• Chunk similarity matrices held in RAM.
• DB session held open during LLM inference.
• get_or_create_chunk_set() uses SELECT-then-INSERT without a lock (safe for single-user, race condition under concurrency).
• Intentional code duplication between GenerativeJob and RAGJob to keep each branch self-contained.
• Augmentation prompts registered but not wired into the pipeline.
• Type system caveats in prompt chunks and score_chunks abstract signatures.