From 8e8891fd5adad9391bc6763034f08f1fb62a2a41 Mon Sep 17 00:00:00 2001 From: Robert Shelton Date: Wed, 24 Jun 2026 09:26:40 -0400 Subject: [PATCH 1/2] feat: add FT.HYBRID support via hybrid_vector_search() Translate hybrid_vector_search(cosine_distance(...), fulltext(...), rrf()|linear()) into a native FT.HYBRID command (Redis 8.4+), fusing a text query and a vector query server-side with RRF or LINEAR fusion. Adds support across the parser, analyzer, translator, and executor layers, a Redis 8.4 version guard, and FT.INFO / hybrid-reply parsing that handles both the RESP2 list and redis-py 8.x RESP3 map shapes. Bumps the test Redis image to 8.4. Jira: RAAE-1322 --- docs/proposals/README.md | 137 +++ docs/proposals/ft-hybrid-code-comparison.md | 223 +++++ .../ft-hybrid-implementation-sketch.md | 402 +++++++++ docs/proposals/ft-hybrid-primitive-design.md | 295 +++++++ docs/proposals/ft-hybrid.md | 295 +++++++ sql_redis/analyzer.py | 47 + sql_redis/executor.py | 102 ++- sql_redis/parser.py | 201 ++++- sql_redis/schema.py | 91 +- sql_redis/translator.py | 123 ++- tests/conftest.py | 8 +- tests/test_ft_hybrid.py | 821 ++++++++++++++++++ tests/test_schema_registry.py | 36 + 13 files changed, 2732 insertions(+), 49 deletions(-) create mode 100644 docs/proposals/README.md create mode 100644 docs/proposals/ft-hybrid-code-comparison.md create mode 100644 docs/proposals/ft-hybrid-implementation-sketch.md create mode 100644 docs/proposals/ft-hybrid-primitive-design.md create mode 100644 docs/proposals/ft-hybrid.md create mode 100644 tests/test_ft_hybrid.py diff --git a/docs/proposals/README.md b/docs/proposals/README.md new file mode 100644 index 0000000..81e9a49 --- /dev/null +++ b/docs/proposals/README.md @@ -0,0 +1,137 @@ +# FT.HYBRID Proposal Documents + +**JIRA:** [RAAE-1322](https://redislabs.atlassian.net/browse/RAAE-1322) +**Status:** Design recommendation phase +**Date:** 2026-06-23 + +## Overview + +This directory contains design documents for adding `FT.HYBRID` support to sql-redis. The proposal introduces a new primitive abstraction that cleanly expresses server-side hybrid fusion (text + vector search combined via RRF or LINEAR). + +## Documents + +### 1. [ft-hybrid.md](ft-hybrid.md) - Original Proposal +**Author:** Robert Shelton +**Purpose:** Initial spec with three syntax designs (A, B, C) + +Contains: +- Goal: Enable server-side fusion (RRF/LINEAR) instead of filter-then-KNN +- Three syntax designs (A: overload, B: hybrid() predicate, C: composable) +- SQL → FT.HYBRID mapping +- Implementation plan by layer +- Testing strategy + +**Key decision point:** Design C (composable, fusion in ORDER BY) was recommended. + +### 2. [ft-hybrid-primitive-design.md](ft-hybrid-primitive-design.md) - Design Recommendation ⭐ +**Author:** Claude (review of original) +**Purpose:** Identify the core design issue and propose a better primitive + +**Main insight:** The original designs treat hybrid fusion as a **syntax mapping problem** rather than a **data structure problem**. This document proposes: + +- **New primitive:** `HybridFusionSpec` dataclass that encapsulates text leg + vector leg + fusion config +- **Design C+:** Keep Design C's SQL syntax but use dedicated primitives internally +- **Type-driven dispatch:** Command path is determined by data structure, not heuristics + +**Key benefits:** +- Single source of truth for fusion state +- Explicit over implicit +- Clean extensibility (add fusion methods, add legs) +- No risk of breaking existing filter-then-KNN + +### 3. [ft-hybrid-code-comparison.md](ft-hybrid-code-comparison.md) - Concrete Comparison +**Purpose:** Side-by-side code showing Design C vs Design C+ + +Compares: +- Parser output (scattered state vs single spec) +- Translator dispatch (heuristics vs type-driven) +- Command builder (extraction logic vs direct access) + +**Takeaway:** Design C+ is cleaner, more maintainable, and safer. + +### 4. [ft-hybrid-implementation-sketch.md](ft-hybrid-implementation-sketch.md) - Code Sketch +**Purpose:** Concrete implementation examples for Design C+ + +Shows: +- Data class definitions (TextRankingLeg, VectorRankingLeg, HybridFusionSpec) +- Parser detection logic (_process_order_by, _build_hybrid_fusion_spec) +- Analyzer validation (_analyze_hybrid_fusion) +- Translator command builder (_build_ft_hybrid) +- Executor version gating (_check_hybrid_support) + +**Takeaway:** All changes are additive and backward compatible. + +## Recommended Path Forward + +**Adopt Design C+ (primitive-based) for the following reasons:** + +1. **Better data model:** `HybridFusionSpec` makes fusion first-class, not inferred. +2. **Cleaner code:** No heuristics, no scattered state, no conditional detection. +3. **Safer extension:** Adding new fusion methods or legs doesn't require refactoring. +4. **Backward compatible:** Existing vector_distance() queries work unchanged. + +## SQL Syntax (Final Recommendation) + +```sql +SELECT page_text, file_id, + vector_distance(embedding, :vec) AS vscore, + fulltext(page_text, 'quarterly earnings') AS tscore +FROM "KM_abc123" +WHERE ticker = 'MSFT' +ORDER BY rrf(vscore, tscore, constant => 60) DESC +LIMIT 10; +``` + +**Detection rules:** +- `vector_distance(...) AS ` in SELECT → vector leg +- `fulltext(...) AS ` in SELECT → text leg +- `rrf()` or `linear()` in ORDER BY → fusion trigger + +If all three are present → `FT.HYBRID`. +If only vector_distance → `FT.SEARCH` with KNN (existing behavior). + +## Implementation Checklist + +- [ ] Add `HybridFusionSpec`, `TextRankingLeg`, `VectorRankingLeg` to parser.py +- [ ] Add detection logic in `SQLParser._process_order_by()` +- [ ] Add `HybridFusionAnalysis` to analyzer.py with field type validation +- [ ] Add `Translator._build_ft_hybrid()` command builder +- [ ] Add version gating in `Executor.execute()` (Redis 8.4+ check) +- [ ] Bump test Redis image to 8.4+ in conftest.py +- [ ] Add unit tests for parser, analyzer, translator +- [ ] Add integration tests in test_sql_queries.py +- [ ] Update docs: relabel "hybrid" → "filtered KNN", add "Hybrid Fusion" guide +- [ ] Update AGENTS.md and llms.txt with FT.HYBRID info + +**Estimated effort:** 3-4 days (implementation + tests + docs) + +## Open Questions + +1. **Fusion defaults:** Require explicit `rrf()/linear()` or default to RRF? + **Recommendation:** Require explicit (fail fast on ambiguity). + +2. **K vs WINDOW:** Default `K = window` or make independent? + **Recommendation:** Default `K = window` for v1; add kwarg later if needed. + +3. **Filter distribution:** Apply WHERE to both legs or SEARCH-only? + **Recommendation:** Both legs (consistency > simplicity). + +4. **Score surfacing:** Auto-project fused score or require explicit SELECT? + **Recommendation:** Let users SELECT it explicitly if needed. + +## Next Steps + +1. Confirm design decision (C+ vs C) with stakeholders. +2. Confirm Redis 8.4 availability in test environment. +3. Begin implementation with parser/analyzer/translator changes. +4. Add tests at each layer (unit → integration). +5. Update docs and deploy. + +## Related Issues + +- **Terminology fix:** Existing docs call filter-then-KNN "hybrid search" but that's not `FT.HYBRID`. Need to relabel to "filtered KNN" vs "hybrid fusion". +- **RedisVL alignment:** If RedisVL adds FT.HYBRID support, mirror parameter names (constant, window, alpha, beta). + +--- + +**Questions?** See the individual docs above or reach out on RAAE-1322. diff --git a/docs/proposals/ft-hybrid-code-comparison.md b/docs/proposals/ft-hybrid-code-comparison.md new file mode 100644 index 0000000..67f860c --- /dev/null +++ b/docs/proposals/ft-hybrid-code-comparison.md @@ -0,0 +1,223 @@ +# FT.HYBRID: Design C vs Design C+ Code Comparison + +**Purpose:** Show concrete code differences between the original Design C (detection-based) and the recommended Design C+ (primitive-based). + +## Scenario: Parse this SQL + +```sql +SELECT page_text, file_id, + vector_distance(embedding, :vec) AS vscore, + fulltext(page_text, 'quarterly earnings') AS tscore +FROM "KM_abc123" +WHERE ticker = 'MSFT' +ORDER BY rrf(vscore, tscore, constant => 60) DESC +LIMIT 10; +``` + +--- + +## Design C (Original): Detection-Based + +### Parser Output + +```python +ParsedQuery( + index="KM_abc123", + fields=["page_text", "file_id"], + vector_search=VectorSearchSpec( + field="embedding", + alias="vscore", + k=None + ), + conditions=[ + Condition(field="page_text", operator="FULLTEXT", value="quarterly earnings"), + Condition(field="ticker", operator="=", value="MSFT") + ], + orderby_fields=[("vscore", "DESC")], # ??? How to represent rrf(vscore, tscore)? + # Problem: No clear place to store fusion method + parameters + # Requires adding fields like: + # fusion_method: str | None = None + # fusion_text_alias: str | None = None + # fusion_params: dict | None = None +) +``` + +**Issues:** +1. `fulltext()` in SELECT is represented as a `Condition`, even though it's not a filter. +2. Fusion method (`rrf`) and its parameters are scattered across multiple optional fields. +3. `orderby_fields` can't represent `rrf(vscore, tscore)` cleanly. +4. Analyzer needs heuristics to detect "this is fusion, not filter-then-KNN." + +### Translator Logic + +```python +def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + # Heuristic detection: if we have vector_search AND a FULLTEXT condition + # that's not in WHERE, assume hybrid fusion + has_text_in_select = any( + c.operator == "FULLTEXT" and c not in analyzed.filters + for c in analyzed.parsed.conditions + ) + + if analyzed.vector_search and has_text_in_select: + # Infer this is FT.HYBRID + if analyzed.parsed.fusion_method == "RRF": + return self._build_ft_hybrid_rrf(analyzed) + elif analyzed.parsed.fusion_method == "LINEAR": + return self._build_ft_hybrid_linear(analyzed) + else: + raise ValueError("Fusion method not detected") + elif analyzed.vector_search: + # Filter-then-KNN + return self._build_ft_search(analyzed) + # ... +``` + +**Problems:** +- Lots of conditional logic to distinguish fusion from filter-then-KNN. +- Hard to extend when adding new fusion methods. +- Fragile: what if someone writes `fulltext()` in SELECT but doesn't want fusion? + +--- + +## Design C+ (Recommended): Primitive-Based + +### Parser Output + +```python +ParsedQuery( + index="KM_abc123", + fields=["page_text", "file_id"], + hybrid_fusion=HybridFusionSpec( + text_leg=TextRankingLeg( + field="page_text", + query="quarterly earnings", + alias="tscore", + scorer="BM25" + ), + vector_leg=VectorRankingLeg( + field="embedding", + alias="vscore", + k=None + ), + fusion_method="RRF", + rrf_constant=60, + window=20 + ), + conditions=[ + Condition(field="ticker", operator="=", value="MSFT") + ] +) +``` + +**Benefits:** +1. All hybrid fusion state lives in one object. +2. `fulltext()` is explicitly a ranking signal, not a filter. +3. Filters are separate from ranking legs. +4. No ambiguity: if `hybrid_fusion` is set, it's `FT.HYBRID`. + +### Translator Logic + +```python +def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + if analyzed.hybrid_fusion: + return self._build_ft_hybrid(analyzed) + elif analyzed.vector_search: + return self._build_ft_search(analyzed) + elif analyzed.use_aggregate: + return self._build_ft_aggregate(analyzed) + else: + return self._build_ft_search(analyzed) +``` + +**Benefits:** +- **Type-driven dispatch:** the data structure dictates the command path. +- No heuristics, no inference, no scattered state. +- Easy to add new fusion methods: just add a field to `HybridFusionSpec`. + +--- + +## Side-by-Side: Building FT.HYBRID Command + +### Design C (Detection-Based) + +```python +def _build_ft_hybrid_rrf(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + # Extract text query from conditions + text_cond = next(c for c in analyzed.parsed.conditions if c.operator == "FULLTEXT") + text_field = text_cond.field + text_query = text_cond.value + text_alias = analyzed.parsed.fusion_text_alias or "tscore" + + # Extract vector field from vector_search + vector_field = analyzed.vector_search.field + vector_alias = analyzed.vector_search.alias or "vscore" + + # Extract fusion params + rrf_constant = analyzed.parsed.fusion_params.get("constant", 60) + window = analyzed.parsed.fusion_params.get("window", 20) + + # Build filters + filters = [c for c in analyzed.parsed.conditions if c.operator != "FULLTEXT"] + filter_expr = self._build_filter_expr(filters) + + # Assemble command + search_leg = f'SEARCH "(@{text_field}:({text_query})) {filter_expr}" YIELD_SCORE_AS {text_alias}' + vsim_leg = f'VSIM @{vector_field} $vec FILTER 1 "{filter_expr}" KNN 2 K {window} YIELD_SCORE_AS {vector_alias}' + combine = f'COMBINE RRF 2 CONSTANT {rrf_constant} WINDOW {window}' + # ... +``` + +**Problems:** +- Scattered extraction logic. +- Fragile assumptions (e.g., "first FULLTEXT condition is the ranking signal"). +- Duplicate filter-building logic. + +### Design C+ (Primitive-Based) + +```python +def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + fusion = analyzed.hybrid_fusion.spec + + # All fusion state is in one place + text_field = fusion.text_leg.field + text_query = fusion.text_leg.query + text_alias = fusion.text_leg.alias + + vector_field = fusion.vector_leg.field + vector_alias = fusion.vector_leg.alias + k = fusion.vector_leg.k or fusion.window + + # Filters are already separated + filter_expr = self._build_filter_expr(analyzed.hybrid_fusion.filters) + + # Assemble command + search_leg = f'SEARCH "(@{text_field}:({text_query})) {filter_expr}" YIELD_SCORE_AS {text_alias}' + vsim_leg = f'VSIM @{vector_field} $vec FILTER 1 "{filter_expr}" KNN 2 K {k} YIELD_SCORE_AS {vector_alias}' + + if fusion.fusion_method == "RRF": + combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' + else: # LINEAR + combine = f'COMBINE LINEAR 2 ALPHA {fusion.linear_alpha} BETA {fusion.linear_beta} WINDOW {fusion.window}' + # ... +``` + +**Benefits:** +- All data is accessible via the spec. +- No searching through conditions. +- Handles both RRF and LINEAR in one method. + +--- + +## Summary + +| Aspect | Design C (Detection) | Design C+ (Primitive) | +|--------|----------------------|----------------------| +| **Data model** | Scattered across `vector_search`, `conditions`, optional fields | Single `HybridFusionSpec` object | +| **Parser complexity** | Must distinguish "fulltext as filter" vs "fulltext as ranking" | Explicit: `WHERE fulltext()` = filter, `SELECT fulltext()` = ranking leg | +| **Translator dispatch** | Heuristics and inference | Type-driven (if `hybrid_fusion`, call `_build_ft_hybrid`) | +| **Command builder** | Separate methods for RRF/LINEAR, duplication | Single method, `if fusion_method` switch | +| **Extensibility** | Add more optional fields, more heuristics | Add fields to `HybridFusionSpec`, no heuristics | +| **Backward compat** | Risk of breaking filter-then-KNN if detection misfires | Safe: new field, existing queries untouched | + +**Recommendation:** Use Design C+ (primitive-based) to avoid technical debt and make the codebase easier to maintain as `FT.HYBRID` evolves. diff --git a/docs/proposals/ft-hybrid-implementation-sketch.md b/docs/proposals/ft-hybrid-implementation-sketch.md new file mode 100644 index 0000000..552073a --- /dev/null +++ b/docs/proposals/ft-hybrid-implementation-sketch.md @@ -0,0 +1,402 @@ +# FT.HYBRID Implementation Sketch (Design C+) + +**Purpose:** Concrete code examples showing how to implement the `HybridFusionSpec` primitive in sql-redis. + +## 1. Parser Changes (parser.py) + +### New Data Classes + +```python +@dataclass +class TextRankingLeg: + """Text ranking leg for hybrid fusion.""" + field: str # TEXT field name + query: str | None # Search query string + alias: str # Score alias (e.g., "tscore") + scorer: str = "BM25" # BM25, TFIDF, DISMAX + +@dataclass +class VectorRankingLeg: + """Vector ranking leg for hybrid fusion.""" + field: str # VECTOR field name + alias: str # Score alias (e.g., "vscore") + k: int | None = None # KNN K (defaults to window if None) + +@dataclass +class HybridFusionSpec: + """Specification for FT.HYBRID server-side fusion. + + Represents two independently-ranked legs (text + vector) fused by + RRF or LINEAR combination. Mutually exclusive with VectorSearchSpec + (filter-then-KNN). + """ + text_leg: TextRankingLeg + vector_leg: VectorRankingLeg + fusion_method: str # "RRF" or "LINEAR" + + # RRF parameters + rrf_constant: int = 60 + + # LINEAR parameters + linear_alpha: float = 0.5 + linear_beta: float = 0.5 + + # Common parameters + window: int = 20 # Fusion window size +``` + +### Update ParsedQuery + +```python +@dataclass +class ParsedQuery: + # ... existing fields ... + vector_search: VectorSearchSpec | None = None # Filter-then-KNN (existing) + hybrid_fusion: HybridFusionSpec | None = None # Server-side fusion (NEW) + # Note: vector_search and hybrid_fusion are mutually exclusive +``` + +### Detection Logic in _process_order_by + +```python +def _process_order_by(self, order: exp.Order, result: ParsedQuery) -> None: + """Process ORDER BY clause, detecting fusion functions.""" + for ordered in order.expressions: + expression = ordered.this + + # Check if it's a fusion function: rrf() or linear() + if isinstance(expression, exp.Anonymous): + func_name = expression.name.upper() + + if func_name in ("RRF", "LINEAR"): + self._build_hybrid_fusion_spec(expression, func_name, result) + continue + + # Regular ORDER BY field + field_name = expression.name if isinstance(expression, exp.Column) else None + direction = "DESC" if ordered.args.get("desc") else "ASC" + if field_name: + result.orderby_fields.append((field_name, direction)) + +def _build_hybrid_fusion_spec( + self, expression: exp.Anonymous, fusion_method: str, result: ParsedQuery +) -> None: + """Build HybridFusionSpec from rrf() or linear() in ORDER BY. + + Expected signatures: + - rrf(vscore, tscore [, constant => 60] [, window => 20]) + - linear(vscore, tscore [, alpha => 0.5] [, beta => 0.5] [, window => 20]) + """ + args = expression.expressions + if len(args) < 2: + raise ValueError( + f"{fusion_method.lower()}() requires at least 2 arguments: " + f"{fusion_method.lower()}(vector_alias, text_alias), got {len(args)}" + ) + + # Extract aliases (first two args) + vector_alias = args[0].name if isinstance(args[0], exp.Column) else None + text_alias = args[1].name if isinstance(args[1], exp.Column) else None + + if not vector_alias or not text_alias: + raise ValueError( + f"{fusion_method.lower()}() requires column aliases: " + f"{fusion_method.lower()}(vscore, tscore)" + ) + + # Find the corresponding vector_distance and fulltext in SELECT + vector_leg = self._find_vector_leg(result, vector_alias) + text_leg = self._find_text_leg(result, text_alias) + + if not vector_leg: + raise ValueError( + f"No vector_distance(...) AS {vector_alias} found in SELECT" + ) + if not text_leg: + raise ValueError( + f"No fulltext(...) AS {text_alias} found in SELECT" + ) + + # Extract fusion parameters from kwargs + params = self._extract_fusion_params(args[2:], fusion_method) + + # Build HybridFusionSpec + result.hybrid_fusion = HybridFusionSpec( + text_leg=text_leg, + vector_leg=vector_leg, + fusion_method=fusion_method, + **params + ) + + # Clear vector_search since hybrid_fusion takes precedence + result.vector_search = None + +def _find_vector_leg(self, result: ParsedQuery, alias: str) -> VectorRankingLeg | None: + """Find vector_distance(...) AS alias in SELECT.""" + if result.vector_search and result.vector_search.alias == alias: + return VectorRankingLeg( + field=result.vector_search.field, + alias=alias, + k=result.vector_search.k + ) + return None + +def _find_text_leg(self, result: ParsedQuery, alias: str) -> TextRankingLeg | None: + """Find fulltext(...) AS alias in SELECT. + + This requires detecting fulltext() in SELECT (not WHERE). + """ + # Look for a computed field or condition with FULLTEXT operator + # that has the matching alias + for cond in result.conditions: + if cond.operator == "FULLTEXT" and hasattr(cond, "alias") and cond.alias == alias: + return TextRankingLeg( + field=cond.field, + query=cond.value, + alias=alias, + scorer="BM25" # Default; could be configurable + ) + return None + +def _extract_fusion_params(self, kwarg_exprs: list, fusion_method: str) -> dict: + """Extract kwargs like constant => 60, window => 20.""" + params = {} + + for expr in kwarg_exprs: + if isinstance(expr, exp.EQ): + key = expr.this.name if isinstance(expr.this, exp.Column) else None + value = self._extract_literal_value(expr.expression) + + if key and value is not None: + params[key] = value + + return params +``` + +## 2. Analyzer Changes (analyzer.py) + +### New Data Class + +```python +@dataclass +class HybridFusionAnalysis: + """Analyzed hybrid fusion with validated field types.""" + spec: HybridFusionSpec + text_field_type: str # Confirmed TEXT from schema + vector_field_type: str # Confirmed VECTOR from schema + filters: list[Condition] # Per-leg filters (applied to both SEARCH and VSIM) +``` + +### Update AnalyzedQuery + +```python +@dataclass +class AnalyzedQuery: + # ... existing fields ... + hybrid_fusion: HybridFusionAnalysis | None = None +``` + +### Validation Logic + +```python +def analyze(self, parsed: ParsedQuery) -> AnalyzedQuery: + # ... existing logic ... + + if parsed.hybrid_fusion: + hybrid_fusion = self._analyze_hybrid_fusion(parsed) + else: + hybrid_fusion = None + + return AnalyzedQuery( + # ... existing fields ... + hybrid_fusion=hybrid_fusion + ) + +def _analyze_hybrid_fusion(self, parsed: ParsedQuery) -> HybridFusionAnalysis: + """Validate hybrid fusion spec against schema.""" + spec = parsed.hybrid_fusion + schema = self._schemas[parsed.index] + + # Validate text field is TEXT + text_field_type = schema.get(spec.text_leg.field) + if not text_field_type: + raise ValueError( + f"Text field '{spec.text_leg.field}' not found in index '{parsed.index}'" + ) + if text_field_type != "TEXT": + raise ValueError( + f"Text field '{spec.text_leg.field}' must be TEXT, got {text_field_type}" + ) + + # Validate vector field is VECTOR + vector_field_type = schema.get(spec.vector_leg.field) + if not vector_field_type: + raise ValueError( + f"Vector field '{spec.vector_leg.field}' not found in index '{parsed.index}'" + ) + if vector_field_type != "VECTOR": + raise ValueError( + f"Vector field '{spec.vector_leg.field}' must be VECTOR, got {vector_field_type}" + ) + + # Extract filters from conditions (exclude hybrid-related conditions) + filters = [ + c for c in parsed.conditions + if c.field != spec.text_leg.field or c.operator != "FULLTEXT" + ] + + return HybridFusionAnalysis( + spec=spec, + text_field_type=text_field_type, + vector_field_type=vector_field_type, + filters=filters + ) +``` + +## 3. Translator Changes (translator.py) + +### Command Dispatch + +```python +def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + """Build Redis command from analyzed query. + + Dispatch order: + 1. FT.HYBRID (hybrid fusion) + 2. FT.AGGREGATE (aggregations/groupby) + 3. FT.SEARCH (default) + """ + if analyzed.hybrid_fusion: + return self._build_ft_hybrid(analyzed) + elif analyzed.use_aggregate: + return self._build_ft_aggregate(analyzed) + else: + return self._build_ft_search(analyzed) +``` + +### FT.HYBRID Builder + +```python +def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + """Build FT.HYBRID command.""" + fusion = analyzed.hybrid_fusion.spec + parsed = analyzed.parsed + + # Build filter expression + filter_expr = self._build_filter_expr(analyzed.hybrid_fusion.filters) + + # Build SEARCH leg + text_query = f"@{fusion.text_leg.field}:({fusion.text_leg.query})" + if filter_expr: + text_query = f"({text_query}) {filter_expr}" + search_leg = f'SEARCH "{text_query}" YIELD_SCORE_AS {fusion.text_leg.alias}' + + # Build VSIM leg + k = fusion.vector_leg.k or fusion.window + filter_count = len(analyzed.hybrid_fusion.filters) + vsim_parts = [ + f'VSIM @{fusion.vector_leg.field} $vec', + ] + if filter_count > 0: + vsim_parts.append(f'FILTER {filter_count} "{filter_expr}"') + vsim_parts.append(f'KNN 2 K {k}') + vsim_parts.append(f'YIELD_SCORE_AS {fusion.vector_leg.alias}') + vsim_leg = ' '.join(vsim_parts) + + # Build COMBINE clause + if fusion.fusion_method == "RRF": + combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' + else: # LINEAR + combine = ( + f'COMBINE LINEAR 2 ' + f'ALPHA {fusion.linear_alpha} ' + f'BETA {fusion.linear_beta} ' + f'WINDOW {fusion.window}' + ) + + # Build LOAD clause + load_fields = parsed.fields if parsed.fields else ["*"] + load_clause = ["LOAD", str(len(load_fields))] + load_fields + + # Build LIMIT clause + offset = parsed.offset or 0 + limit = parsed.limit or 10 + limit_clause = ["LIMIT", str(offset), str(limit)] + + # Assemble command + cmd = [ + "FT.HYBRID", + parsed.index, + "2", # Number of legs + search_leg, + vsim_leg, + combine, + *load_clause, + *limit_clause, + "DIALECT", "2" + ] + + return TranslatedQuery( + command="FT.HYBRID", + args=cmd[1:], + index=parsed.index, + return_fields=parsed.fields + ) +``` + +## 4. Executor Changes (executor.py) + +### Version Gating + +```python +def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: + """Execute SQL query against Redis.""" + params = params or {} + + # Substitute and translate + sql = _substitute_params(sql, params) + translated = self._translator.translate(sql) + + # Version gate for FT.HYBRID + if translated.command == "FT.HYBRID": + self._check_hybrid_support() + + # Execute command + # ... existing logic ... + +def _check_hybrid_support(self) -> None: + """Check if Redis supports FT.HYBRID (8.4+).""" + info = self._client.info("server") + version = info.get("redis_version", "0.0.0") + + if not self._meets_version(version, "8.4.0"): + raise ValueError( + f"FT.HYBRID requires Redis 8.4 or later, found {version}. " + "Use filter-then-KNN syntax (vector_distance without fulltext in SELECT) instead." + ) + +def _meets_version(self, current: str, required: str) -> bool: + """Check if current version >= required version.""" + current_parts = [int(x) for x in current.split(".")[:3]] + required_parts = [int(x) for x in required.split(".")] + + for c, r in zip(current_parts, required_parts): + if c > r: + return True + if c < r: + return False + return True +``` + +## Summary + +This implementation sketch shows: + +1. **Parser:** Detect fusion functions in `ORDER BY`, build `HybridFusionSpec`. +2. **Analyzer:** Validate field types, separate filters. +3. **Translator:** Dispatch to `_build_ft_hybrid()` when `hybrid_fusion` is present. +4. **Executor:** Gate on Redis 8.4+ before executing `FT.HYBRID`. + +All changes are **additive** and **backward compatible** with existing filter-then-KNN queries. + + diff --git a/docs/proposals/ft-hybrid-primitive-design.md b/docs/proposals/ft-hybrid-primitive-design.md new file mode 100644 index 0000000..13cadc0 --- /dev/null +++ b/docs/proposals/ft-hybrid-primitive-design.md @@ -0,0 +1,295 @@ +# FT.HYBRID Primitive Design - Recommendation + +**Status:** Design recommendation for RAAE-1322 +**Supersedes:** Design discussion in `ft-hybrid.md` +**Date:** 2026-06-23 + +## Executive Summary + +After reviewing the `ft-hybrid.md` proposal and the sql-redis codebase, I recommend **Design C with composable ranking functions**, but with a significant improvement: introduce a **new primitive abstraction** that better encapsulates hybrid fusion and aligns with sql-redis's design philosophy. + +## Core Issue with Current Proposal + +The three designs (A, B, C) in `ft-hybrid.md` all treat hybrid fusion as a **syntax mapping problem** rather than a **data structure problem**. The proposals focus on where to put `rrf()` or `hybrid()` in SQL, but don't address the deeper design question: + +**How should hybrid fusion be represented in sql-redis's internal data model?** + +Currently, sql-redis has: +- `VectorSearchSpec` - represents a single vector leg +- `Condition` - represents text/filter predicates +- `ScoringSpec` - represents relevance scoring (BM25, etc.) + +None of these primitives can cleanly express **two independent ranking signals fused together**. Adding `FT.HYBRID` support by bolting detection logic onto existing specs will create technical debt. + +## Recommended Primitive: `HybridFusionSpec` + +### Data Structure + +```python +@dataclass +class TextRankingLeg: + """Text ranking leg for hybrid fusion.""" + field: str # TEXT field to search + query: str | None # Search query text + alias: str # Score alias (e.g., "tscore") + scorer: str = "BM25" # BM25, TFIDF, DISMAX + +@dataclass +class VectorRankingLeg: + """Vector ranking leg for hybrid fusion.""" + field: str # VECTOR field to search + alias: str # Score alias (e.g., "vscore") + k: int | None = None # KNN candidate pool size + +@dataclass +class HybridFusionSpec: + """Specification for FT.HYBRID fusion.""" + text_leg: TextRankingLeg + vector_leg: VectorRankingLeg + fusion_method: str # "RRF" or "LINEAR" + + # RRF parameters + rrf_constant: int = 60 + + # LINEAR parameters + linear_alpha: float = 0.5 + linear_beta: float = 0.5 + + # Common parameters + window: int = 20 # Fusion window size +``` + +This primitive makes hybrid fusion **first-class** instead of inferring it from the presence of multiple specs. + +### Integration into ParsedQuery + +```python +@dataclass +class ParsedQuery: + # ... existing fields ... + vector_search: VectorSearchSpec | None = None # For filter-then-KNN + hybrid_fusion: HybridFusionSpec | None = None # For FT.HYBRID fusion +``` + +**Mutual exclusion:** A query has EITHER `vector_search` (filter-then-KNN) OR `hybrid_fusion` (server-side fusion), never both. + +## Why This Is Better + +### 1. **Explicit over implicit** +Design C detects `fulltext(...) AS tscore` + `vector_distance(...) AS vscore` + `rrf(tscore, vscore)` and infers hybrid mode. The new primitive makes the intent **explicit** in the data model. + +### 2. **Single source of truth** +All hybrid parameters live in `HybridFusionSpec`. No need to reconcile state scattered across `VectorSearchSpec`, `Condition`, and `ORDER BY`. + +### 3. **Clear validation** +The analyzer can enforce: +- `text_leg.field` is TYPE `TEXT` +- `vector_leg.field` is TYPE `VECTOR` +- Fusion method matches parameters (RRF → constant, LINEAR → alpha/beta) +- Filters are compatible with both legs + +### 4. **Command-path clarity** +```python +if analyzed.hybrid_fusion: + return build_ft_hybrid_command(analyzed) +elif analyzed.vector_search: + return build_ft_search_knn_command(analyzed) +else: + return build_ft_search_or_aggregate_command(analyzed) +``` + +No branching on combinations of flags — the data structure dictates the path. + +### 5. **Extensibility** +When Redis adds more fusion methods (e.g., `DBSF`, `CombSUM`), you add a new `fusion_method` value. When `FT.HYBRID` gains a third leg (e.g., sparse vector), you add `SparseVectorRankingLeg`. + +## Recommended SQL Syntax (Design C+) + +Keep Design C's composability, but tighten the semantics around the new primitive: + +```sql +SELECT page_text, file_id, + vector_distance(embedding, :vec) AS vscore, + fulltext(page_text, 'quarterly earnings') AS tscore +FROM "KM_abc123" +WHERE ticker = 'MSFT' +ORDER BY rrf(vscore, tscore, constant => 60) DESC +LIMIT 10; +``` + +### Detection Logic (Parser) + +The parser builds `HybridFusionSpec` when **all** of these are true: + +1. `SELECT` contains `vector_distance(vec_field, :param) AS ` +2. `SELECT` contains `fulltext(text_field, 'query') AS ` +3. `ORDER BY` contains `rrf(, , ...)` or `linear(, , ...)` + +If only (1) is present → `VectorSearchSpec` (filter-then-KNN, existing behavior). +If (1) + (2) but no fusion in `ORDER BY` → `ValueError("ambiguous: use rrf() or linear() to specify fusion")`. + +This forces users to be explicit about fusion vs. returning two separate scores. + +### Why Not Design B (`hybrid()` predicate)? + +Design B is more compact: + +```sql +WHERE hybrid(page_text, 'quarterly earnings', embedding, :vec) +``` + +But it has problems: + +1. **Breaks the separation of concerns.** `WHERE` is for filtering; ranking belongs in `SELECT`/`ORDER BY`. +2. **No place for score aliases.** Users can't reference `tscore`/`vscore` separately for debugging. +3. **Harder to extend.** Adding scorer config (BM25 vs TFIDF), KNN params, or window size requires kwargs in `WHERE`, which is unnatural. + +Design C keeps ranking in `SELECT`/`ORDER BY` where it belongs. + +## Analyzer Changes + +```python +@dataclass +class AnalyzedQuery: + # ... existing fields ... + hybrid_fusion: HybridFusionAnalysis | None = None + +@dataclass +class HybridFusionAnalysis: + """Analyzed hybrid fusion with resolved field types.""" + spec: HybridFusionSpec + text_field_type: str # Confirmed TEXT from schema + vector_field_type: str # Confirmed VECTOR from schema + filters: list[Condition] # Per-leg filters (applied to both SEARCH and VSIM) +``` + +The analyzer: +1. Validates `text_leg.field` exists and is `TEXT` +2. Validates `vector_leg.field` exists and is `VECTOR` +3. Splits `WHERE` conditions into per-leg filters (no special text-leg-only logic — apply to both for consistency) +4. Computes KNN `K` from `window` if not explicitly set + +## Translator Changes + +Add a new command-path branch **before** the search-vs-aggregate decision: + +```python +def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + if analyzed.hybrid_fusion: + return self._build_ft_hybrid(analyzed) + elif analyzed.use_aggregate: + return self._build_ft_aggregate(analyzed) + else: + return self._build_ft_search(analyzed) + +def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + """Build FT.HYBRID command from HybridFusionAnalysis.""" + fusion = analyzed.hybrid_fusion.spec + + # Build SEARCH leg + search_query = build_text_query(fusion.text_leg.field, fusion.text_leg.query) + search_filter = build_filter_expr(analyzed.hybrid_fusion.filters) + search_leg = f'SEARCH "{search_query} {search_filter}" YIELD_SCORE_AS {fusion.text_leg.alias}' + + # Build VSIM leg + vsim_filter_count = len(analyzed.hybrid_fusion.filters) + k = fusion.vector_leg.k or fusion.window + vsim_leg = ( + f'VSIM @{fusion.vector_leg.field} $vec ' + f'FILTER {vsim_filter_count} "{search_filter}" ' + f'KNN 2 K {k} YIELD_SCORE_AS {fusion.vector_leg.alias}' + ) + + # Build COMBINE clause + if fusion.fusion_method == "RRF": + combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' + else: # LINEAR + combine = ( + f'COMBINE LINEAR 2 ALPHA {fusion.linear_alpha} ' + f'BETA {fusion.linear_beta} WINDOW {fusion.window}' + ) + + # Assemble command + cmd = [ + "FT.HYBRID", analyzed.parsed.index, "2", + search_leg, + vsim_leg, + combine, + "LOAD", str(len(analyzed.parsed.fields)), *analyzed.parsed.fields, + "LIMIT", str(analyzed.parsed.offset or 0), str(analyzed.parsed.limit or 10), + "DIALECT", "2" + ] + + return TranslatedQuery(command="FT.HYBRID", args=cmd[1:], ...) +``` + +This is **much cleaner** than trying to reuse `_build_ft_search` and injecting hybrid logic via conditionals. + +## Version Gating + +Add a Redis version check in the executor: + +```python +def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: + translated = self._translator.translate(sql) + + if translated.command == "FT.HYBRID": + # Check Redis version >= 8.4 + info = self._client.info("server") + version = info.get("redis_version", "0.0.0") + if not _meets_version(version, "8.4.0"): + raise ValueError( + f"FT.HYBRID requires Redis 8.4+, found {version}. " + "Use filter-then-KNN syntax instead." + ) + + # ... execute command ... +``` + +Fail **fast** with a clear message rather than letting Redis return a cryptic error. + +## Migration Path + +This design is **backward compatible**: + +- Existing `vector_distance()` queries without `fulltext()` → still build `VectorSearchSpec` → still emit `FT.SEARCH` with KNN. +- Existing `fulltext()` queries in `WHERE` → still build `Condition` → still work as filters. + +No existing queries break. Hybrid fusion is purely **additive**. + +## Comparison to Original Designs + +| Aspect | Design A | Design B | Design C (original) | **Design C+ (new primitive)** | +|--------|----------|----------|---------------------|-------------------------------| +| Syntax clarity | ❌ Ambiguous | ✅ Compact | ✅ Composable | ✅ Composable + explicit | +| Data model | ❌ Inferred | ⚠️ New WHERE func | ⚠️ Scattered state | ✅ Dedicated primitive | +| Extensibility | ❌ Breaking | ⚠️ Kwargs hell | ⚠️ Inference fragile | ✅ Clean extension points | +| Command path | ❌ Heuristics | ⚠️ Detection | ⚠️ Detection | ✅ Type-driven dispatch | +| Score surfacing | ❌ Unclear | ❌ No aliases | ✅ Explicit aliases | ✅ Explicit aliases | +| Backward compat | ❌ Breaking | ✅ New func | ✅ Additive | ✅ Additive | + +**Verdict:** Design C+ (composable syntax + dedicated primitive) is the cleanest path forward. + +## Open Questions for Confirmation + +1. **Fusion defaults:** Should `ORDER BY DESC` on a score without `rrf()/linear()` default to RRF, or require explicit fusion? **Recommendation:** Require explicit fusion (fail fast on ambiguity). + +2. **Score surfacing:** Should the fused score be auto-projected as a column (e.g., `__fused_score`)? **Recommendation:** Let users explicitly `SELECT` it via an alias if needed. + +3. **K vs WINDOW:** Should KNN `K` default to `window`, or be independently configurable? **Recommendation:** Default `K = window` for v1; add `k =>` kwarg to `rrf()/linear()` later if needed. + +4. **Filter distribution:** Apply `WHERE` filters to both legs, or SEARCH-only? **Recommendation:** Both legs (consistency > simplicity). + +## Next Steps + +1. Implement `HybridFusionSpec`, `TextRankingLeg`, `VectorRankingLeg` dataclasses in `parser.py`. +2. Add detection logic in `SQLParser._process_order_by()` to build `HybridFusionSpec` when fusion functions are present. +3. Add `HybridFusionAnalysis` to analyzer and validation logic. +4. Implement `Translator._build_ft_hybrid()` command builder. +5. Add version gating in executor. +6. Update docs to relabel "hybrid search" → "filtered KNN" and add "Hybrid Fusion (FT.HYBRID)" guide. +7. Bump test Redis image to 8.4+ and add integration tests. + +**Estimated effort:** 3-4 days for implementation + tests + docs (assuming no blockers on Redis 8.4 availability). + + diff --git a/docs/proposals/ft-hybrid.md b/docs/proposals/ft-hybrid.md new file mode 100644 index 0000000..2de81fc --- /dev/null +++ b/docs/proposals/ft-hybrid.md @@ -0,0 +1,295 @@ +# Spec: `FT.HYBRID` support in sql-redis + +**Jira:** [RAAE-1322](https://redislabs.atlassian.net/browse/RAAE-1322) · +**Status:** Draft · +**Requires:** Redis 8.4+ and redis-py >= 7.1.0 (RediSearch `FT.HYBRID`) + +# Goal + +Add a `hybrid_vector_search(...)` SELECT function that translates a SQL `SELECT` into a native +`FT.HYBRID` command, so a text query and a vector query are **fused server-side** (RRF or +LINEAR) into a single ranking. This is distinct from today's pre-filter hybrid search, +where text is only a hard prefilter and the ranking comes from the vector leg alone. sql-redis exists +to give a familiar SQL surface over a Redis query; `hybrid_vector_search()` makes the new +server-side fusion query just as easy to express. + +Hard constraint: the syntax must read as a natural extension of the vector syntax we have +today (`cosine_distance(field, :vec)` plus `:param` substitution), and its options must +mirror what `FT.HYBRID` exposes so the two RedisVL surfaces (native `HybridQuery` and +`SQLQuery`) stay coherent. + +> Terminology note. The README/docs currently label filter-then-KNN as +> "Hybrid search (filters + vector)". That is not `FT.HYBRID`. This work uses: +> +> - **pre-filter hybrid search**: existing `WHERE ... cosine_distance(...)` to +> `(prefilter)=>[KNN ...]`. One ranking signal (vector); text/tags are hard filters. +> - **hybrid fusion** (this spec): new `hybrid_vector_search(...)` to `FT.HYBRID`. Two +> independently ranked legs (text + vector) fused by RRF/LINEAR. +> +> Part of this work is relabeling the existing docs to "pre-filter hybrid search" so the +> two are not conflated. + +# Flow + +```mermaid +flowchart LR + SQL["SELECT ..., hybrid_vector_search(vec_leg, text_leg, combine) AS score\nFROM idx WHERE region='us-central' LIMIT 10"] --> P[SQLParser] + P -->|"HybridSearchSpec\n(SEARCH leg + VSIM leg + COMBINE)"| A[Analyzer] + A -->|"resolve TEXT + VECTOR field types\nWHERE to per-leg FILTER"| QB[QueryBuilder] + QB --> T[Translator] + T -->|"emit FT.HYBRID\n(new third command path)"| EX[Executor] + EX -->|"two-stage param sub\n(vector bytes injected via PARAMS)"| R[(Redis 8.4+)] + R -->|fused rows + scores| EX --> Res[QueryResult] +``` + +# Syntax + +`hybrid_vector_search(...)` is a SELECT-clause function that composes the two ranking functions +the package already exposes: the vector leg reuses `cosine_distance(field, :vec)` and the +text leg reuses `fulltext(field, 'query')`. A third argument selects the fusion method. + +```sql +SELECT user, job, job_description, + hybrid_vector_search( + cosine_distance(job_embedding, :vec), -- VSIM leg: vector field + param + fulltext(job_description, 'use base principles to solve problems'), -- SEARCH leg: text field + query + rrf() -- COMBINE: fusion method (default RRF) + ) AS hybrid_score +FROM user_simple +WHERE region = 'us-central' -- FILTER, applied to both legs +ORDER BY hybrid_score DESC +LIMIT 10; +``` + +Why this aligns: + +- `cosine_distance(job_embedding, :vec)` is the exact vector function used today; here it + is the `VSIM` leg instead of a standalone KNN. +- `fulltext(job_description, 'query')` is the exact text function used today; as an + argument to `hybrid_vector_search` it becomes the `SEARCH` leg. +- `WHERE` maps to the per-leg `FILTER` (applied to both legs so the candidate sets agree). +- `SELECT` columns map to `LOAD`; `GROUP BY` maps to `GROUPBY`; `LIMIT` maps to `LIMIT`. +- `AS hybrid_score` surfaces the combined `YIELD_SCORE_AS` as a column. + +## Full knobs + +Defaults mirror RedisVL's native `HybridQuery` so the two surfaces match. + +**Fusion method (third argument)** + +| SQL | FT.HYBRID | Notes | +|---|---|---| +| `rrf()` | `COMBINE RRF` (server default) | omit the third arg for the same effect | +| `rrf(constant => 60, window => 20)` | `COMBINE RRF 4 CONSTANT 60 WINDOW 20` | defaults: constant 60, window 20 | +| `linear(alpha => 0.3)` | `COMBINE LINEAR ... ALPHA 0.3 BETA 0.7` | v1 exposes `alpha` only; `beta = 1 - alpha` (matches native `HybridQuery`) | +| `linear(alpha => 0.3, window => 20)` | `COMBINE LINEAR ... ALPHA 0.3 ... WINDOW 20` | | + +**Vector leg (`cosine_distance` / `vector_distance`)** + +| SQL | FT.HYBRID | +|---|---| +| `cosine_distance(job_embedding, :vec)` | `VSIM @job_embedding $vec KNN 2 K ` (K defaults to `LIMIT`, else 10) | +| `vector_distance(job_embedding, :vec, ef_runtime => 10)` | `... KNN ... EF_RUNTIME 10` (knob form; see kwarg note) | +| `vector_range(job_embedding, :vec, radius => 0.2, epsilon => 0.01)` | `VSIM @job_embedding $vec RANGE ... RADIUS 0.2 EPSILON 0.01` | + +**Text leg (`fulltext`)** + +| SQL | FT.HYBRID | +|---|---| +| `fulltext(job_description, 'principles')` | `SEARCH "@job_description:(principles)"` | +| `fulltext(job_description, 'principles', scorer => 'BM25STD')` | `SEARCH "..." SCORER BM25STD` (default `BM25STD`) | + +**Surfacing per-leg scores (optional)** + +`AS hybrid_score` yields the combined score. To also return the individual leg scores, add +kwargs to the legs: `cosine_distance(..., yield_score_as => 'vsim')` and +`fulltext(..., yield_score_as => 'tscore')`, mapping to each leg's `YIELD_SCORE_AS`. + +## SQL to `FT.HYBRID` mapping (worked example) + +Verified against Redis 8.4 (`redis:8.4`). Note three command-shape rules +confirmed empirically: the VSIM method clause (`KNN`/`RANGE`) must come **before** +`FILTER`; `LOAD` fields require an `@` prefix; and `FT.HYBRID` **rejects** an +explicit `DIALECT` argument (it uses the server's configured default). + +``` +FT.HYBRID user_simple + SEARCH "(@job_description:(use base principles to solve problems)) (@region:{us\-central})" SCORER BM25STD + VSIM @job_embedding $vector KNN 2 K 10 FILTER 1 "@region:{us\-central}" + COMBINE RRF 6 CONSTANT 60 WINDOW 20 YIELD_SCORE_AS hybrid_score + LOAD 3 @user @job @job_description + LIMIT 0 10 + PARAMS 2 vector +``` + +The reply is a flat map (not the FT.AGGREGATE array shape): +`[total_results, N, results, [[field, val, ...], ...], warnings, [...], execution_time, ...]`. + +| SQL element | FT.HYBRID target | +|---|---| +| `hybrid_vector_search(...)` in SELECT | triggers the `FT.HYBRID` command path | +| `cosine_distance(vec_field, :vec)` (nested) | `VSIM @vec_field $vector KNN ... K ...` | +| `fulltext(text_field, 'q')` (nested) | `SEARCH "@text_field:(q)" [SCORER ...]` | +| `WHERE ` | folded into the `SEARCH` query string **and** `VSIM ... FILTER n ""` (after the method clause) | +| `rrf(...)` / `linear(...)` | `COMBINE RRF \| LINEAR ...` | +| `SELECT col1, col2, ...` | `LOAD n @col1 @col2 ...` | +| `GROUP BY ...` + aggregations | `GROUPBY n @prop REDUCE ...` | +| `ORDER BY hybrid_score DESC` | `SORTBY` / default combined-score sort | +| `LIMIT n` / `LIMIT m, n` | `LIMIT m n` (also sets KNN `K` when the vector leg is KNN) | +| `:vec` param (bytes) | `PARAMS 2 vector ` via stage-2 substitution | +| (note) | no `DIALECT` argument (FT.HYBRID rejects it) | + +## End-user surface (RedisVL `SQLQuery`) + +Most users reach sql-redis through RedisVL's `SQLQuery`. The query author writes the same +SQL and passes the vector blob as a param; `index.query(...)` runs it through the +sql-redis executor and returns rows. No RedisVL execution code changes once sql-redis emits +and parses `FT.HYBRID` (the dispatch path `index.query` to `_sql_query` to +`executor.execute` is unchanged). Companion spec: +`applied-ai/redis-vl-python/docs/proposals/sqlquery-ft-hybrid.md`. + +Default RRF fusion: + +```python +from redisvl.query import SQLQuery +from redisvl.index import SearchIndex + +index = SearchIndex.from_dict(schema, redis_url="redis://localhost:6379") +vec = hf.embed("use base principles to solve problems", as_buffer=True) + +sql_query = SQLQuery( + """ + SELECT user, job, job_description, + hybrid_vector_search( + cosine_distance(job_embedding, :vec), + fulltext(job_description, 'use base principles to solve problems'), + rrf() + ) AS hybrid_score + FROM user_simple + WHERE region = 'us-central' + ORDER BY hybrid_score DESC + LIMIT 10 + """, + params={"vec": vec}, +) + +# Inspect the generated command before running it +print(sql_query.redis_query_string(redis_url="redis://localhost:6379")) +# FT.HYBRID user_simple SEARCH "@job_description:(...) (@region:{us\-central})" SCORER BM25STD +# VSIM @job_embedding $vec FILTER 1 "@region:{us\-central}" KNN 2 K 10 +# COMBINE RRF 4 CONSTANT 60 WINDOW 20 YIELD_SCORE_AS hybrid_score +# LOAD 3 user job job_description LIMIT 0 10 PARAMS 2 vec DIALECT 2 + +results = index.query(sql_query) +``` + +LINEAR fusion with a custom text scorer and an explicit KNN tuning knob: + +```python +sql_query = SQLQuery( + """ + SELECT user, job, job_description, + hybrid_vector_search( + cosine_distance(job_embedding, :vec, ef_runtime => 20), + fulltext(job_description, 'principles', scorer => 'BM25STD'), + linear(alpha => 0.3) + ) AS hybrid_score + FROM user_simple + ORDER BY hybrid_score DESC + LIMIT 5 + """, + params={"vec": vec}, +) +results = index.query(sql_query) +``` + +Async usage is identical via `AsyncSearchIndex.query(sql_query)`. + +# Implementation plan (by layer) + +Mirrors the existing `cosine_distance` / `vector_distance` path. There is no function +registry, so dispatch is added to the same sites that already handle vector and text +functions ([`sql_redis/parser.py`](../../sql_redis/parser.py) `_process_select_expression` +and `_add_function_condition`). + +1. **Parser** ([`sql_redis/parser.py`](../../sql_redis/parser.py)) + - Add a `HybridSearchSpec` dataclass: text field/query/scorer, vector field/param, + vector method (KNN/RANGE) and its params, fusion method and params, per-leg and + combined score aliases. + - Detect `hybrid_vector_search(...)` in the SELECT projection. Parse its three arguments by + reusing the existing `cosine_distance` / `vector_distance` and `fulltext` extraction + so the legs behave identically to their standalone forms. +2. **Analyzer** ([`sql_redis/analyzer.py`](../../sql_redis/analyzer.py)) + - Add `hybrid_search: HybridSearchAnalysis | None`. Resolve that the text field is + `TEXT` and the vector field is `VECTOR`; reject mismatches (mirror the existing + TEXT-operator guard). Split `WHERE` into the shared leg filter; resolve `K` from `LIMIT`. +3. **QueryBuilder** ([`sql_redis/query_builder.py`](../../sql_redis/query_builder.py)) + - Add `build_hybrid_command(...)` to emit the `SEARCH ... VSIM ... COMBINE ...` argv. + Reuse `build_text_condition` for the `SEARCH` query string and the existing filter + builders for the `FILTER` expression. +4. **Translator** ([`sql_redis/translator.py`](../../sql_redis/translator.py)) + - `FT.HYBRID` is a **new third command path** alongside `FT.SEARCH` / `FT.AGGREGATE`. + Branch on `analyzed.hybrid_search` before the search-vs-aggregate decision. Map + `LOAD`, `LIMIT`, `GROUPBY`, `PARAMS`, and `DIALECT 2`. +5. **Executor** ([`sql_redis/executor.py`](../../sql_redis/executor.py)) + - Reuse the two-stage param substitution (scalars inlined in stage 1; vector bytes kept + as `$vec` and injected via `PARAMS` in stage 2, exactly as vectors work today). + - Run the command and parse the hybrid reply (its shape differs from search/aggregate). + Prefer redis-py's `hybrid_search` result parsing (available in redis-py >= 7.1.0) + rather than hand-rolling reply parsing. + - Add a version guard: probe `FT.HYBRID` availability (or Redis >= 8.4) and raise a + clear `ValueError` ("FT.HYBRID requires Redis 8.4+") instead of a raw Redis error. +6. **Docs** ([`docs/user_guide/how_to_guides/vector-search.md`](../user_guide/how_to_guides/vector-search.md)) + - Relabel the existing "hybrid" section to "pre-filter hybrid search" and add a "Hybrid fusion + (FT.HYBRID)" section. Update the README capability list. + +# Testing + +Follow the layered convention (100% coverage enforced, no `# pragma: no cover`): + +- **Unit:** parser (spec extraction from the nested functions), analyzer (field-type + resolution, filter split), query_builder (exact argv), translator (full command plus + `DIALECT 2`). +- **Integration** ([`tests/test_sql_queries.py`](../../tests/test_sql_queries.py), + testcontainers): a `TestHybridFusion` class. The conftest Redis image is currently + **8.0.2**; `FT.HYBRID` needs **8.4+**, so the image must be bumped (and tests skipped on + unsupported versions). +- **Parity:** SQL result equals a hand-written `FT.HYBRID` result (mirrors + [`tests/test_redis_queries.py`](../../tests/test_redis_queries.py)). + +# Things to consider + +- **`K` vs `WINDOW` vs `LIMIT`.** `K` = vector neighbors feeding fusion; `WINDOW` = how + many per-leg results RRF/LINEAR consider; `LIMIT` = final rows. RedisVL's `HybridQuery` + collapses `num_results` into both KNN `K` and the final limit. Proposal: match it, + `LIMIT` sets KNN `K` and the final cut, `WINDOW` defaults to 20 unless set in the combine + function. Is exposing `WINDOW` and explicit `K` separately too much surface for v1? +- **LINEAR alpha/beta.** v1 exposes `alpha` only and derives `beta = 1 - alpha`, matching + native `HybridQuery`. `FT.HYBRID` accepts an explicit `beta`, but exposing it would let a + SQL query produce a command the object API cannot; defer it. (Decided.) +- **Filter on both legs vs SEARCH only.** Applying `WHERE` to both `SEARCH` and `VSIM` + keeps candidate sets consistent and matches RedisVL (`filter_expression` is passed to + both). Recommend both legs. +- **Default fusion.** Omitting the third argument yields the server default (RRF). Require + the explicit `rrf()`/`linear()` to set knobs; do not silently default `ORDER BY` into + fusion. +- **kwarg parsing (validated).** The `name => value` form parses through sqlglot as a + `Kwarg` inside anonymous functions (`fulltext`, `rrf`, `linear`, `vector_distance`), so + the fusion/scorer knobs parse cleanly. One constraint: `cosine_distance` is a sqlglot + built-in capped at 2 arguments, so a 3-arg `cosine_distance(field, :vec, ef_runtime => n)` + is a `ParseError`. Vector-leg tuning knobs (`ef_runtime`, RANGE `radius`/`epsilon`) + must therefore ride on the anonymous `vector_distance(...)` / `vector_range(...)` forms, + not `cosine_distance(...)`. Plain `cosine_distance(field, :vec)` (2 args) remains valid. +- **Score comparability.** RRF/LINEAR scores are not comparable to a raw + `cosine_distance`, so `ORDER BY hybrid_score` is the natural sort and per-leg scores are + opt-in via `yield_score_as`. +- **redis-py floor.** `FT.HYBRID` execution/parsing needs redis-py >= 7.1.0; gate and + document alongside the Redis 8.4 floor. + +# Decisions to confirm + +1. **Fusion config surface for v1.** Confirmed: RRF constant/window, LINEAR `alpha` only + (`beta = 1 - alpha`), KNN ef_runtime, RANGE radius/epsilon. +2. **Execution/parsing.** Use redis-py's `hybrid_search` result parsing inside the + executor (recommended), vs. hand-rolled reply parsing. See the companion RedisVL + packaging spec for how `SQLQuery` surfaces this end to end. diff --git a/sql_redis/analyzer.py b/sql_redis/analyzer.py index a6eeb3a..c02f70d 100644 --- a/sql_redis/analyzer.py +++ b/sql_redis/analyzer.py @@ -9,6 +9,7 @@ ComputedField, Condition, DateFunctionSpec, + HybridSearchSpec, ParsedQuery, ) @@ -22,6 +23,19 @@ class VectorSearchAnalysis: alias: str +@dataclass +class HybridSearchAnalysis: + """Analyzed FT.HYBRID fusion search details. + + Wraps the parsed spec and adds the resolved KNN ``k`` (from LIMIT). Field + types are validated during analysis (text leg must be TEXT, vector leg must + be VECTOR). + """ + + spec: HybridSearchSpec + k: int + + @dataclass class AnalyzedQuery: """Result of analyzing a parsed SQL query with schema context.""" @@ -34,6 +48,7 @@ class AnalyzedQuery: groupby_fields: list[str] = field(default_factory=list) is_global_aggregation: bool = False vector_search: VectorSearchAnalysis | None = None + hybrid_search: HybridSearchAnalysis | None = None has_prefilter: bool = False def get_field_type(self, field_name: str) -> str | None: @@ -121,6 +136,11 @@ def analyze(self, parsed: ParsedQuery) -> AnalyzedQuery: if parsed.vector_search: referenced_fields.add(parsed.vector_search.field) + # Fields from hybrid fusion search (both legs) + if parsed.hybrid_search: + referenced_fields.add(parsed.hybrid_search.vector_field) + referenced_fields.add(parsed.hybrid_search.text_field) + # Fields from date functions (YEAR, MONTH, etc.) for date_func in parsed.date_functions: referenced_fields.add(date_func.field) @@ -141,6 +161,10 @@ def analyze(self, parsed: ParsedQuery) -> AnalyzedQuery: # KNN similarity; the alias is a computed column, not an indexed # field, so it must not be looked up in the schema. alias_names.add(parsed.vector_search.alias) + if parsed.hybrid_search is not None and parsed.hybrid_search.alias: + # ORDER BY sorts by the fused score; like the + # vector alias, it is a computed column, not an indexed field. + alias_names.add(parsed.hybrid_search.alias) # Fields from GROUP BY (exclude aliases since they're computed) for field_name in parsed.groupby_fields: @@ -180,4 +204,27 @@ def analyze(self, parsed: ParsedQuery) -> AnalyzedQuery: # Has prefilter if there are conditions result.has_prefilter = len(parsed.conditions) > 0 + # Analyze hybrid fusion search + if parsed.hybrid_search: + spec = parsed.hybrid_search + vector_type = schema.get(spec.vector_field) + if vector_type != "VECTOR": + raise ValueError( + f"hybrid_vector_search() vector leg field " + f"'{spec.vector_field}' must be a VECTOR field, " + f"got {vector_type}." + ) + text_type = schema.get(spec.text_field) + if text_type != "TEXT": + raise ValueError( + f"hybrid_vector_search() text leg field " + f"'{spec.text_field}' must be a TEXT field, got {text_type}." + ) + result.hybrid_search = HybridSearchAnalysis( + spec=spec, + k=parsed.limit or spec.k or 10, + ) + # Conditions become per-leg filters. + result.has_prefilter = len(parsed.conditions) > 0 + return result diff --git a/sql_redis/executor.py b/sql_redis/executor.py index 6672d6c..1790ff7 100644 --- a/sql_redis/executor.py +++ b/sql_redis/executor.py @@ -118,6 +118,54 @@ class QueryResult: class _ScoreParseMixin: """Shared helpers for score-related response parsing.""" + @staticmethod + def _is_unknown_command(error_msg: str) -> bool: + """Return True when a ResponseError means the command is unsupported.""" + lowered = error_msg.lower() + return "unknown command" in lowered or "unknown subcommand" in lowered + + @staticmethod + def _parse_hybrid_reply(raw_result) -> tuple[Any, list[dict]]: + """Parse an FT.HYBRID reply into (count, rows). + + FT.HYBRID does not use the FT.AGGREGATE array shape. The reply is a map + ``{total_results: N, results: [...], warnings: [...], ...}`` that arrives + either as a dict (redis-py 8.x / RESP3) or as a flat list + (``[total_results, N, results, [...], ...]``) on RESP2. Each result row + is likewise a dict or a flat ``[field, val, ...]`` list. Keys/values may + be bytes or str depending on the client's decode_responses setting. + """ + if isinstance(raw_result, dict): + reply = raw_result + else: + reply = dict(zip(raw_result[::2], raw_result[1::2])) + + def _field(name: str): + if name in reply: + return reply[name] + return reply.get(name.encode()) + + count = _field("total_results") or 0 + results = _field("results") or [] + rows = [ + dict(row) if isinstance(row, dict) else dict(zip(row[::2], row[1::2])) + for row in results + ] + return count, rows + + @staticmethod + def _inject_vector_param(cmd: list[str | bytes], vector_param: bytes) -> None: + """Replace the vector PARAMS value with the actual bytes, in place. + + Only the ``$vector`` token in the PARAMS value position (the one + preceded by the param name ``vector``) is replaced. Query-side + references to ``$vector`` (FT.SEARCH KNN expressions, FT.HYBRID VSIM) + must stay as parameter references so Redis resolves them from PARAMS. + """ + for i, arg in enumerate(cmd): + if arg == "$vector" and i > 0 and cmd[i - 1] == "vector": + cmd[i] = vector_param + @staticmethod def _has_return_0(args: list[str]) -> bool: """Return True when the args contain 'RETURN 0' (no document fields).""" @@ -188,17 +236,24 @@ def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: vector_param = value break - # Replace $vector placeholder with actual bytes + # Replace the $vector PARAMS value with actual bytes (query/VSIM + # references to $vector stay as parameter references). if vector_param: - for i, arg in enumerate(cmd): - if arg == "$vector": - cmd[i] = vector_param + self._inject_vector_param(cmd, vector_param) # Execute command try: raw_result = self._client.execute_command(*cmd) except redis.ResponseError as e: error_msg = str(e) + if translated.command == "FT.HYBRID" and self._is_unknown_command( + error_msg + ): + raise redis.ResponseError( + f"{error_msg}. hybrid_vector_search() translates to FT.HYBRID, " + "which requires Redis 8.4+ (RediSearch with hybrid search) " + "and redis-py >= 7.1.0." + ) from e _ismissing_signatures = ( "Unknown function", "No such function", @@ -217,10 +272,14 @@ def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: raise # Parse result based on command type - count = raw_result[0] if raw_result else 0 - rows = [] - - if translated.command == "FT.SEARCH": + # FT.SEARCH/FT.AGGREGATE replies are arrays with the count first; + # FT.HYBRID replies are maps (dict) and set count during parsing below. + count = raw_result[0] if isinstance(raw_result, list) and raw_result else 0 + rows: list[dict] = [] + + if translated.command == "FT.HYBRID": + count, rows = self._parse_hybrid_reply(raw_result) + elif translated.command == "FT.SEARCH": # Use the explicit score_alias signal rather than scanning args # for the literal token "WITHSCORES", which could false-positive # if a returned field happened to be named "WITHSCORES". @@ -323,17 +382,24 @@ async def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: vector_param = value break - # Replace $vector placeholder with actual bytes + # Replace the $vector PARAMS value with actual bytes (query/VSIM + # references to $vector stay as parameter references). if vector_param: - for i, arg in enumerate(cmd): - if arg == "$vector": - cmd[i] = vector_param + self._inject_vector_param(cmd, vector_param) # Execute command asynchronously try: raw_result = await self._client.execute_command(*cmd) except redis.ResponseError as e: error_msg = str(e) + if translated.command == "FT.HYBRID" and self._is_unknown_command( + error_msg + ): + raise redis.ResponseError( + f"{error_msg}. hybrid_vector_search() translates to FT.HYBRID, " + "which requires Redis 8.4+ (RediSearch with hybrid search) " + "and redis-py >= 7.1.0." + ) from e _ismissing_signatures = ( "Unknown function", "No such function", @@ -352,10 +418,14 @@ async def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: raise # Parse result based on command type - count = raw_result[0] if raw_result else 0 - rows = [] - - if translated.command == "FT.SEARCH": + # FT.SEARCH/FT.AGGREGATE replies are arrays with the count first; + # FT.HYBRID replies are maps (dict) and set count during parsing below. + count = raw_result[0] if isinstance(raw_result, list) and raw_result else 0 + rows: list[dict] = [] + + if translated.command == "FT.HYBRID": + count, rows = self._parse_hybrid_reply(raw_result) + elif translated.command == "FT.SEARCH": with_scores = translated.score_alias is not None no_content = self._has_return_0(translated.args) diff --git a/sql_redis/parser.py b/sql_redis/parser.py index 61ba228..5c73564 100644 --- a/sql_redis/parser.py +++ b/sql_redis/parser.py @@ -156,6 +156,34 @@ class VectorSearchSpec: k: int | None = None +@dataclass +class HybridSearchSpec: + """Specification for FT.HYBRID fusion search. + + Populated when a SELECT projection contains + ``hybrid_vector_search(, , )``. The vector + leg reuses ``cosine_distance``/``vector_distance``/``vector_range`` and the + text leg reuses ``fulltext``; the third argument is ``rrf(...)`` or + ``linear(...)``. Distinct from ``VectorSearchSpec`` (pre-filter hybrid + search), which fuses nothing. + """ + + vector_field: str + text_field: str + text_query: str + alias: str = "hybrid_score" # combined-score column (the SELECT alias) + text_scorer: str = "BM25STD" # SEARCH scorer + vector_method: str = "KNN" # "KNN" | "RANGE" + ef_runtime: int | None = None # KNN tuning knob + radius: float | None = None # RANGE knob + epsilon: float | None = None # RANGE knob + combine_method: str = "RRF" # "RRF" | "LINEAR" + rrf_constant: int | None = None # RRF knob (server default 60) + rrf_window: int | None = None # fusion window knob (server default 20) + linear_alpha: float | None = None # LINEAR knob; beta derived as (1 - alpha) + k: int | None = None # KNN K, derived from LIMIT by the analyzer + + @dataclass class Condition: """A WHERE condition.""" @@ -256,6 +284,7 @@ class ParsedQuery: computed_fields: list[ComputedField] = dataclasses.field(default_factory=list) date_functions: list[DateFunctionSpec] = dataclasses.field(default_factory=list) vector_search: VectorSearchSpec | None = None + hybrid_search: HybridSearchSpec | None = None groupby_fields: list[str] = dataclasses.field(default_factory=list) orderby_fields: list[tuple[str, str]] = dataclasses.field( default_factory=list @@ -531,7 +560,10 @@ def _process_select_expression_inner( "quantile", "random_sample", } - if func_name_lower == "vector_distance": + if func_name_lower == "hybrid_vector_search": + # FT.HYBRID fusion: hybrid_vector_search(vector_leg, text_leg, combine) + self._process_hybrid_vector_search(expression, result, alias) + elif func_name_lower == "vector_distance": # Extract the vector field name from first argument if expression.expressions: first_arg = expression.expressions[0] @@ -634,6 +666,173 @@ def _process_vector_distance( alias=alias or "vector_distance", ) + def _process_hybrid_vector_search( + self, expression, result: ParsedQuery, alias: str | None + ) -> None: + """Process a hybrid_vector_search() call into a HybridSearchSpec. + + Shape: hybrid_vector_search(, , ) where + the vector leg is cosine_distance/vector_distance/vector_range, the text + leg is fulltext(field, 'query'), and the optional combine is rrf()/linear(). + """ + args = expression.expressions + if len(args) < 2: + raise ValueError( + "hybrid_vector_search() requires a vector leg and a text leg, " + "e.g. hybrid_vector_search(cosine_distance(field, :vec), " + "fulltext(field, 'query'), rrf())." + ) + + vector_field, vector_method, ef_runtime, radius, epsilon = ( + self._parse_hybrid_vector_leg(args[0]) + ) + text_field, text_query, text_scorer = self._parse_hybrid_text_leg(args[1]) + combine_method, rrf_constant, rrf_window, linear_alpha = ( + self._parse_hybrid_combine(args[2] if len(args) >= 3 else None) + ) + + result.hybrid_search = HybridSearchSpec( + vector_field=vector_field, + text_field=text_field, + text_query=text_query, + alias=alias or "hybrid_score", + text_scorer=text_scorer, + vector_method=vector_method, + ef_runtime=ef_runtime, + radius=radius, + epsilon=epsilon, + combine_method=combine_method, + rrf_constant=rrf_constant, + rrf_window=rrf_window, + linear_alpha=linear_alpha, + ) + + def _extract_function_kwargs(self, func_expr) -> dict[str, object]: + """Return a {name: value} mapping for ``name => value`` kwargs in a call.""" + kwargs: dict[str, object] = {} + for child in func_expr.expressions: + if isinstance(child, exp.Kwarg): + key = getattr(child.this, "name", None) or str(child.this) + kwargs[key.lower()] = self._extract_literal_value(child.expression) + return kwargs + + def _parse_hybrid_vector_leg(self, leg): + """Parse the vector leg of hybrid_vector_search(). + + Returns (field, method, ef_runtime, radius, epsilon). + """ + # cosine_distance()/L2 distance parse as builtins (this == field column). + if isinstance(leg, (exp.CosineDistance, exp.Distance)): + if not isinstance(leg.this, exp.Column): + raise ValueError( + "hybrid_vector_search() vector leg field must be a column name." + ) + return leg.this.name, "KNN", None, None, None + + # vector_distance()/vector_range() parse as anonymous functions, which + # (unlike the cosine_distance builtin) accept extra tuning kwargs. + if isinstance(leg, exp.Anonymous): + name = leg.name.lower() + if name not in ("vector_distance", "cosine_distance", "vector_range"): + raise ValueError( + "hybrid_vector_search() first argument must be a vector leg " + "(cosine_distance, vector_distance, or vector_range), " + f"got {leg.name}()." + ) + field_node = leg.expressions[0] if leg.expressions else None + if not isinstance(field_node, exp.Column): + raise ValueError( + "hybrid_vector_search() vector leg field must be a column name." + ) + kwargs = self._extract_function_kwargs(leg) + if name == "vector_range": + radius = kwargs.get("radius") + if radius is None: + raise ValueError( + "vector_range() requires a radius, e.g. " + "vector_range(field, :vec, radius => 0.2)." + ) + epsilon = kwargs.get("epsilon") + return ( + field_node.name, + "RANGE", + None, + float(radius), + float(epsilon) if epsilon is not None else None, + ) + ef = kwargs.get("ef_runtime") + return ( + field_node.name, + "KNN", + int(ef) if ef is not None else None, + None, + None, + ) + + raise ValueError( + "hybrid_vector_search() first argument must be a vector leg, " + "e.g. cosine_distance(field, :vec)." + ) + + def _parse_hybrid_text_leg(self, leg): + """Parse the fulltext() text leg. Returns (field, query, scorer).""" + if not isinstance(leg, exp.Anonymous) or leg.name.lower() != "fulltext": + raise ValueError( + "hybrid_vector_search() second argument must be " + "fulltext(field, 'query')." + ) + if len(leg.expressions) < 2: + raise ValueError( + "fulltext() in hybrid_vector_search() requires a field and a " + "query string." + ) + field_node = leg.expressions[0] + if not isinstance(field_node, exp.Column): + raise ValueError("fulltext() field must be a column name.") + query_val = self._extract_literal_value(leg.expressions[1]) + if not isinstance(query_val, str): + raise ValueError("fulltext() query must be a string literal.") + kwargs = self._extract_function_kwargs(leg) + scorer = kwargs.get("scorer", "BM25STD") + return field_node.name, query_val, str(scorer) + + def _parse_hybrid_combine(self, combine): + """Parse the optional rrf()/linear() combine. + + Returns (method, rrf_constant, rrf_window, linear_alpha). When ``combine`` + is None the server default (RRF) applies. + """ + if combine is None: + return "RRF", None, None, None + if not isinstance(combine, exp.Anonymous): + raise ValueError( + "hybrid_vector_search() fusion argument must be rrf(...) or " + "linear(...)." + ) + name = combine.name.lower() + kwargs = self._extract_function_kwargs(combine) + window = kwargs.get("window") + rrf_window = int(window) if window is not None else None + if name == "rrf": + constant = kwargs.get("constant") + return ( + "RRF", + int(constant) if constant is not None else None, + rrf_window, + None, + ) + if name == "linear": + alpha = kwargs.get("alpha") + return ( + "LINEAR", + None, + rrf_window, + float(alpha) if alpha is not None else None, + ) + raise ValueError( + f"Unknown fusion method {combine.name}(). Use rrf(...) or linear(...)." + ) + def _process_geo_distance_select( self, expression, result: ParsedQuery, alias: str | None ) -> None: diff --git a/sql_redis/schema.py b/sql_redis/schema.py index 936954a..1d89f31 100644 --- a/sql_redis/schema.py +++ b/sql_redis/schema.py @@ -3,7 +3,7 @@ from __future__ import annotations import asyncio -from typing import TYPE_CHECKING, Callable +from typing import TYPE_CHECKING, Any, Callable import redis @@ -11,41 +11,74 @@ import redis.asyncio as async_redis -def _parse_schema_from_info(info: list) -> dict[str, str]: - """Parse field types from FT.INFO response. +def _decode(value: Any) -> Any: + """Decode a bytes value to str; pass through everything else.""" + return value.decode("utf-8") if isinstance(value, bytes) else value - This is a pure function with no I/O operations, shared by both - sync and async schema registries. + +def _extract_attributes(info: Any) -> list: + """Pull the ``attributes`` section out of an FT.INFO reply. + + Handles both reply shapes: the RESP2 flat list + (``[..., 'attributes', [...], ...]``) and the redis-py 8.x / RESP3 map + (``{b'attributes': [...], ...}``), whose keys may be bytes or str. + """ + if isinstance(info, dict): + for key, val in info.items(): + if _decode(key) == "attributes": + return val or [] + return [] + for i, item in enumerate(info): + if _decode(item) == "attributes": + return info[i + 1] + return [] + + +def _attribute_name_and_type(attr: Any) -> tuple[str | None, str | None]: + """Extract (field_name, field_type) from a single FT.INFO attribute. + + An attribute is either a dict (redis-py 8.x), e.g. + ``{b'attribute': b'title', b'type': b'TEXT', ...}``, or a flat list, e.g. + ``[b'identifier', b'title', b'attribute', b'title', b'type', b'TEXT', ...]``. + """ + if isinstance(attr, dict): + name = next( + (_decode(v) for k, v in attr.items() if _decode(k) == "attribute"), None + ) + ftype = next( + (_decode(v) for k, v in attr.items() if _decode(k) == "type"), None + ) + return name, ftype + + name = None + ftype = None + for j, val in enumerate(attr): + val_str = _decode(val) + if val_str == "attribute" and j + 1 < len(attr): + name = _decode(attr[j + 1]) + if val_str == "type" and j + 1 < len(attr): + ftype = _decode(attr[j + 1]) + return name, ftype + + +def _parse_schema_from_info(info: Any) -> dict[str, str]: + """Parse field types from an FT.INFO response. + + This is a pure function with no I/O operations, shared by both the sync + and async schema registries. It accepts both the RESP2 list reply and the + redis-py 8.x / RESP3 map reply (see ``_extract_attributes``). Args: - info: The raw response from FT.INFO command. + info: The raw response from the FT.INFO command (list or dict). Returns: Dictionary mapping field names to their types (e.g., {"title": "TEXT"}). """ - schema = {} - # Find the 'attributes' section in the info response - for i, item in enumerate(info): - # Handle bytes or string comparison - item_str = item.decode("utf-8") if isinstance(item, bytes) else item - if item_str == "attributes": - attributes = info[i + 1] - for attr in attributes: - field_name = None - field_type = None - # Each attribute is a list like: - # [b'identifier', b'title', b'attribute', b'title', b'type', b'TEXT', ...] - for j, val in enumerate(attr): - val_str = val.decode("utf-8") if isinstance(val, bytes) else val - if val_str == "attribute" and j + 1 < len(attr): - fn = attr[j + 1] - field_name = fn.decode("utf-8") if isinstance(fn, bytes) else fn - if val_str == "type" and j + 1 < len(attr): - ft = attr[j + 1] - field_type = ft.decode("utf-8") if isinstance(ft, bytes) else ft - if field_name and field_type: - schema[field_name] = field_type - break + schema: dict[str, str] = {} + for attr in _extract_attributes(info): + field_name, field_type = _attribute_name_and_type(attr) + if field_name and field_type: + schema[field_name] = field_type return schema diff --git a/sql_redis/translator.py b/sql_redis/translator.py index ac7673f..372ce02 100644 --- a/sql_redis/translator.py +++ b/sql_redis/translator.py @@ -21,23 +21,39 @@ from sql_redis.schema import AsyncSchemaRegistry, SchemaRegistry +def _fmt_num(value: float) -> str: + """Format a numeric FT.HYBRID parameter without trailing float noise.""" + return f"{value:g}" + + @dataclass class TranslatedQuery: """Result of translating SQL to Redis.""" - command: str # FT.SEARCH or FT.AGGREGATE + command: str # FT.SEARCH, FT.AGGREGATE, or FT.HYBRID index: str query_string: str args: list[str] = field(default_factory=list) params: dict[str, object] = field(default_factory=dict) # Named parameters score_alias: str | None = None # Alias for score column when WITHSCORES is used + # FT.HYBRID has no single top-level query string (its SEARCH/VSIM legs live + # in args), so it is rendered without the quoted query_string slot. + is_hybrid: bool = False def to_command_list(self) -> list[str]: """Return as a list suitable for redis.execute_command().""" + if self.is_hybrid: + return [self.command, self.index, *self.args] return [self.command, self.index, self.query_string, *self.args] def to_command_string(self) -> str: """Return as a human-readable command string.""" + if self.is_hybrid: + # Quote multi-word tokens (the SEARCH query and filter expressions) + # for readability; execution uses to_command_list (raw tokens). + rendered = [self.command, self.index] + rendered.extend(f'"{tok}"' if " " in tok else tok for tok in self.args) + return " ".join(rendered) parts = [self.command, self.index, f'"{self.query_string}"'] parts.extend(self.args) return " ".join(parts) @@ -117,6 +133,11 @@ def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: """Build the Redis command from analyzed query.""" parsed = analyzed.parsed + # FT.HYBRID fusion is a dedicated command path with its own + # SEARCH/VSIM/COMBINE layout, distinct from FT.SEARCH/FT.AGGREGATE. + if analyzed.hybrid_search is not None: + return self._build_hybrid(analyzed) + # Validate: geo_distance cannot be combined with OR # Geo filters are applied as top-level command args (GEOFILTER/FILTER) and # are not part of the boolean expression. Combining with OR would change @@ -510,6 +531,106 @@ def _build_search( score_alias=(parsed.scoring.alias if parsed.scoring is not None else None), ) + def _build_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: + """Build an FT.HYBRID command from a hybrid_vector_search() query. + + Layout: ``FT.HYBRID index SEARCH "" [SCORER s] VSIM @field $vector + [FILTER n ] (KNN|RANGE ...) COMBINE (RRF|LINEAR ...) [LOAD ...] + [LIMIT ...] PARAMS 2 vector $vector DIALECT 2``. The WHERE clause is + applied to both legs: folded into the SEARCH query and emitted as the + VSIM FILTER so the candidate sets agree. + """ + parsed = analyzed.parsed + hybrid = analyzed.hybrid_search + assert hybrid is not None # guaranteed by the caller's dispatch check + spec = hybrid.spec + args: list[str] = [] + + # WHERE clause becomes the shared per-leg filter (reuses the standard + # query-string builder; vector_search is unset on the hybrid path). + filter_expr = self._build_query_string(analyzed) + has_filter = bool(filter_expr) and filter_expr != "*" + + # SEARCH leg: tokenized text query, with the filter folded in. + text_query = self._query_builder.build_text_condition( + spec.text_field, "FULLTEXT", spec.text_query + ) + search_query = f"({text_query}) ({filter_expr})" if has_filter else text_query + args.extend(["SEARCH", search_query]) + if spec.text_scorer: + args.extend(["SCORER", spec.text_scorer]) + + # VSIM leg: vector field + param placeholder, then the KNN/RANGE method + # clause, then the optional per-leg filter (the method must precede + # FILTER in the VSIM grammar). + args.extend(["VSIM", f"@{spec.vector_field}", "$vector"]) + if spec.vector_method == "RANGE": + assert spec.radius is not None # parser requires radius for RANGE + method_tokens = ["RADIUS", _fmt_num(spec.radius)] + if spec.epsilon is not None: + method_tokens.extend(["EPSILON", _fmt_num(spec.epsilon)]) + args.extend(["RANGE", str(len(method_tokens)), *method_tokens]) + else: + method_tokens = ["K", str(hybrid.k)] + if spec.ef_runtime is not None: + method_tokens.extend(["EF_RUNTIME", str(spec.ef_runtime)]) + args.extend(["KNN", str(len(method_tokens)), *method_tokens]) + if has_filter: + args.extend(["FILTER", "1", filter_expr]) + + # COMBINE: RRF (default) or LINEAR. The combined score is yielded under + # the SELECT alias so it comes back as a column. + combine_tokens: list[str] = [] + if spec.combine_method == "LINEAR": + if spec.linear_alpha is not None: + combine_tokens.extend( + [ + "ALPHA", + _fmt_num(spec.linear_alpha), + "BETA", + _fmt_num(1 - spec.linear_alpha), + ] + ) + if spec.rrf_window is not None: + combine_tokens.extend(["WINDOW", str(spec.rrf_window)]) + combine_tokens.extend(["YIELD_SCORE_AS", spec.alias]) + args.extend( + ["COMBINE", "LINEAR", str(len(combine_tokens)), *combine_tokens] + ) + else: + if spec.rrf_constant is not None: + combine_tokens.extend(["CONSTANT", str(spec.rrf_constant)]) + if spec.rrf_window is not None: + combine_tokens.extend(["WINDOW", str(spec.rrf_window)]) + combine_tokens.extend(["YIELD_SCORE_AS", spec.alias]) + args.extend(["COMBINE", "RRF", str(len(combine_tokens)), *combine_tokens]) + + # LOAD: the SELECT columns (field names require an @ prefix). + load_fields = [f for f in parsed.fields if f != "*"] + if load_fields: + args.extend( + ["LOAD", str(len(load_fields)), *(f"@{f}" for f in load_fields)] + ) + + # LIMIT (final row cut; KNN K is set separately above). + if parsed.limit is not None: + args.extend(["LIMIT", str(parsed.offset or 0), str(parsed.limit)]) + + # PARAMS placeholder — the executor injects the vector bytes for "vector". + # Note: FT.HYBRID rejects an explicit DIALECT argument (the server uses + # its configured search-default-dialect), so none is appended here. + args.extend(["PARAMS", "2", "vector", "$vector"]) + + return TranslatedQuery( + command="FT.HYBRID", + index=parsed.index, + query_string="", + args=args, + params={"vector": None}, + score_alias=spec.alias, + is_hybrid=True, + ) + def _build_geo_filter_args(self, geo_cond: GeoDistanceCondition) -> list[str]: """Build GEOFILTER args from a GeoDistanceCondition.""" return [ diff --git a/tests/conftest.py b/tests/conftest.py index 7a11da2..c0e7524 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -9,8 +9,12 @@ @pytest.fixture(scope="module") def redis_container(): - """Create a Redis 8 container for testing.""" - with RedisContainer(image="redis:8.0.2") as container: + """Create a Redis 8 container for testing. + + Uses 8.4+ so FT.HYBRID (hybrid_vector_search) integration tests can run; + older versions cause those tests to skip via a server-capability check. + """ + with RedisContainer(image="redis:8.4") as container: yield container diff --git a/tests/test_ft_hybrid.py b/tests/test_ft_hybrid.py new file mode 100644 index 0000000..46b3a1d --- /dev/null +++ b/tests/test_ft_hybrid.py @@ -0,0 +1,821 @@ +"""TDD tests for FT.HYBRID support via the hybrid_vector_search() function. + +These tests are written ahead of the implementation (RAAE-1322). They define the +contract for translating + + hybrid_vector_search(, , ) + +into a native ``FT.HYBRID`` command (Redis 8.4+), which fuses an independently +ranked text search and vector search server-side (RRF or LINEAR). This is distinct +from the existing pre-filter hybrid search, where text is only a hard prefilter and +the ranking comes from the vector leg alone. + +Expected (not yet implemented) API contract +-------------------------------------------- +``ParsedQuery.hybrid_search``: ``HybridSearchSpec | None``, populated when the SELECT +projection contains ``hybrid_vector_search(...)``. The function composes the existing +``cosine_distance(field, :vec)`` (vector leg) and ``fulltext(field, 'query')`` (text +leg) functions, plus a third ``rrf(...)`` / ``linear(...)`` argument for fusion. + +``HybridSearchSpec`` fields: + vector_field: str # VSIM field + text_field: str # SEARCH field + text_query: str # SEARCH query string + text_scorer: str = "BM25STD" # SEARCH scorer + vector_method: str = "KNN" # "KNN" | "RANGE" + ef_runtime: int | None # KNN tuning knob + radius / epsilon: float|None # RANGE knobs + combine_method: str = "RRF" # "RRF" | "LINEAR" + rrf_constant: int | None # RRF knob (default 60) + rrf_window: int | None # RRF/LINEAR window knob (default 20) + linear_alpha: float | None # LINEAR knob; beta derived as (1 - alpha) + alias: str # combined-score column (the SELECT alias) + k: int | None # KNN K, derived from LIMIT + +``AnalyzedQuery.hybrid_search``: ``HybridSearchAnalysis | None`` with field types +resolved. ``Translator.translate(...)`` returns a ``TranslatedQuery`` whose +``command == "FT.HYBRID"``. +""" + +import struct + +import pytest +import redis + +from sql_redis.analyzer import Analyzer +from sql_redis.executor import Executor +from sql_redis.parser import SQLParser +from sql_redis.schema import SchemaRegistry +from sql_redis.translator import Translator + + +def float_vector_to_bytes(vector: list[float]) -> bytes: + """Convert a list of floats to binary format for Redis vector storage.""" + return struct.pack(f"{len(vector)}f", *vector) + + +def _hybrid_supported(client: redis.Redis) -> bool: + """Return True if the connected server understands FT.HYBRID (Redis 8.4+).""" + try: + client.execute_command("FT.HYBRID") + except redis.ResponseError as exc: + message = str(exc).lower() + # No args -> arity/syntax error means the command exists; only an + # "unknown command" reply means the server is too old. + return "unknown command" not in message and "unknown subcommand" not in message + return True + + +@pytest.fixture +def sample_schema() -> dict[str, dict[str, str]]: + """Schema with text, tag, vector, and numeric fields for hybrid tests.""" + return { + "items": { + "name": "TEXT", + "category": "TAG", + "description": "TEXT", + "price": "NUMERIC", + "embedding": "VECTOR", + } + } + + +@pytest.fixture(scope="module") +def hybrid_translator(redis_client: redis.Redis, items_index: str) -> Translator: + """Translator with the items index (text + tag + vector) loaded.""" + registry = SchemaRegistry(redis_client) + registry.load_all() + return Translator(registry) + + +@pytest.fixture(scope="module") +def hybrid_executor(redis_client: redis.Redis, items_data: str) -> Executor: + """Executor against the items index; skips when FT.HYBRID is unavailable.""" + if not _hybrid_supported(redis_client): + pytest.skip("FT.HYBRID requires Redis 8.4+ (the test container is 8.0.2)") + registry = SchemaRegistry(redis_client) + registry.load_all() + return Executor(redis_client, registry) + + +# A canonical hybrid query reused across layers. +HYBRID_SQL = ( + "SELECT name, description, " + "hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone features'), " + "rrf()" + ") AS hybrid_score " + "FROM items " + "WHERE category = 'electronics' " + "ORDER BY hybrid_score DESC " + "LIMIT 5" +) + + +class TestHybridParserSelect: + """Parsing hybrid_vector_search() out of the SELECT clause.""" + + def test_detects_hybrid_search(self): + """A hybrid_vector_search() projection populates ParsedQuery.hybrid_search.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + assert result.hybrid_search is not None + assert result.index == "items" + + def test_extracts_vector_and_text_legs(self): + """The vector and text legs are extracted from the nested functions.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + spec = result.hybrid_search + assert spec.vector_field == "embedding" + assert spec.text_field == "description" + assert spec.text_query == "smartphone features" + + def test_combined_score_alias(self): + """The SELECT alias becomes the combined-score column name.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + assert result.hybrid_search.alias == "hybrid_score" + + def test_defaults_rrf_and_bm25std(self): + """rrf() with no scorer override yields RRF + the default BM25STD scorer.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + spec = result.hybrid_search + assert spec.combine_method == "RRF" + assert spec.text_scorer == "BM25STD" + + def test_defaults_to_knn_vector_method(self): + """cosine_distance() selects the KNN vector method by default.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + assert result.hybrid_search.vector_method == "KNN" + + def test_where_condition_is_preserved(self): + """The WHERE clause is retained as the per-leg filter.""" + parser = SQLParser() + result = parser.parse(HYBRID_SQL) + + assert len(result.conditions) == 1 + assert result.conditions[0].field == "category" + + +class TestHybridParserKnobs: + """Parsing the full set of fusion / leg knobs.""" + + def test_linear_alpha(self): + """linear(alpha => 0.3) selects LINEAR fusion with the given alpha.""" + parser = SQLParser() + result = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'phone'), " + "linear(alpha => 0.3)" + ") AS score FROM items LIMIT 5" + ) + + spec = result.hybrid_search + assert spec.combine_method == "LINEAR" + assert spec.linear_alpha == 0.3 + + def test_rrf_constant_and_window(self): + """rrf(constant => 60, window => 20) captures both RRF knobs.""" + parser = SQLParser() + result = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'phone'), " + "rrf(constant => 60, window => 20)" + ") AS score FROM items LIMIT 5" + ) + + spec = result.hybrid_search + assert spec.rrf_constant == 60 + assert spec.rrf_window == 20 + + def test_custom_text_scorer(self): + """fulltext(..., scorer => 'TFIDF') overrides the default scorer.""" + parser = SQLParser() + result = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'phone', scorer => 'TFIDF'), " + "rrf()" + ") AS score FROM items LIMIT 5" + ) + + assert result.hybrid_search.text_scorer == "TFIDF" + + def test_knn_ef_runtime(self): + """vector_distance(..., ef_runtime => 20) captures the KNN tuning knob. + + The tuning knob rides on vector_distance() rather than cosine_distance(): + sqlglot models cosine_distance as a built-in capped at 2 args, while + vector_distance() parses as an anonymous function and accepts the extra arg. + """ + parser = SQLParser() + result = parser.parse( + "SELECT name, hybrid_vector_search(" + "vector_distance(embedding, :vec, ef_runtime => 20), " + "fulltext(description, 'phone'), " + "rrf()" + ") AS score FROM items LIMIT 5" + ) + + assert result.hybrid_search.ef_runtime == 20 + + +class TestHybridParserValidation: + """Error handling for malformed hybrid_vector_search() calls.""" + + def test_missing_text_leg_raises(self): + """hybrid_vector_search() without a text leg is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec)" + ") AS score FROM items LIMIT 5" + ) + + def test_combine_omitted_defaults_to_rrf(self): + """A two-argument call (no combine) defaults to RRF fusion.""" + parser = SQLParser() + result = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'phone')" + ") AS score FROM items LIMIT 5" + ) + + assert result.hybrid_search.combine_method == "RRF" + assert result.hybrid_search.rrf_constant is None + + def test_non_distance_vector_leg_raises(self): + """A bare column as the vector leg is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="vector leg"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "embedding, fulltext(description, 'phone'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_unknown_vector_function_raises(self): + """An unrecognized vector-leg function is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="vector leg"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "made_up(embedding, :vec), fulltext(description, 'phone'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_text_leg_not_fulltext_raises(self): + """A non-fulltext() text leg is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="fulltext"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), made_up(description, 'p'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_text_leg_non_string_query_raises(self): + """A non-string fulltext() query is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="string literal"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), fulltext(description, 123), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_unknown_combine_method_raises(self): + """An unrecognized fusion function is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="rrf|linear"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), fulltext(description, 'p'), foo()" + ") AS score FROM items LIMIT 5" + ) + + def test_cosine_distance_non_column_field_raises(self): + """A literal (not a column) as the cosine_distance field is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="column name"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance('lit', :vec), fulltext(description, 'p'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_vector_distance_non_column_field_raises(self): + """A literal (not a column) as the vector_distance field is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="column name"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "vector_distance(123, :vec), fulltext(description, 'p'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_fulltext_insufficient_args_raises(self): + """fulltext() with only a field (no query) is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="field and a"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), fulltext(description), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_fulltext_non_column_field_raises(self): + """A literal (not a column) as the fulltext field is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="column name"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), fulltext(123, 'p'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_vector_range_without_radius_raises(self): + """vector_range() without a radius is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="radius"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "vector_range(embedding, :vec), fulltext(description, 'p'), rrf()" + ") AS score FROM items LIMIT 5" + ) + + def test_non_function_combine_raises(self): + """A literal (not rrf/linear) as the fusion argument is rejected.""" + parser = SQLParser() + with pytest.raises(ValueError, match="rrf|linear"): + parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), fulltext(description, 'p'), 99" + ") AS score FROM items LIMIT 5" + ) + + +class TestHybridAnalyzer: + """Analyzing hybrid_vector_search() against a schema.""" + + def test_detects_hybrid_search(self, sample_schema): + """Analyzer surfaces the hybrid search on the analyzed query.""" + parser = SQLParser() + parsed = parser.parse(HYBRID_SQL) + result = Analyzer(sample_schema).analyze(parsed) + + assert result.hybrid_search is not None + + def test_resolves_leg_field_types(self, sample_schema): + """Both leg fields resolve to their schema types.""" + parser = SQLParser() + parsed = parser.parse(HYBRID_SQL) + result = Analyzer(sample_schema).analyze(parsed) + + assert result.get_field_type("embedding") == "VECTOR" + assert result.get_field_type("description") == "TEXT" + + def test_knn_k_derived_from_limit(self, sample_schema): + """LIMIT becomes the KNN K for the vector leg.""" + parser = SQLParser() + parsed = parser.parse(HYBRID_SQL) + result = Analyzer(sample_schema).analyze(parsed) + + assert result.hybrid_search.k == 5 + + def test_vector_leg_on_non_vector_field_raises(self, sample_schema): + """Using a non-VECTOR field as the vector leg is an error.""" + parser = SQLParser() + parsed = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(price, :vec), " + "fulltext(description, 'phone'), " + "rrf()" + ") AS score FROM items LIMIT 5" + ) + with pytest.raises(ValueError): + Analyzer(sample_schema).analyze(parsed) + + def test_text_leg_on_non_text_field_raises(self, sample_schema): + """Using a non-TEXT field as the text leg is an error.""" + parser = SQLParser() + parsed = parser.parse( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(price, 'phone'), " + "rrf()" + ") AS score FROM items LIMIT 5" + ) + with pytest.raises(ValueError): + Analyzer(sample_schema).analyze(parsed) + + +class TestHybridTranslator: + """Translating hybrid_vector_search() to an FT.HYBRID command.""" + + def test_emits_ft_hybrid_command( + self, hybrid_translator: Translator, items_index: str + ): + """The translated command targets FT.HYBRID, not FT.SEARCH/FT.AGGREGATE.""" + result = hybrid_translator.translate(HYBRID_SQL) + + assert result.command == "FT.HYBRID" + + def test_command_has_search_and_vsim_legs( + self, hybrid_translator: Translator, items_index: str + ): + """Both the SEARCH and VSIM legs appear in the rendered command.""" + cmd = hybrid_translator.translate(HYBRID_SQL).to_command_string() + + assert "SEARCH" in cmd + assert "VSIM" in cmd + assert "@embedding" in cmd + + def test_command_has_rrf_combine( + self, hybrid_translator: Translator, items_index: str + ): + """Default fusion renders a COMBINE RRF clause.""" + cmd = hybrid_translator.translate(HYBRID_SQL).to_command_string() + + assert "COMBINE" in cmd + assert "RRF" in cmd + + def test_command_omits_dialect( + self, hybrid_translator: Translator, items_index: str + ): + """FT.HYBRID rejects an explicit DIALECT argument, so none is emitted.""" + result = hybrid_translator.translate(HYBRID_SQL) + + assert "DIALECT" not in result.to_command_string() + + def test_where_becomes_filter( + self, hybrid_translator: Translator, items_index: str + ): + """The WHERE clause is rendered as a category filter on the legs.""" + cmd = hybrid_translator.translate(HYBRID_SQL).to_command_string() + + assert "electronics" in cmd + + def test_linear_combine_renders_alpha( + self, hybrid_translator: Translator, items_index: str + ): + """A linear() fusion renders COMBINE LINEAR with ALPHA.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "linear(alpha => 0.3)" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "LINEAR" in cmd + assert "ALPHA" in cmd + + def test_linear_derives_beta_from_alpha( + self, hybrid_translator: Translator, items_index: str + ): + """LINEAR exposes alpha only; beta is derived as (1 - alpha).""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "linear(alpha => 0.3)" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "ALPHA 0.3" in cmd + assert "BETA 0.7" in cmd + + def test_rrf_constant_and_window_in_command( + self, hybrid_translator: Translator, items_index: str + ): + """RRF knobs render as CONSTANT and WINDOW.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "rrf(constant => 60, window => 20)" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "CONSTANT 60" in cmd + assert "WINDOW 20" in cmd + + def test_knn_ef_runtime_in_command( + self, hybrid_translator: Translator, items_index: str + ): + """A KNN ef_runtime knob renders EF_RUNTIME in the VSIM leg.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "vector_distance(embedding, :vec, ef_runtime => 20), " + "fulltext(description, 'smartphone'), " + "rrf()" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "EF_RUNTIME 20" in cmd + + def test_range_method_renders_radius( + self, hybrid_translator: Translator, items_index: str + ): + """A vector_range() leg renders a RANGE method with RADIUS/EPSILON.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "vector_range(embedding, :vec, radius => 0.2, epsilon => 0.01), " + "fulltext(description, 'smartphone'), " + "rrf()" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "RANGE" in cmd + assert "RADIUS 0.2" in cmd + assert "EPSILON 0.01" in cmd + + def test_linear_window_in_command( + self, hybrid_translator: Translator, items_index: str + ): + """A linear() window knob renders WINDOW in the COMBINE clause.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "linear(alpha => 0.3, window => 30)" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "WINDOW 30" in cmd + + def test_range_without_epsilon( + self, hybrid_translator: Translator, items_index: str + ): + """vector_range() without epsilon renders RADIUS and no EPSILON.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "vector_range(embedding, :vec, radius => 0.2), " + "fulltext(description, 'smartphone'), " + "rrf()" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "RADIUS 0.2" in cmd + assert "EPSILON" not in cmd + + def test_score_only_select_omits_load( + self, hybrid_translator: Translator, items_index: str + ): + """With only the fused score projected, no LOAD clause is emitted.""" + cmd = hybrid_translator.translate( + "SELECT hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "rrf()" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "LOAD" not in cmd + + def test_no_where_omits_vsim_filter( + self, hybrid_translator: Translator, items_index: str + ): + """With no WHERE clause, the VSIM leg carries no FILTER.""" + cmd = hybrid_translator.translate( + "SELECT name, hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "rrf()" + f") AS score FROM {items_index} LIMIT 5" + ).to_command_string() + + assert "FILTER" not in cmd + + +class _FakeRegistry: + """Minimal schema registry returning the items schema for translation.""" + + def get_schema(self, index: str) -> dict[str, str]: + return { + "name": "TEXT", + "category": "TAG", + "description": "TEXT", + "embedding": "VECTOR", + } + + +class _FakeClient: + """Sync client stub that returns a canned reply for execute_command.""" + + def __init__(self, reply): + self.reply = reply + self.last_command: tuple | None = None + + def execute_command(self, *args): + self.last_command = args + if isinstance(self.reply, Exception): + raise self.reply + return self.reply + + +_HYBRID_SQL_NO_WHERE = ( + "SELECT name, description, " + "hybrid_vector_search(" + "cosine_distance(embedding, :vec), " + "fulltext(description, 'smartphone'), " + "rrf()" + ") AS hybrid_score " + "FROM items LIMIT 5" +) + + +class TestHybridExecutorVersionGuard: + """The executor raises a clear error when FT.HYBRID is unsupported.""" + + def test_unknown_command_raises_version_hint(self): + """An 'unknown command' reply is rewrapped with the 8.4 requirement.""" + client = _FakeClient(redis.ResponseError("ERR unknown command 'FT.HYBRID'")) + executor = Executor(client, _FakeRegistry()) + + with pytest.raises(redis.ResponseError, match="8.4"): + executor.execute(_HYBRID_SQL_NO_WHERE, params={"vec": b"\x00" * 16}) + + +class TestHybridExecutorParsing: + """The executor parses FT.HYBRID replies into rows with the fused score.""" + + def test_parses_rows_with_combined_score(self): + """A hybrid reply maps field/value pairs (incl. the score) into rows.""" + reply = [ + "total_results", + 1, + "results", + [["name", "iPhone 15", "description", "smartphone", "hybrid_score", "0.5"]], + "warnings", + [], + "execution_time", + "0.1", + ] + client = _FakeClient(reply) + executor = Executor(client, _FakeRegistry()) + + result = executor.execute(_HYBRID_SQL_NO_WHERE, params={"vec": b"\x00" * 16}) + + assert result.count == 1 + assert result.rows[0]["name"] == "iPhone 15" + assert result.rows[0]["hybrid_score"] == "0.5" + + def test_parses_rows_from_resp3_dict_reply(self): + """A redis-py 8.x / RESP3 map reply (dict of dict rows) parses to rows.""" + reply = { + b"total_results": 1, + b"results": [{b"name": b"iPhone 15", b"hybrid_score": b"0.5"}], + b"warnings": [], + b"execution_time": 0.1, + } + client = _FakeClient(reply) + executor = Executor(client, _FakeRegistry()) + + result = executor.execute(_HYBRID_SQL_NO_WHERE, params={"vec": b"\x00" * 16}) + + assert result.count == 1 + assert result.rows[0][b"name"] == b"iPhone 15" + assert result.rows[0][b"hybrid_score"] == b"0.5" + + def test_vector_bytes_injected_into_command(self): + """The vector param bytes replace the $vector placeholder in the command.""" + client = _FakeClient([0]) + executor = Executor(client, _FakeRegistry()) + blob = b"\x01" * 16 + + executor.execute(_HYBRID_SQL_NO_WHERE, params={"vec": blob}) + + assert client.last_command[0] == "FT.HYBRID" + # The bytes are injected as the PARAMS value... + assert blob in client.last_command + # ...while the VSIM leg keeps the $vector parameter reference. + assert "$vector" in client.last_command + + +class _FakeAsyncRegistry: + """Minimal async schema registry for the async executor unit tests.""" + + async def ensure_schema(self, index: str) -> None: + return None + + def get_schema(self, index: str) -> dict[str, str]: + return { + "name": "TEXT", + "category": "TAG", + "description": "TEXT", + "embedding": "VECTOR", + } + + +class _FakeAsyncClient: + """Async client stub that returns a canned reply for execute_command.""" + + def __init__(self, reply): + self.reply = reply + self.last_command: tuple | None = None + + async def execute_command(self, *args): + self.last_command = args + if isinstance(self.reply, Exception): + raise self.reply + return self.reply + + +class TestHybridAsyncExecutor: + """Async executor mirrors the sync FT.HYBRID guard and parsing.""" + + async def test_async_version_guard(self): + """An 'unknown command' reply is rewrapped with the 8.4 requirement.""" + from sql_redis.executor import AsyncExecutor + + client = _FakeAsyncClient( + redis.ResponseError("ERR unknown command 'FT.HYBRID'") + ) + executor = AsyncExecutor(client, _FakeAsyncRegistry()) + + with pytest.raises(redis.ResponseError, match="8.4"): + await executor.execute(_HYBRID_SQL_NO_WHERE, params={"vec": b"\x00" * 16}) + + async def test_async_parses_rows(self): + """An async hybrid reply parses into rows with the fused score.""" + from sql_redis.executor import AsyncExecutor + + reply = [ + "total_results", + 1, + "results", + [["name", "iPhone 15", "hybrid_score", "0.5"]], + "warnings", + [], + ] + client = _FakeAsyncClient(reply) + executor = AsyncExecutor(client, _FakeAsyncRegistry()) + + result = await executor.execute( + _HYBRID_SQL_NO_WHERE, params={"vec": b"\x00" * 16} + ) + + assert result.rows[0]["name"] == "iPhone 15" + assert result.rows[0]["hybrid_score"] == "0.5" + + +class TestHybridFusionIntegration: + """End-to-end FT.HYBRID execution (requires Redis 8.4+).""" + + def test_returns_fused_rows(self, hybrid_executor: Executor, items_data: str): + """A hybrid fusion query returns rows with the combined-score column.""" + query_vector = float_vector_to_bytes([0.1, 0.2, 0.3, 0.4]) + + result = hybrid_executor.execute( + f""" + SELECT name, description, + hybrid_vector_search( + cosine_distance(embedding, :vec), + fulltext(description, 'smartphone features'), + rrf() + ) AS hybrid_score + FROM {items_data} + WHERE category = 'electronics' + ORDER BY hybrid_score DESC + LIMIT 5 + """, + params={"vec": query_vector}, + ) + + assert len(result.rows) >= 1 + assert "hybrid_score" in result.rows[0] + + def test_linear_fusion_executes(self, hybrid_executor: Executor, items_data: str): + """LINEAR fusion with an alpha weight executes end-to-end.""" + query_vector = float_vector_to_bytes([0.1, 0.2, 0.3, 0.4]) + + result = hybrid_executor.execute( + f""" + SELECT name, + hybrid_vector_search( + cosine_distance(embedding, :vec), + fulltext(description, 'smartphone'), + linear(alpha => 0.3) + ) AS hybrid_score + FROM {items_data} + LIMIT 5 + """, + params={"vec": query_vector}, + ) + + assert len(result.rows) >= 1 diff --git a/tests/test_schema_registry.py b/tests/test_schema_registry.py index 99026cd..91683cc 100644 --- a/tests/test_schema_registry.py +++ b/tests/test_schema_registry.py @@ -247,6 +247,42 @@ def test_parse_schema_incomplete_attribute(self): # Only field2 should be captured (field1 has no type) assert schema == {"field2": "TEXT"} + def test_parse_schema_dict_reply_with_bytes_keys(self): + """_parse_schema_from_info handles the redis-py 8.x / RESP3 map reply. + + redis-py 8.x applies a response callback to FT.INFO, returning a dict + with bytes keys whose ``attributes`` value is a list of dicts. + """ + fake_info = { + b"index_name": b"items", + b"attributes": [ + {b"identifier": b"title", b"attribute": b"title", b"type": b"TEXT"}, + {b"identifier": b"genre", b"attribute": b"genre", b"type": b"TAG"}, + { + b"identifier": b"embedding", + b"attribute": b"embedding", + b"type": b"VECTOR", + }, + ], + } + schema = _parse_schema_from_info(fake_info) + + assert schema == {"title": "TEXT", "genre": "TAG", "embedding": "VECTOR"} + + def test_parse_schema_dict_reply_without_attributes(self): + """A dict reply with no attributes section yields an empty schema.""" + assert _parse_schema_from_info({b"index_name": b"items"}) == {} + + def test_parse_schema_dict_attribute_missing_type(self): + """A dict attribute without a type is skipped.""" + fake_info = { + b"attributes": [ + {b"attribute": b"field1"}, # no type + {b"attribute": b"field2", b"type": b"TEXT"}, + ], + } + assert _parse_schema_from_info(fake_info) == {"field2": "TEXT"} + class TestSchemaRegistryRefresh: """Tests for schema refresh functionality.""" From 559b683118c170a3422af4459f03e0f95196dd61 Mon Sep 17 00:00:00 2001 From: Robert Shelton Date: Wed, 24 Jun 2026 09:51:59 -0400 Subject: [PATCH 2/2] remove proposal docs --- docs/proposals/README.md | 137 ------ docs/proposals/ft-hybrid-code-comparison.md | 223 ---------- .../ft-hybrid-implementation-sketch.md | 402 ------------------ docs/proposals/ft-hybrid-primitive-design.md | 295 ------------- docs/proposals/ft-hybrid.md | 295 ------------- 5 files changed, 1352 deletions(-) delete mode 100644 docs/proposals/README.md delete mode 100644 docs/proposals/ft-hybrid-code-comparison.md delete mode 100644 docs/proposals/ft-hybrid-implementation-sketch.md delete mode 100644 docs/proposals/ft-hybrid-primitive-design.md delete mode 100644 docs/proposals/ft-hybrid.md diff --git a/docs/proposals/README.md b/docs/proposals/README.md deleted file mode 100644 index 81e9a49..0000000 --- a/docs/proposals/README.md +++ /dev/null @@ -1,137 +0,0 @@ -# FT.HYBRID Proposal Documents - -**JIRA:** [RAAE-1322](https://redislabs.atlassian.net/browse/RAAE-1322) -**Status:** Design recommendation phase -**Date:** 2026-06-23 - -## Overview - -This directory contains design documents for adding `FT.HYBRID` support to sql-redis. The proposal introduces a new primitive abstraction that cleanly expresses server-side hybrid fusion (text + vector search combined via RRF or LINEAR). - -## Documents - -### 1. [ft-hybrid.md](ft-hybrid.md) - Original Proposal -**Author:** Robert Shelton -**Purpose:** Initial spec with three syntax designs (A, B, C) - -Contains: -- Goal: Enable server-side fusion (RRF/LINEAR) instead of filter-then-KNN -- Three syntax designs (A: overload, B: hybrid() predicate, C: composable) -- SQL → FT.HYBRID mapping -- Implementation plan by layer -- Testing strategy - -**Key decision point:** Design C (composable, fusion in ORDER BY) was recommended. - -### 2. [ft-hybrid-primitive-design.md](ft-hybrid-primitive-design.md) - Design Recommendation ⭐ -**Author:** Claude (review of original) -**Purpose:** Identify the core design issue and propose a better primitive - -**Main insight:** The original designs treat hybrid fusion as a **syntax mapping problem** rather than a **data structure problem**. This document proposes: - -- **New primitive:** `HybridFusionSpec` dataclass that encapsulates text leg + vector leg + fusion config -- **Design C+:** Keep Design C's SQL syntax but use dedicated primitives internally -- **Type-driven dispatch:** Command path is determined by data structure, not heuristics - -**Key benefits:** -- Single source of truth for fusion state -- Explicit over implicit -- Clean extensibility (add fusion methods, add legs) -- No risk of breaking existing filter-then-KNN - -### 3. [ft-hybrid-code-comparison.md](ft-hybrid-code-comparison.md) - Concrete Comparison -**Purpose:** Side-by-side code showing Design C vs Design C+ - -Compares: -- Parser output (scattered state vs single spec) -- Translator dispatch (heuristics vs type-driven) -- Command builder (extraction logic vs direct access) - -**Takeaway:** Design C+ is cleaner, more maintainable, and safer. - -### 4. [ft-hybrid-implementation-sketch.md](ft-hybrid-implementation-sketch.md) - Code Sketch -**Purpose:** Concrete implementation examples for Design C+ - -Shows: -- Data class definitions (TextRankingLeg, VectorRankingLeg, HybridFusionSpec) -- Parser detection logic (_process_order_by, _build_hybrid_fusion_spec) -- Analyzer validation (_analyze_hybrid_fusion) -- Translator command builder (_build_ft_hybrid) -- Executor version gating (_check_hybrid_support) - -**Takeaway:** All changes are additive and backward compatible. - -## Recommended Path Forward - -**Adopt Design C+ (primitive-based) for the following reasons:** - -1. **Better data model:** `HybridFusionSpec` makes fusion first-class, not inferred. -2. **Cleaner code:** No heuristics, no scattered state, no conditional detection. -3. **Safer extension:** Adding new fusion methods or legs doesn't require refactoring. -4. **Backward compatible:** Existing vector_distance() queries work unchanged. - -## SQL Syntax (Final Recommendation) - -```sql -SELECT page_text, file_id, - vector_distance(embedding, :vec) AS vscore, - fulltext(page_text, 'quarterly earnings') AS tscore -FROM "KM_abc123" -WHERE ticker = 'MSFT' -ORDER BY rrf(vscore, tscore, constant => 60) DESC -LIMIT 10; -``` - -**Detection rules:** -- `vector_distance(...) AS ` in SELECT → vector leg -- `fulltext(...) AS ` in SELECT → text leg -- `rrf()` or `linear()` in ORDER BY → fusion trigger - -If all three are present → `FT.HYBRID`. -If only vector_distance → `FT.SEARCH` with KNN (existing behavior). - -## Implementation Checklist - -- [ ] Add `HybridFusionSpec`, `TextRankingLeg`, `VectorRankingLeg` to parser.py -- [ ] Add detection logic in `SQLParser._process_order_by()` -- [ ] Add `HybridFusionAnalysis` to analyzer.py with field type validation -- [ ] Add `Translator._build_ft_hybrid()` command builder -- [ ] Add version gating in `Executor.execute()` (Redis 8.4+ check) -- [ ] Bump test Redis image to 8.4+ in conftest.py -- [ ] Add unit tests for parser, analyzer, translator -- [ ] Add integration tests in test_sql_queries.py -- [ ] Update docs: relabel "hybrid" → "filtered KNN", add "Hybrid Fusion" guide -- [ ] Update AGENTS.md and llms.txt with FT.HYBRID info - -**Estimated effort:** 3-4 days (implementation + tests + docs) - -## Open Questions - -1. **Fusion defaults:** Require explicit `rrf()/linear()` or default to RRF? - **Recommendation:** Require explicit (fail fast on ambiguity). - -2. **K vs WINDOW:** Default `K = window` or make independent? - **Recommendation:** Default `K = window` for v1; add kwarg later if needed. - -3. **Filter distribution:** Apply WHERE to both legs or SEARCH-only? - **Recommendation:** Both legs (consistency > simplicity). - -4. **Score surfacing:** Auto-project fused score or require explicit SELECT? - **Recommendation:** Let users SELECT it explicitly if needed. - -## Next Steps - -1. Confirm design decision (C+ vs C) with stakeholders. -2. Confirm Redis 8.4 availability in test environment. -3. Begin implementation with parser/analyzer/translator changes. -4. Add tests at each layer (unit → integration). -5. Update docs and deploy. - -## Related Issues - -- **Terminology fix:** Existing docs call filter-then-KNN "hybrid search" but that's not `FT.HYBRID`. Need to relabel to "filtered KNN" vs "hybrid fusion". -- **RedisVL alignment:** If RedisVL adds FT.HYBRID support, mirror parameter names (constant, window, alpha, beta). - ---- - -**Questions?** See the individual docs above or reach out on RAAE-1322. diff --git a/docs/proposals/ft-hybrid-code-comparison.md b/docs/proposals/ft-hybrid-code-comparison.md deleted file mode 100644 index 67f860c..0000000 --- a/docs/proposals/ft-hybrid-code-comparison.md +++ /dev/null @@ -1,223 +0,0 @@ -# FT.HYBRID: Design C vs Design C+ Code Comparison - -**Purpose:** Show concrete code differences between the original Design C (detection-based) and the recommended Design C+ (primitive-based). - -## Scenario: Parse this SQL - -```sql -SELECT page_text, file_id, - vector_distance(embedding, :vec) AS vscore, - fulltext(page_text, 'quarterly earnings') AS tscore -FROM "KM_abc123" -WHERE ticker = 'MSFT' -ORDER BY rrf(vscore, tscore, constant => 60) DESC -LIMIT 10; -``` - ---- - -## Design C (Original): Detection-Based - -### Parser Output - -```python -ParsedQuery( - index="KM_abc123", - fields=["page_text", "file_id"], - vector_search=VectorSearchSpec( - field="embedding", - alias="vscore", - k=None - ), - conditions=[ - Condition(field="page_text", operator="FULLTEXT", value="quarterly earnings"), - Condition(field="ticker", operator="=", value="MSFT") - ], - orderby_fields=[("vscore", "DESC")], # ??? How to represent rrf(vscore, tscore)? - # Problem: No clear place to store fusion method + parameters - # Requires adding fields like: - # fusion_method: str | None = None - # fusion_text_alias: str | None = None - # fusion_params: dict | None = None -) -``` - -**Issues:** -1. `fulltext()` in SELECT is represented as a `Condition`, even though it's not a filter. -2. Fusion method (`rrf`) and its parameters are scattered across multiple optional fields. -3. `orderby_fields` can't represent `rrf(vscore, tscore)` cleanly. -4. Analyzer needs heuristics to detect "this is fusion, not filter-then-KNN." - -### Translator Logic - -```python -def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - # Heuristic detection: if we have vector_search AND a FULLTEXT condition - # that's not in WHERE, assume hybrid fusion - has_text_in_select = any( - c.operator == "FULLTEXT" and c not in analyzed.filters - for c in analyzed.parsed.conditions - ) - - if analyzed.vector_search and has_text_in_select: - # Infer this is FT.HYBRID - if analyzed.parsed.fusion_method == "RRF": - return self._build_ft_hybrid_rrf(analyzed) - elif analyzed.parsed.fusion_method == "LINEAR": - return self._build_ft_hybrid_linear(analyzed) - else: - raise ValueError("Fusion method not detected") - elif analyzed.vector_search: - # Filter-then-KNN - return self._build_ft_search(analyzed) - # ... -``` - -**Problems:** -- Lots of conditional logic to distinguish fusion from filter-then-KNN. -- Hard to extend when adding new fusion methods. -- Fragile: what if someone writes `fulltext()` in SELECT but doesn't want fusion? - ---- - -## Design C+ (Recommended): Primitive-Based - -### Parser Output - -```python -ParsedQuery( - index="KM_abc123", - fields=["page_text", "file_id"], - hybrid_fusion=HybridFusionSpec( - text_leg=TextRankingLeg( - field="page_text", - query="quarterly earnings", - alias="tscore", - scorer="BM25" - ), - vector_leg=VectorRankingLeg( - field="embedding", - alias="vscore", - k=None - ), - fusion_method="RRF", - rrf_constant=60, - window=20 - ), - conditions=[ - Condition(field="ticker", operator="=", value="MSFT") - ] -) -``` - -**Benefits:** -1. All hybrid fusion state lives in one object. -2. `fulltext()` is explicitly a ranking signal, not a filter. -3. Filters are separate from ranking legs. -4. No ambiguity: if `hybrid_fusion` is set, it's `FT.HYBRID`. - -### Translator Logic - -```python -def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - if analyzed.hybrid_fusion: - return self._build_ft_hybrid(analyzed) - elif analyzed.vector_search: - return self._build_ft_search(analyzed) - elif analyzed.use_aggregate: - return self._build_ft_aggregate(analyzed) - else: - return self._build_ft_search(analyzed) -``` - -**Benefits:** -- **Type-driven dispatch:** the data structure dictates the command path. -- No heuristics, no inference, no scattered state. -- Easy to add new fusion methods: just add a field to `HybridFusionSpec`. - ---- - -## Side-by-Side: Building FT.HYBRID Command - -### Design C (Detection-Based) - -```python -def _build_ft_hybrid_rrf(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - # Extract text query from conditions - text_cond = next(c for c in analyzed.parsed.conditions if c.operator == "FULLTEXT") - text_field = text_cond.field - text_query = text_cond.value - text_alias = analyzed.parsed.fusion_text_alias or "tscore" - - # Extract vector field from vector_search - vector_field = analyzed.vector_search.field - vector_alias = analyzed.vector_search.alias or "vscore" - - # Extract fusion params - rrf_constant = analyzed.parsed.fusion_params.get("constant", 60) - window = analyzed.parsed.fusion_params.get("window", 20) - - # Build filters - filters = [c for c in analyzed.parsed.conditions if c.operator != "FULLTEXT"] - filter_expr = self._build_filter_expr(filters) - - # Assemble command - search_leg = f'SEARCH "(@{text_field}:({text_query})) {filter_expr}" YIELD_SCORE_AS {text_alias}' - vsim_leg = f'VSIM @{vector_field} $vec FILTER 1 "{filter_expr}" KNN 2 K {window} YIELD_SCORE_AS {vector_alias}' - combine = f'COMBINE RRF 2 CONSTANT {rrf_constant} WINDOW {window}' - # ... -``` - -**Problems:** -- Scattered extraction logic. -- Fragile assumptions (e.g., "first FULLTEXT condition is the ranking signal"). -- Duplicate filter-building logic. - -### Design C+ (Primitive-Based) - -```python -def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - fusion = analyzed.hybrid_fusion.spec - - # All fusion state is in one place - text_field = fusion.text_leg.field - text_query = fusion.text_leg.query - text_alias = fusion.text_leg.alias - - vector_field = fusion.vector_leg.field - vector_alias = fusion.vector_leg.alias - k = fusion.vector_leg.k or fusion.window - - # Filters are already separated - filter_expr = self._build_filter_expr(analyzed.hybrid_fusion.filters) - - # Assemble command - search_leg = f'SEARCH "(@{text_field}:({text_query})) {filter_expr}" YIELD_SCORE_AS {text_alias}' - vsim_leg = f'VSIM @{vector_field} $vec FILTER 1 "{filter_expr}" KNN 2 K {k} YIELD_SCORE_AS {vector_alias}' - - if fusion.fusion_method == "RRF": - combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' - else: # LINEAR - combine = f'COMBINE LINEAR 2 ALPHA {fusion.linear_alpha} BETA {fusion.linear_beta} WINDOW {fusion.window}' - # ... -``` - -**Benefits:** -- All data is accessible via the spec. -- No searching through conditions. -- Handles both RRF and LINEAR in one method. - ---- - -## Summary - -| Aspect | Design C (Detection) | Design C+ (Primitive) | -|--------|----------------------|----------------------| -| **Data model** | Scattered across `vector_search`, `conditions`, optional fields | Single `HybridFusionSpec` object | -| **Parser complexity** | Must distinguish "fulltext as filter" vs "fulltext as ranking" | Explicit: `WHERE fulltext()` = filter, `SELECT fulltext()` = ranking leg | -| **Translator dispatch** | Heuristics and inference | Type-driven (if `hybrid_fusion`, call `_build_ft_hybrid`) | -| **Command builder** | Separate methods for RRF/LINEAR, duplication | Single method, `if fusion_method` switch | -| **Extensibility** | Add more optional fields, more heuristics | Add fields to `HybridFusionSpec`, no heuristics | -| **Backward compat** | Risk of breaking filter-then-KNN if detection misfires | Safe: new field, existing queries untouched | - -**Recommendation:** Use Design C+ (primitive-based) to avoid technical debt and make the codebase easier to maintain as `FT.HYBRID` evolves. diff --git a/docs/proposals/ft-hybrid-implementation-sketch.md b/docs/proposals/ft-hybrid-implementation-sketch.md deleted file mode 100644 index 552073a..0000000 --- a/docs/proposals/ft-hybrid-implementation-sketch.md +++ /dev/null @@ -1,402 +0,0 @@ -# FT.HYBRID Implementation Sketch (Design C+) - -**Purpose:** Concrete code examples showing how to implement the `HybridFusionSpec` primitive in sql-redis. - -## 1. Parser Changes (parser.py) - -### New Data Classes - -```python -@dataclass -class TextRankingLeg: - """Text ranking leg for hybrid fusion.""" - field: str # TEXT field name - query: str | None # Search query string - alias: str # Score alias (e.g., "tscore") - scorer: str = "BM25" # BM25, TFIDF, DISMAX - -@dataclass -class VectorRankingLeg: - """Vector ranking leg for hybrid fusion.""" - field: str # VECTOR field name - alias: str # Score alias (e.g., "vscore") - k: int | None = None # KNN K (defaults to window if None) - -@dataclass -class HybridFusionSpec: - """Specification for FT.HYBRID server-side fusion. - - Represents two independently-ranked legs (text + vector) fused by - RRF or LINEAR combination. Mutually exclusive with VectorSearchSpec - (filter-then-KNN). - """ - text_leg: TextRankingLeg - vector_leg: VectorRankingLeg - fusion_method: str # "RRF" or "LINEAR" - - # RRF parameters - rrf_constant: int = 60 - - # LINEAR parameters - linear_alpha: float = 0.5 - linear_beta: float = 0.5 - - # Common parameters - window: int = 20 # Fusion window size -``` - -### Update ParsedQuery - -```python -@dataclass -class ParsedQuery: - # ... existing fields ... - vector_search: VectorSearchSpec | None = None # Filter-then-KNN (existing) - hybrid_fusion: HybridFusionSpec | None = None # Server-side fusion (NEW) - # Note: vector_search and hybrid_fusion are mutually exclusive -``` - -### Detection Logic in _process_order_by - -```python -def _process_order_by(self, order: exp.Order, result: ParsedQuery) -> None: - """Process ORDER BY clause, detecting fusion functions.""" - for ordered in order.expressions: - expression = ordered.this - - # Check if it's a fusion function: rrf() or linear() - if isinstance(expression, exp.Anonymous): - func_name = expression.name.upper() - - if func_name in ("RRF", "LINEAR"): - self._build_hybrid_fusion_spec(expression, func_name, result) - continue - - # Regular ORDER BY field - field_name = expression.name if isinstance(expression, exp.Column) else None - direction = "DESC" if ordered.args.get("desc") else "ASC" - if field_name: - result.orderby_fields.append((field_name, direction)) - -def _build_hybrid_fusion_spec( - self, expression: exp.Anonymous, fusion_method: str, result: ParsedQuery -) -> None: - """Build HybridFusionSpec from rrf() or linear() in ORDER BY. - - Expected signatures: - - rrf(vscore, tscore [, constant => 60] [, window => 20]) - - linear(vscore, tscore [, alpha => 0.5] [, beta => 0.5] [, window => 20]) - """ - args = expression.expressions - if len(args) < 2: - raise ValueError( - f"{fusion_method.lower()}() requires at least 2 arguments: " - f"{fusion_method.lower()}(vector_alias, text_alias), got {len(args)}" - ) - - # Extract aliases (first two args) - vector_alias = args[0].name if isinstance(args[0], exp.Column) else None - text_alias = args[1].name if isinstance(args[1], exp.Column) else None - - if not vector_alias or not text_alias: - raise ValueError( - f"{fusion_method.lower()}() requires column aliases: " - f"{fusion_method.lower()}(vscore, tscore)" - ) - - # Find the corresponding vector_distance and fulltext in SELECT - vector_leg = self._find_vector_leg(result, vector_alias) - text_leg = self._find_text_leg(result, text_alias) - - if not vector_leg: - raise ValueError( - f"No vector_distance(...) AS {vector_alias} found in SELECT" - ) - if not text_leg: - raise ValueError( - f"No fulltext(...) AS {text_alias} found in SELECT" - ) - - # Extract fusion parameters from kwargs - params = self._extract_fusion_params(args[2:], fusion_method) - - # Build HybridFusionSpec - result.hybrid_fusion = HybridFusionSpec( - text_leg=text_leg, - vector_leg=vector_leg, - fusion_method=fusion_method, - **params - ) - - # Clear vector_search since hybrid_fusion takes precedence - result.vector_search = None - -def _find_vector_leg(self, result: ParsedQuery, alias: str) -> VectorRankingLeg | None: - """Find vector_distance(...) AS alias in SELECT.""" - if result.vector_search and result.vector_search.alias == alias: - return VectorRankingLeg( - field=result.vector_search.field, - alias=alias, - k=result.vector_search.k - ) - return None - -def _find_text_leg(self, result: ParsedQuery, alias: str) -> TextRankingLeg | None: - """Find fulltext(...) AS alias in SELECT. - - This requires detecting fulltext() in SELECT (not WHERE). - """ - # Look for a computed field or condition with FULLTEXT operator - # that has the matching alias - for cond in result.conditions: - if cond.operator == "FULLTEXT" and hasattr(cond, "alias") and cond.alias == alias: - return TextRankingLeg( - field=cond.field, - query=cond.value, - alias=alias, - scorer="BM25" # Default; could be configurable - ) - return None - -def _extract_fusion_params(self, kwarg_exprs: list, fusion_method: str) -> dict: - """Extract kwargs like constant => 60, window => 20.""" - params = {} - - for expr in kwarg_exprs: - if isinstance(expr, exp.EQ): - key = expr.this.name if isinstance(expr.this, exp.Column) else None - value = self._extract_literal_value(expr.expression) - - if key and value is not None: - params[key] = value - - return params -``` - -## 2. Analyzer Changes (analyzer.py) - -### New Data Class - -```python -@dataclass -class HybridFusionAnalysis: - """Analyzed hybrid fusion with validated field types.""" - spec: HybridFusionSpec - text_field_type: str # Confirmed TEXT from schema - vector_field_type: str # Confirmed VECTOR from schema - filters: list[Condition] # Per-leg filters (applied to both SEARCH and VSIM) -``` - -### Update AnalyzedQuery - -```python -@dataclass -class AnalyzedQuery: - # ... existing fields ... - hybrid_fusion: HybridFusionAnalysis | None = None -``` - -### Validation Logic - -```python -def analyze(self, parsed: ParsedQuery) -> AnalyzedQuery: - # ... existing logic ... - - if parsed.hybrid_fusion: - hybrid_fusion = self._analyze_hybrid_fusion(parsed) - else: - hybrid_fusion = None - - return AnalyzedQuery( - # ... existing fields ... - hybrid_fusion=hybrid_fusion - ) - -def _analyze_hybrid_fusion(self, parsed: ParsedQuery) -> HybridFusionAnalysis: - """Validate hybrid fusion spec against schema.""" - spec = parsed.hybrid_fusion - schema = self._schemas[parsed.index] - - # Validate text field is TEXT - text_field_type = schema.get(spec.text_leg.field) - if not text_field_type: - raise ValueError( - f"Text field '{spec.text_leg.field}' not found in index '{parsed.index}'" - ) - if text_field_type != "TEXT": - raise ValueError( - f"Text field '{spec.text_leg.field}' must be TEXT, got {text_field_type}" - ) - - # Validate vector field is VECTOR - vector_field_type = schema.get(spec.vector_leg.field) - if not vector_field_type: - raise ValueError( - f"Vector field '{spec.vector_leg.field}' not found in index '{parsed.index}'" - ) - if vector_field_type != "VECTOR": - raise ValueError( - f"Vector field '{spec.vector_leg.field}' must be VECTOR, got {vector_field_type}" - ) - - # Extract filters from conditions (exclude hybrid-related conditions) - filters = [ - c for c in parsed.conditions - if c.field != spec.text_leg.field or c.operator != "FULLTEXT" - ] - - return HybridFusionAnalysis( - spec=spec, - text_field_type=text_field_type, - vector_field_type=vector_field_type, - filters=filters - ) -``` - -## 3. Translator Changes (translator.py) - -### Command Dispatch - -```python -def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - """Build Redis command from analyzed query. - - Dispatch order: - 1. FT.HYBRID (hybrid fusion) - 2. FT.AGGREGATE (aggregations/groupby) - 3. FT.SEARCH (default) - """ - if analyzed.hybrid_fusion: - return self._build_ft_hybrid(analyzed) - elif analyzed.use_aggregate: - return self._build_ft_aggregate(analyzed) - else: - return self._build_ft_search(analyzed) -``` - -### FT.HYBRID Builder - -```python -def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - """Build FT.HYBRID command.""" - fusion = analyzed.hybrid_fusion.spec - parsed = analyzed.parsed - - # Build filter expression - filter_expr = self._build_filter_expr(analyzed.hybrid_fusion.filters) - - # Build SEARCH leg - text_query = f"@{fusion.text_leg.field}:({fusion.text_leg.query})" - if filter_expr: - text_query = f"({text_query}) {filter_expr}" - search_leg = f'SEARCH "{text_query}" YIELD_SCORE_AS {fusion.text_leg.alias}' - - # Build VSIM leg - k = fusion.vector_leg.k or fusion.window - filter_count = len(analyzed.hybrid_fusion.filters) - vsim_parts = [ - f'VSIM @{fusion.vector_leg.field} $vec', - ] - if filter_count > 0: - vsim_parts.append(f'FILTER {filter_count} "{filter_expr}"') - vsim_parts.append(f'KNN 2 K {k}') - vsim_parts.append(f'YIELD_SCORE_AS {fusion.vector_leg.alias}') - vsim_leg = ' '.join(vsim_parts) - - # Build COMBINE clause - if fusion.fusion_method == "RRF": - combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' - else: # LINEAR - combine = ( - f'COMBINE LINEAR 2 ' - f'ALPHA {fusion.linear_alpha} ' - f'BETA {fusion.linear_beta} ' - f'WINDOW {fusion.window}' - ) - - # Build LOAD clause - load_fields = parsed.fields if parsed.fields else ["*"] - load_clause = ["LOAD", str(len(load_fields))] + load_fields - - # Build LIMIT clause - offset = parsed.offset or 0 - limit = parsed.limit or 10 - limit_clause = ["LIMIT", str(offset), str(limit)] - - # Assemble command - cmd = [ - "FT.HYBRID", - parsed.index, - "2", # Number of legs - search_leg, - vsim_leg, - combine, - *load_clause, - *limit_clause, - "DIALECT", "2" - ] - - return TranslatedQuery( - command="FT.HYBRID", - args=cmd[1:], - index=parsed.index, - return_fields=parsed.fields - ) -``` - -## 4. Executor Changes (executor.py) - -### Version Gating - -```python -def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: - """Execute SQL query against Redis.""" - params = params or {} - - # Substitute and translate - sql = _substitute_params(sql, params) - translated = self._translator.translate(sql) - - # Version gate for FT.HYBRID - if translated.command == "FT.HYBRID": - self._check_hybrid_support() - - # Execute command - # ... existing logic ... - -def _check_hybrid_support(self) -> None: - """Check if Redis supports FT.HYBRID (8.4+).""" - info = self._client.info("server") - version = info.get("redis_version", "0.0.0") - - if not self._meets_version(version, "8.4.0"): - raise ValueError( - f"FT.HYBRID requires Redis 8.4 or later, found {version}. " - "Use filter-then-KNN syntax (vector_distance without fulltext in SELECT) instead." - ) - -def _meets_version(self, current: str, required: str) -> bool: - """Check if current version >= required version.""" - current_parts = [int(x) for x in current.split(".")[:3]] - required_parts = [int(x) for x in required.split(".")] - - for c, r in zip(current_parts, required_parts): - if c > r: - return True - if c < r: - return False - return True -``` - -## Summary - -This implementation sketch shows: - -1. **Parser:** Detect fusion functions in `ORDER BY`, build `HybridFusionSpec`. -2. **Analyzer:** Validate field types, separate filters. -3. **Translator:** Dispatch to `_build_ft_hybrid()` when `hybrid_fusion` is present. -4. **Executor:** Gate on Redis 8.4+ before executing `FT.HYBRID`. - -All changes are **additive** and **backward compatible** with existing filter-then-KNN queries. - - diff --git a/docs/proposals/ft-hybrid-primitive-design.md b/docs/proposals/ft-hybrid-primitive-design.md deleted file mode 100644 index 13cadc0..0000000 --- a/docs/proposals/ft-hybrid-primitive-design.md +++ /dev/null @@ -1,295 +0,0 @@ -# FT.HYBRID Primitive Design - Recommendation - -**Status:** Design recommendation for RAAE-1322 -**Supersedes:** Design discussion in `ft-hybrid.md` -**Date:** 2026-06-23 - -## Executive Summary - -After reviewing the `ft-hybrid.md` proposal and the sql-redis codebase, I recommend **Design C with composable ranking functions**, but with a significant improvement: introduce a **new primitive abstraction** that better encapsulates hybrid fusion and aligns with sql-redis's design philosophy. - -## Core Issue with Current Proposal - -The three designs (A, B, C) in `ft-hybrid.md` all treat hybrid fusion as a **syntax mapping problem** rather than a **data structure problem**. The proposals focus on where to put `rrf()` or `hybrid()` in SQL, but don't address the deeper design question: - -**How should hybrid fusion be represented in sql-redis's internal data model?** - -Currently, sql-redis has: -- `VectorSearchSpec` - represents a single vector leg -- `Condition` - represents text/filter predicates -- `ScoringSpec` - represents relevance scoring (BM25, etc.) - -None of these primitives can cleanly express **two independent ranking signals fused together**. Adding `FT.HYBRID` support by bolting detection logic onto existing specs will create technical debt. - -## Recommended Primitive: `HybridFusionSpec` - -### Data Structure - -```python -@dataclass -class TextRankingLeg: - """Text ranking leg for hybrid fusion.""" - field: str # TEXT field to search - query: str | None # Search query text - alias: str # Score alias (e.g., "tscore") - scorer: str = "BM25" # BM25, TFIDF, DISMAX - -@dataclass -class VectorRankingLeg: - """Vector ranking leg for hybrid fusion.""" - field: str # VECTOR field to search - alias: str # Score alias (e.g., "vscore") - k: int | None = None # KNN candidate pool size - -@dataclass -class HybridFusionSpec: - """Specification for FT.HYBRID fusion.""" - text_leg: TextRankingLeg - vector_leg: VectorRankingLeg - fusion_method: str # "RRF" or "LINEAR" - - # RRF parameters - rrf_constant: int = 60 - - # LINEAR parameters - linear_alpha: float = 0.5 - linear_beta: float = 0.5 - - # Common parameters - window: int = 20 # Fusion window size -``` - -This primitive makes hybrid fusion **first-class** instead of inferring it from the presence of multiple specs. - -### Integration into ParsedQuery - -```python -@dataclass -class ParsedQuery: - # ... existing fields ... - vector_search: VectorSearchSpec | None = None # For filter-then-KNN - hybrid_fusion: HybridFusionSpec | None = None # For FT.HYBRID fusion -``` - -**Mutual exclusion:** A query has EITHER `vector_search` (filter-then-KNN) OR `hybrid_fusion` (server-side fusion), never both. - -## Why This Is Better - -### 1. **Explicit over implicit** -Design C detects `fulltext(...) AS tscore` + `vector_distance(...) AS vscore` + `rrf(tscore, vscore)` and infers hybrid mode. The new primitive makes the intent **explicit** in the data model. - -### 2. **Single source of truth** -All hybrid parameters live in `HybridFusionSpec`. No need to reconcile state scattered across `VectorSearchSpec`, `Condition`, and `ORDER BY`. - -### 3. **Clear validation** -The analyzer can enforce: -- `text_leg.field` is TYPE `TEXT` -- `vector_leg.field` is TYPE `VECTOR` -- Fusion method matches parameters (RRF → constant, LINEAR → alpha/beta) -- Filters are compatible with both legs - -### 4. **Command-path clarity** -```python -if analyzed.hybrid_fusion: - return build_ft_hybrid_command(analyzed) -elif analyzed.vector_search: - return build_ft_search_knn_command(analyzed) -else: - return build_ft_search_or_aggregate_command(analyzed) -``` - -No branching on combinations of flags — the data structure dictates the path. - -### 5. **Extensibility** -When Redis adds more fusion methods (e.g., `DBSF`, `CombSUM`), you add a new `fusion_method` value. When `FT.HYBRID` gains a third leg (e.g., sparse vector), you add `SparseVectorRankingLeg`. - -## Recommended SQL Syntax (Design C+) - -Keep Design C's composability, but tighten the semantics around the new primitive: - -```sql -SELECT page_text, file_id, - vector_distance(embedding, :vec) AS vscore, - fulltext(page_text, 'quarterly earnings') AS tscore -FROM "KM_abc123" -WHERE ticker = 'MSFT' -ORDER BY rrf(vscore, tscore, constant => 60) DESC -LIMIT 10; -``` - -### Detection Logic (Parser) - -The parser builds `HybridFusionSpec` when **all** of these are true: - -1. `SELECT` contains `vector_distance(vec_field, :param) AS ` -2. `SELECT` contains `fulltext(text_field, 'query') AS ` -3. `ORDER BY` contains `rrf(, , ...)` or `linear(, , ...)` - -If only (1) is present → `VectorSearchSpec` (filter-then-KNN, existing behavior). -If (1) + (2) but no fusion in `ORDER BY` → `ValueError("ambiguous: use rrf() or linear() to specify fusion")`. - -This forces users to be explicit about fusion vs. returning two separate scores. - -### Why Not Design B (`hybrid()` predicate)? - -Design B is more compact: - -```sql -WHERE hybrid(page_text, 'quarterly earnings', embedding, :vec) -``` - -But it has problems: - -1. **Breaks the separation of concerns.** `WHERE` is for filtering; ranking belongs in `SELECT`/`ORDER BY`. -2. **No place for score aliases.** Users can't reference `tscore`/`vscore` separately for debugging. -3. **Harder to extend.** Adding scorer config (BM25 vs TFIDF), KNN params, or window size requires kwargs in `WHERE`, which is unnatural. - -Design C keeps ranking in `SELECT`/`ORDER BY` where it belongs. - -## Analyzer Changes - -```python -@dataclass -class AnalyzedQuery: - # ... existing fields ... - hybrid_fusion: HybridFusionAnalysis | None = None - -@dataclass -class HybridFusionAnalysis: - """Analyzed hybrid fusion with resolved field types.""" - spec: HybridFusionSpec - text_field_type: str # Confirmed TEXT from schema - vector_field_type: str # Confirmed VECTOR from schema - filters: list[Condition] # Per-leg filters (applied to both SEARCH and VSIM) -``` - -The analyzer: -1. Validates `text_leg.field` exists and is `TEXT` -2. Validates `vector_leg.field` exists and is `VECTOR` -3. Splits `WHERE` conditions into per-leg filters (no special text-leg-only logic — apply to both for consistency) -4. Computes KNN `K` from `window` if not explicitly set - -## Translator Changes - -Add a new command-path branch **before** the search-vs-aggregate decision: - -```python -def _build_command(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - if analyzed.hybrid_fusion: - return self._build_ft_hybrid(analyzed) - elif analyzed.use_aggregate: - return self._build_ft_aggregate(analyzed) - else: - return self._build_ft_search(analyzed) - -def _build_ft_hybrid(self, analyzed: AnalyzedQuery) -> TranslatedQuery: - """Build FT.HYBRID command from HybridFusionAnalysis.""" - fusion = analyzed.hybrid_fusion.spec - - # Build SEARCH leg - search_query = build_text_query(fusion.text_leg.field, fusion.text_leg.query) - search_filter = build_filter_expr(analyzed.hybrid_fusion.filters) - search_leg = f'SEARCH "{search_query} {search_filter}" YIELD_SCORE_AS {fusion.text_leg.alias}' - - # Build VSIM leg - vsim_filter_count = len(analyzed.hybrid_fusion.filters) - k = fusion.vector_leg.k or fusion.window - vsim_leg = ( - f'VSIM @{fusion.vector_leg.field} $vec ' - f'FILTER {vsim_filter_count} "{search_filter}" ' - f'KNN 2 K {k} YIELD_SCORE_AS {fusion.vector_leg.alias}' - ) - - # Build COMBINE clause - if fusion.fusion_method == "RRF": - combine = f'COMBINE RRF 2 CONSTANT {fusion.rrf_constant} WINDOW {fusion.window}' - else: # LINEAR - combine = ( - f'COMBINE LINEAR 2 ALPHA {fusion.linear_alpha} ' - f'BETA {fusion.linear_beta} WINDOW {fusion.window}' - ) - - # Assemble command - cmd = [ - "FT.HYBRID", analyzed.parsed.index, "2", - search_leg, - vsim_leg, - combine, - "LOAD", str(len(analyzed.parsed.fields)), *analyzed.parsed.fields, - "LIMIT", str(analyzed.parsed.offset or 0), str(analyzed.parsed.limit or 10), - "DIALECT", "2" - ] - - return TranslatedQuery(command="FT.HYBRID", args=cmd[1:], ...) -``` - -This is **much cleaner** than trying to reuse `_build_ft_search` and injecting hybrid logic via conditionals. - -## Version Gating - -Add a Redis version check in the executor: - -```python -def execute(self, sql: str, *, params: dict | None = None) -> QueryResult: - translated = self._translator.translate(sql) - - if translated.command == "FT.HYBRID": - # Check Redis version >= 8.4 - info = self._client.info("server") - version = info.get("redis_version", "0.0.0") - if not _meets_version(version, "8.4.0"): - raise ValueError( - f"FT.HYBRID requires Redis 8.4+, found {version}. " - "Use filter-then-KNN syntax instead." - ) - - # ... execute command ... -``` - -Fail **fast** with a clear message rather than letting Redis return a cryptic error. - -## Migration Path - -This design is **backward compatible**: - -- Existing `vector_distance()` queries without `fulltext()` → still build `VectorSearchSpec` → still emit `FT.SEARCH` with KNN. -- Existing `fulltext()` queries in `WHERE` → still build `Condition` → still work as filters. - -No existing queries break. Hybrid fusion is purely **additive**. - -## Comparison to Original Designs - -| Aspect | Design A | Design B | Design C (original) | **Design C+ (new primitive)** | -|--------|----------|----------|---------------------|-------------------------------| -| Syntax clarity | ❌ Ambiguous | ✅ Compact | ✅ Composable | ✅ Composable + explicit | -| Data model | ❌ Inferred | ⚠️ New WHERE func | ⚠️ Scattered state | ✅ Dedicated primitive | -| Extensibility | ❌ Breaking | ⚠️ Kwargs hell | ⚠️ Inference fragile | ✅ Clean extension points | -| Command path | ❌ Heuristics | ⚠️ Detection | ⚠️ Detection | ✅ Type-driven dispatch | -| Score surfacing | ❌ Unclear | ❌ No aliases | ✅ Explicit aliases | ✅ Explicit aliases | -| Backward compat | ❌ Breaking | ✅ New func | ✅ Additive | ✅ Additive | - -**Verdict:** Design C+ (composable syntax + dedicated primitive) is the cleanest path forward. - -## Open Questions for Confirmation - -1. **Fusion defaults:** Should `ORDER BY DESC` on a score without `rrf()/linear()` default to RRF, or require explicit fusion? **Recommendation:** Require explicit fusion (fail fast on ambiguity). - -2. **Score surfacing:** Should the fused score be auto-projected as a column (e.g., `__fused_score`)? **Recommendation:** Let users explicitly `SELECT` it via an alias if needed. - -3. **K vs WINDOW:** Should KNN `K` default to `window`, or be independently configurable? **Recommendation:** Default `K = window` for v1; add `k =>` kwarg to `rrf()/linear()` later if needed. - -4. **Filter distribution:** Apply `WHERE` filters to both legs, or SEARCH-only? **Recommendation:** Both legs (consistency > simplicity). - -## Next Steps - -1. Implement `HybridFusionSpec`, `TextRankingLeg`, `VectorRankingLeg` dataclasses in `parser.py`. -2. Add detection logic in `SQLParser._process_order_by()` to build `HybridFusionSpec` when fusion functions are present. -3. Add `HybridFusionAnalysis` to analyzer and validation logic. -4. Implement `Translator._build_ft_hybrid()` command builder. -5. Add version gating in executor. -6. Update docs to relabel "hybrid search" → "filtered KNN" and add "Hybrid Fusion (FT.HYBRID)" guide. -7. Bump test Redis image to 8.4+ and add integration tests. - -**Estimated effort:** 3-4 days for implementation + tests + docs (assuming no blockers on Redis 8.4 availability). - - diff --git a/docs/proposals/ft-hybrid.md b/docs/proposals/ft-hybrid.md deleted file mode 100644 index 2de81fc..0000000 --- a/docs/proposals/ft-hybrid.md +++ /dev/null @@ -1,295 +0,0 @@ -# Spec: `FT.HYBRID` support in sql-redis - -**Jira:** [RAAE-1322](https://redislabs.atlassian.net/browse/RAAE-1322) · -**Status:** Draft · -**Requires:** Redis 8.4+ and redis-py >= 7.1.0 (RediSearch `FT.HYBRID`) - -# Goal - -Add a `hybrid_vector_search(...)` SELECT function that translates a SQL `SELECT` into a native -`FT.HYBRID` command, so a text query and a vector query are **fused server-side** (RRF or -LINEAR) into a single ranking. This is distinct from today's pre-filter hybrid search, -where text is only a hard prefilter and the ranking comes from the vector leg alone. sql-redis exists -to give a familiar SQL surface over a Redis query; `hybrid_vector_search()` makes the new -server-side fusion query just as easy to express. - -Hard constraint: the syntax must read as a natural extension of the vector syntax we have -today (`cosine_distance(field, :vec)` plus `:param` substitution), and its options must -mirror what `FT.HYBRID` exposes so the two RedisVL surfaces (native `HybridQuery` and -`SQLQuery`) stay coherent. - -> Terminology note. The README/docs currently label filter-then-KNN as -> "Hybrid search (filters + vector)". That is not `FT.HYBRID`. This work uses: -> -> - **pre-filter hybrid search**: existing `WHERE ... cosine_distance(...)` to -> `(prefilter)=>[KNN ...]`. One ranking signal (vector); text/tags are hard filters. -> - **hybrid fusion** (this spec): new `hybrid_vector_search(...)` to `FT.HYBRID`. Two -> independently ranked legs (text + vector) fused by RRF/LINEAR. -> -> Part of this work is relabeling the existing docs to "pre-filter hybrid search" so the -> two are not conflated. - -# Flow - -```mermaid -flowchart LR - SQL["SELECT ..., hybrid_vector_search(vec_leg, text_leg, combine) AS score\nFROM idx WHERE region='us-central' LIMIT 10"] --> P[SQLParser] - P -->|"HybridSearchSpec\n(SEARCH leg + VSIM leg + COMBINE)"| A[Analyzer] - A -->|"resolve TEXT + VECTOR field types\nWHERE to per-leg FILTER"| QB[QueryBuilder] - QB --> T[Translator] - T -->|"emit FT.HYBRID\n(new third command path)"| EX[Executor] - EX -->|"two-stage param sub\n(vector bytes injected via PARAMS)"| R[(Redis 8.4+)] - R -->|fused rows + scores| EX --> Res[QueryResult] -``` - -# Syntax - -`hybrid_vector_search(...)` is a SELECT-clause function that composes the two ranking functions -the package already exposes: the vector leg reuses `cosine_distance(field, :vec)` and the -text leg reuses `fulltext(field, 'query')`. A third argument selects the fusion method. - -```sql -SELECT user, job, job_description, - hybrid_vector_search( - cosine_distance(job_embedding, :vec), -- VSIM leg: vector field + param - fulltext(job_description, 'use base principles to solve problems'), -- SEARCH leg: text field + query - rrf() -- COMBINE: fusion method (default RRF) - ) AS hybrid_score -FROM user_simple -WHERE region = 'us-central' -- FILTER, applied to both legs -ORDER BY hybrid_score DESC -LIMIT 10; -``` - -Why this aligns: - -- `cosine_distance(job_embedding, :vec)` is the exact vector function used today; here it - is the `VSIM` leg instead of a standalone KNN. -- `fulltext(job_description, 'query')` is the exact text function used today; as an - argument to `hybrid_vector_search` it becomes the `SEARCH` leg. -- `WHERE` maps to the per-leg `FILTER` (applied to both legs so the candidate sets agree). -- `SELECT` columns map to `LOAD`; `GROUP BY` maps to `GROUPBY`; `LIMIT` maps to `LIMIT`. -- `AS hybrid_score` surfaces the combined `YIELD_SCORE_AS` as a column. - -## Full knobs - -Defaults mirror RedisVL's native `HybridQuery` so the two surfaces match. - -**Fusion method (third argument)** - -| SQL | FT.HYBRID | Notes | -|---|---|---| -| `rrf()` | `COMBINE RRF` (server default) | omit the third arg for the same effect | -| `rrf(constant => 60, window => 20)` | `COMBINE RRF 4 CONSTANT 60 WINDOW 20` | defaults: constant 60, window 20 | -| `linear(alpha => 0.3)` | `COMBINE LINEAR ... ALPHA 0.3 BETA 0.7` | v1 exposes `alpha` only; `beta = 1 - alpha` (matches native `HybridQuery`) | -| `linear(alpha => 0.3, window => 20)` | `COMBINE LINEAR ... ALPHA 0.3 ... WINDOW 20` | | - -**Vector leg (`cosine_distance` / `vector_distance`)** - -| SQL | FT.HYBRID | -|---|---| -| `cosine_distance(job_embedding, :vec)` | `VSIM @job_embedding $vec KNN 2 K ` (K defaults to `LIMIT`, else 10) | -| `vector_distance(job_embedding, :vec, ef_runtime => 10)` | `... KNN ... EF_RUNTIME 10` (knob form; see kwarg note) | -| `vector_range(job_embedding, :vec, radius => 0.2, epsilon => 0.01)` | `VSIM @job_embedding $vec RANGE ... RADIUS 0.2 EPSILON 0.01` | - -**Text leg (`fulltext`)** - -| SQL | FT.HYBRID | -|---|---| -| `fulltext(job_description, 'principles')` | `SEARCH "@job_description:(principles)"` | -| `fulltext(job_description, 'principles', scorer => 'BM25STD')` | `SEARCH "..." SCORER BM25STD` (default `BM25STD`) | - -**Surfacing per-leg scores (optional)** - -`AS hybrid_score` yields the combined score. To also return the individual leg scores, add -kwargs to the legs: `cosine_distance(..., yield_score_as => 'vsim')` and -`fulltext(..., yield_score_as => 'tscore')`, mapping to each leg's `YIELD_SCORE_AS`. - -## SQL to `FT.HYBRID` mapping (worked example) - -Verified against Redis 8.4 (`redis:8.4`). Note three command-shape rules -confirmed empirically: the VSIM method clause (`KNN`/`RANGE`) must come **before** -`FILTER`; `LOAD` fields require an `@` prefix; and `FT.HYBRID` **rejects** an -explicit `DIALECT` argument (it uses the server's configured default). - -``` -FT.HYBRID user_simple - SEARCH "(@job_description:(use base principles to solve problems)) (@region:{us\-central})" SCORER BM25STD - VSIM @job_embedding $vector KNN 2 K 10 FILTER 1 "@region:{us\-central}" - COMBINE RRF 6 CONSTANT 60 WINDOW 20 YIELD_SCORE_AS hybrid_score - LOAD 3 @user @job @job_description - LIMIT 0 10 - PARAMS 2 vector -``` - -The reply is a flat map (not the FT.AGGREGATE array shape): -`[total_results, N, results, [[field, val, ...], ...], warnings, [...], execution_time, ...]`. - -| SQL element | FT.HYBRID target | -|---|---| -| `hybrid_vector_search(...)` in SELECT | triggers the `FT.HYBRID` command path | -| `cosine_distance(vec_field, :vec)` (nested) | `VSIM @vec_field $vector KNN ... K ...` | -| `fulltext(text_field, 'q')` (nested) | `SEARCH "@text_field:(q)" [SCORER ...]` | -| `WHERE ` | folded into the `SEARCH` query string **and** `VSIM ... FILTER n ""` (after the method clause) | -| `rrf(...)` / `linear(...)` | `COMBINE RRF \| LINEAR ...` | -| `SELECT col1, col2, ...` | `LOAD n @col1 @col2 ...` | -| `GROUP BY ...` + aggregations | `GROUPBY n @prop REDUCE ...` | -| `ORDER BY hybrid_score DESC` | `SORTBY` / default combined-score sort | -| `LIMIT n` / `LIMIT m, n` | `LIMIT m n` (also sets KNN `K` when the vector leg is KNN) | -| `:vec` param (bytes) | `PARAMS 2 vector ` via stage-2 substitution | -| (note) | no `DIALECT` argument (FT.HYBRID rejects it) | - -## End-user surface (RedisVL `SQLQuery`) - -Most users reach sql-redis through RedisVL's `SQLQuery`. The query author writes the same -SQL and passes the vector blob as a param; `index.query(...)` runs it through the -sql-redis executor and returns rows. No RedisVL execution code changes once sql-redis emits -and parses `FT.HYBRID` (the dispatch path `index.query` to `_sql_query` to -`executor.execute` is unchanged). Companion spec: -`applied-ai/redis-vl-python/docs/proposals/sqlquery-ft-hybrid.md`. - -Default RRF fusion: - -```python -from redisvl.query import SQLQuery -from redisvl.index import SearchIndex - -index = SearchIndex.from_dict(schema, redis_url="redis://localhost:6379") -vec = hf.embed("use base principles to solve problems", as_buffer=True) - -sql_query = SQLQuery( - """ - SELECT user, job, job_description, - hybrid_vector_search( - cosine_distance(job_embedding, :vec), - fulltext(job_description, 'use base principles to solve problems'), - rrf() - ) AS hybrid_score - FROM user_simple - WHERE region = 'us-central' - ORDER BY hybrid_score DESC - LIMIT 10 - """, - params={"vec": vec}, -) - -# Inspect the generated command before running it -print(sql_query.redis_query_string(redis_url="redis://localhost:6379")) -# FT.HYBRID user_simple SEARCH "@job_description:(...) (@region:{us\-central})" SCORER BM25STD -# VSIM @job_embedding $vec FILTER 1 "@region:{us\-central}" KNN 2 K 10 -# COMBINE RRF 4 CONSTANT 60 WINDOW 20 YIELD_SCORE_AS hybrid_score -# LOAD 3 user job job_description LIMIT 0 10 PARAMS 2 vec DIALECT 2 - -results = index.query(sql_query) -``` - -LINEAR fusion with a custom text scorer and an explicit KNN tuning knob: - -```python -sql_query = SQLQuery( - """ - SELECT user, job, job_description, - hybrid_vector_search( - cosine_distance(job_embedding, :vec, ef_runtime => 20), - fulltext(job_description, 'principles', scorer => 'BM25STD'), - linear(alpha => 0.3) - ) AS hybrid_score - FROM user_simple - ORDER BY hybrid_score DESC - LIMIT 5 - """, - params={"vec": vec}, -) -results = index.query(sql_query) -``` - -Async usage is identical via `AsyncSearchIndex.query(sql_query)`. - -# Implementation plan (by layer) - -Mirrors the existing `cosine_distance` / `vector_distance` path. There is no function -registry, so dispatch is added to the same sites that already handle vector and text -functions ([`sql_redis/parser.py`](../../sql_redis/parser.py) `_process_select_expression` -and `_add_function_condition`). - -1. **Parser** ([`sql_redis/parser.py`](../../sql_redis/parser.py)) - - Add a `HybridSearchSpec` dataclass: text field/query/scorer, vector field/param, - vector method (KNN/RANGE) and its params, fusion method and params, per-leg and - combined score aliases. - - Detect `hybrid_vector_search(...)` in the SELECT projection. Parse its three arguments by - reusing the existing `cosine_distance` / `vector_distance` and `fulltext` extraction - so the legs behave identically to their standalone forms. -2. **Analyzer** ([`sql_redis/analyzer.py`](../../sql_redis/analyzer.py)) - - Add `hybrid_search: HybridSearchAnalysis | None`. Resolve that the text field is - `TEXT` and the vector field is `VECTOR`; reject mismatches (mirror the existing - TEXT-operator guard). Split `WHERE` into the shared leg filter; resolve `K` from `LIMIT`. -3. **QueryBuilder** ([`sql_redis/query_builder.py`](../../sql_redis/query_builder.py)) - - Add `build_hybrid_command(...)` to emit the `SEARCH ... VSIM ... COMBINE ...` argv. - Reuse `build_text_condition` for the `SEARCH` query string and the existing filter - builders for the `FILTER` expression. -4. **Translator** ([`sql_redis/translator.py`](../../sql_redis/translator.py)) - - `FT.HYBRID` is a **new third command path** alongside `FT.SEARCH` / `FT.AGGREGATE`. - Branch on `analyzed.hybrid_search` before the search-vs-aggregate decision. Map - `LOAD`, `LIMIT`, `GROUPBY`, `PARAMS`, and `DIALECT 2`. -5. **Executor** ([`sql_redis/executor.py`](../../sql_redis/executor.py)) - - Reuse the two-stage param substitution (scalars inlined in stage 1; vector bytes kept - as `$vec` and injected via `PARAMS` in stage 2, exactly as vectors work today). - - Run the command and parse the hybrid reply (its shape differs from search/aggregate). - Prefer redis-py's `hybrid_search` result parsing (available in redis-py >= 7.1.0) - rather than hand-rolling reply parsing. - - Add a version guard: probe `FT.HYBRID` availability (or Redis >= 8.4) and raise a - clear `ValueError` ("FT.HYBRID requires Redis 8.4+") instead of a raw Redis error. -6. **Docs** ([`docs/user_guide/how_to_guides/vector-search.md`](../user_guide/how_to_guides/vector-search.md)) - - Relabel the existing "hybrid" section to "pre-filter hybrid search" and add a "Hybrid fusion - (FT.HYBRID)" section. Update the README capability list. - -# Testing - -Follow the layered convention (100% coverage enforced, no `# pragma: no cover`): - -- **Unit:** parser (spec extraction from the nested functions), analyzer (field-type - resolution, filter split), query_builder (exact argv), translator (full command plus - `DIALECT 2`). -- **Integration** ([`tests/test_sql_queries.py`](../../tests/test_sql_queries.py), - testcontainers): a `TestHybridFusion` class. The conftest Redis image is currently - **8.0.2**; `FT.HYBRID` needs **8.4+**, so the image must be bumped (and tests skipped on - unsupported versions). -- **Parity:** SQL result equals a hand-written `FT.HYBRID` result (mirrors - [`tests/test_redis_queries.py`](../../tests/test_redis_queries.py)). - -# Things to consider - -- **`K` vs `WINDOW` vs `LIMIT`.** `K` = vector neighbors feeding fusion; `WINDOW` = how - many per-leg results RRF/LINEAR consider; `LIMIT` = final rows. RedisVL's `HybridQuery` - collapses `num_results` into both KNN `K` and the final limit. Proposal: match it, - `LIMIT` sets KNN `K` and the final cut, `WINDOW` defaults to 20 unless set in the combine - function. Is exposing `WINDOW` and explicit `K` separately too much surface for v1? -- **LINEAR alpha/beta.** v1 exposes `alpha` only and derives `beta = 1 - alpha`, matching - native `HybridQuery`. `FT.HYBRID` accepts an explicit `beta`, but exposing it would let a - SQL query produce a command the object API cannot; defer it. (Decided.) -- **Filter on both legs vs SEARCH only.** Applying `WHERE` to both `SEARCH` and `VSIM` - keeps candidate sets consistent and matches RedisVL (`filter_expression` is passed to - both). Recommend both legs. -- **Default fusion.** Omitting the third argument yields the server default (RRF). Require - the explicit `rrf()`/`linear()` to set knobs; do not silently default `ORDER BY` into - fusion. -- **kwarg parsing (validated).** The `name => value` form parses through sqlglot as a - `Kwarg` inside anonymous functions (`fulltext`, `rrf`, `linear`, `vector_distance`), so - the fusion/scorer knobs parse cleanly. One constraint: `cosine_distance` is a sqlglot - built-in capped at 2 arguments, so a 3-arg `cosine_distance(field, :vec, ef_runtime => n)` - is a `ParseError`. Vector-leg tuning knobs (`ef_runtime`, RANGE `radius`/`epsilon`) - must therefore ride on the anonymous `vector_distance(...)` / `vector_range(...)` forms, - not `cosine_distance(...)`. Plain `cosine_distance(field, :vec)` (2 args) remains valid. -- **Score comparability.** RRF/LINEAR scores are not comparable to a raw - `cosine_distance`, so `ORDER BY hybrid_score` is the natural sort and per-leg scores are - opt-in via `yield_score_as`. -- **redis-py floor.** `FT.HYBRID` execution/parsing needs redis-py >= 7.1.0; gate and - document alongside the Redis 8.4 floor. - -# Decisions to confirm - -1. **Fusion config surface for v1.** Confirmed: RRF constant/window, LINEAR `alpha` only - (`beta = 1 - alpha`), KNN ef_runtime, RANGE radius/epsilon. -2. **Execution/parsing.** Use redis-py's `hybrid_search` result parsing inside the - executor (recommended), vs. hand-rolled reply parsing. See the companion RedisVL - packaging spec for how `SQLQuery` surfaces this end to end.