diff --git a/docs/issues/memory-recall-hot-path-keyword-recall/plan.md b/docs/issues/memory-recall-hot-path-keyword-recall/plan.md new file mode 100644 index 000000000..0187ccc8a --- /dev/null +++ b/docs/issues/memory-recall-hot-path-keyword-recall/plan.md @@ -0,0 +1,30 @@ +# Plan + +## Implementation + +- Add small pure helpers in the memory presenter for recall keywordization and soft timeout handling. +- Extend the agent-memory repository search contract with a keyword match mode, defaulting to current all-term behavior. +- Add a corpus-aware recall keyword stats query to the repository contract. It counts active recallable rows per candidate term and excludes persona, working, archived, conflicted, and superseded rows. +- Implement corpus-aware term stats as one aggregate SQL statement per recall: `COUNT(*)` plus `SUM(CASE WHEN content LIKE ? ESCAPE '\\' THEN 1 ELSE 0 END)` for bounded candidate terms. +- Guard warm query embedding with one tracked in-flight entry per `agentId + providerId + modelId`. Skip vector recall while a fresh entry is active; replace stale entries after 30 seconds and let late promises clear only their own map entry. +- Add `recordAccessBatch` to the repository contract, SQLite implementation, and fake repository. +- Route public agent-facing recall and injection through dynamically selected keywordized OR matching; leave management search and internal neighbor lookups on precise all-term matching. +- Keep keyword candidate extraction pure and stopword-free. Extract ASCII/code and CJK candidates into one position-ordered pool before applying the candidate cap, then rank terms by low corpus hit count, term length, and original position; emit the selected terms back in original query order. +- Add settings-panel hints for missing embedding/extraction model configuration. +- Archive memory SDD #19 and #20 with explicit rejection notes and update the memory README. + +## Compatibility + +- Existing callers of `repository.search(agentId, query, limit)` continue to work because search mode defaults to all-term matching. +- Existing public contracts and persisted memory rows are unchanged. +- Late query embedding promises are intentionally not aborted; their result is ignored after timeout. The in-flight guard is a tracked rate gate, not a provider-level hard concurrency cap, so stale replacement may leave old provider calls running until they settle. +- If no extracted term hits the active memory corpus, the keyword branch returns no rows; vector recall still uses the original query when available. + +## Test Strategy + +- Main presenter tests cover timeout degradation, corpus-aware English and CJK recall, management search precision, no-hit keyword behavior, and batch access updates. +- SQLite table tests cover OR search mode, aggregate corpus term stats filtering, and batch access persistence. +- Presenter tests cover query-embedding in-flight suppression and stale replacement. +- Pure recall keyword tests cover extraction/ranking edge cases without presenter setup, including mixed CJK/ASCII candidate caps. +- Renderer tests cover new missing-model hints. +- Final validation runs through mise. diff --git a/docs/issues/memory-recall-hot-path-keyword-recall/spec.md b/docs/issues/memory-recall-hot-path-keyword-recall/spec.md new file mode 100644 index 000000000..1cc0d927f --- /dev/null +++ b/docs/issues/memory-recall-hot-path-keyword-recall/spec.md @@ -0,0 +1,46 @@ +# Memory Recall Hot Path and Keyword Recall + +## User Need + +Memory recall must not add unbounded first-token latency, and FTS-only recall must remain useful for normal chat messages. Today a warm vector recall waits for the query embedding in the pre-stream memory injection path, and keyword recall uses the full user message with all terms required to match. + +## Goal + +- Add a bounded soft timeout to the hot-path query embedding call so a slow provider degrades the current turn to FTS-only recall. +- Use a recall-specific keyword query for agent-facing recall and memory injection so long English or CJK messages can still match relevant memories without requiring the whole message to match. +- Keep management search precise and unchanged by default. +- Archive rejected Wave 2 recall experiments #19 and #20 while preserving their decision history. + +## Acceptance Criteria + +- Warm vector recall returns from FTS-only when query embedding exceeds the soft timeout, without clearing vector readiness, starting reindex, or blocking on the late embedding result. +- Non-timeout vector failures keep the existing degrade-to-FTS behavior. +- Agent-facing recall and injection keyword search use a bounded keywordized query and OR-style matching. +- Recall keyword selection is corpus-aware: query terms are extracted without a static stopword list, terms with no active corpus hits are dropped, and high-frequency terms are filtered when better lower-frequency terms exist. +- Mixed ASCII/code/CJK query term extraction preserves original query order before applying the candidate cap, so earlier CJK terms cannot be starved by later ASCII/code tokens. +- Corpus-aware term stats are collected with one bounded aggregate query per recall, not one query per candidate term. +- At most one tracked warm query-embedding entry per agent/model is active; later turns skip vector recall while that entry is fresh, and stale entries older than 30 seconds can be replaced without aborting old provider requests. +- `memory.search` management search keeps the existing all-term semantics. +- Access counter updates happen in one repository call for a recalled result set. +- Settings explain the degraded FTS-only state when memory is enabled without an embedding model and warn when extraction falls back to the chat model. +- #21 remains unchanged. + +## Constraints + +- No DB schema migration. +- No public route, IPC, or tool schema changes. +- No new external dependencies. +- No query embedding abort requirement; late provider requests are ignored when the caller has already degraded. +- Validation commands must run through `mise exec -- pnpm ...`. + +## Non-Goals + +- Do not implement #19 query expansion or #20 reranker. +- Do not implement #21 policy port or deduplicate `deriveRecall`. +- Do not maintain a static recall stopword list. +- Do not add DuckDB vacuum/orphan vector cleanup. +- Do not auto-select models or change memory defaults. + +## Open Questions + +None. diff --git a/docs/issues/memory-recall-hot-path-keyword-recall/tasks.md b/docs/issues/memory-recall-hot-path-keyword-recall/tasks.md new file mode 100644 index 000000000..596aef6d5 --- /dev/null +++ b/docs/issues/memory-recall-hot-path-keyword-recall/tasks.md @@ -0,0 +1,19 @@ +# Tasks + +- [x] Archive #19/#20 and update memory README. +- [x] Add repository search mode and batch access update. +- [x] Add recall keywordizer and query embedding soft timeout. +- [x] Wire agent-facing recall/injection to keywordized OR matching while keeping management search precise. +- [x] Add settings missing-model hints. +- [x] Add/update focused tests. +- [x] Run focused and final validation with `mise exec -- pnpm ...`. +- [x] Replace the static recall stopword list with corpus-aware term selection. +- [x] Add repository term stats support and tests. +- [x] Re-run focused and final validation with `mise exec -- pnpm ...`. +- [x] Change term stats to one aggregate SQL query. +- [x] Add query embedding in-flight suppression with stale replacement. +- [x] Add pure recall keyword tests and in-flight presenter tests. +- [x] Re-run focused, native-sqlite, renderer, and final validation with `mise exec -- pnpm ...`. +- [x] Fix mixed ASCII/code/CJK keyword candidate cap ordering. +- [x] Clarify query embedding in-flight guard semantics. +- [x] Add recall keyword edge test and rerun focused validation. diff --git a/src/main/presenter/memoryPresenter/index.ts b/src/main/presenter/memoryPresenter/index.ts index 6cc5d0e0b..af27bd430 100644 --- a/src/main/presenter/memoryPresenter/index.ts +++ b/src/main/presenter/memoryPresenter/index.ts @@ -38,6 +38,11 @@ import { fuse, resolveRetrieval } from './scoring' +import { + buildRecallKeywordQuery, + extractRecallKeywordCandidates, + selectRecallKeywordTerms +} from './recallKeyword' import { deriveLifecycle, type DeriveLifecycleOptions } from './lifecycle' import { ARCHIVE_AGE_MS, ARCHIVE_DECAY_THRESHOLD } from './lifecycleConstants' import { ADD_DECISION, buildDecisionPrompt, parseDecision, type MemoryDecision } from './decision' @@ -128,6 +133,31 @@ const MEMORY_HEALTH_TOP_ACCESSED_LIMIT = 5 const MEMORY_HEALTH_AUDIT_SCAN_LIMIT = MEMORY_HEALTH_DEFAULT_AUDIT_SCAN_LIMIT const MEMORY_HEALTH_RECENT_FAILURES_LIMIT = 5 const MEMORY_CREATED_IDS_EVENT_LIMIT = 50 +const RECALL_QUERY_EMBEDDING_TIMEOUT_MS = 800 +const RECALL_QUERY_EMBEDDING_STALE_MS = 30 * 1000 + +type SoftTimeoutResult = { timedOut: true } | { timedOut: false; value: T } +type QueryEmbeddingInFlight = { + startedAt: number + promise: Promise +} + +async function withSoftTimeout( + promise: Promise, + timeoutMs: number +): Promise> { + let timeoutId: NodeJS.Timeout | undefined + const guarded = promise.then((value) => ({ timedOut: false, value }) as SoftTimeoutResult) + guarded.catch(() => undefined) + const timeout = new Promise>((resolve) => { + timeoutId = setTimeout(() => resolve({ timedOut: true }), timeoutMs) + }) + try { + return await Promise.race([guarded, timeout]) + } finally { + if (timeoutId) clearTimeout(timeoutId) + } +} function embeddingFingerprint(providerId: string, modelId: string): string { return `${providerId}:${modelId}` @@ -282,6 +312,7 @@ export class MemoryPresenter implements MemoryRuntimePort { private readonly vectorStoreDimensionFailures = new Map() private readonly vectorStoreLocks = new Map>() private readonly embeddingWarmups = new Map>() + private readonly queryEmbeddingInFlight = new Map() // Serializes an agent's embedding drains. Distinct from vectorStoreLocks on purpose: this one // spans the network embedding call, the file lock must not. private readonly embeddingDrains = new Map>() @@ -1713,13 +1744,53 @@ export class MemoryPresenter implements MemoryRuntimePort { // embedding config, when the query has no vector hits, or while a reindex is rebuilding vectors. async recall(agentId: string, query: string, now = Date.now()): Promise { if (!this.canReadAgentMemory(agentId)) return [] - return this.retrieve(agentId, query, now, true) + return this.retrieve(agentId, query, now, true, { + keywordQuery: this.buildAgentFacingRecallKeywordQuery(agentId, query), + keywordMatchMode: 'any' + }) + } + + private buildAgentFacingRecallKeywordQuery(agentId: string, query: string): string { + const candidates = extractRecallKeywordCandidates(query) + if (!candidates.length) return '' + const stats = this.deps.repository.getRecallKeywordTermStats( + agentId, + candidates.map((candidate) => candidate.term) + ) + return buildRecallKeywordQuery(selectRecallKeywordTerms(candidates, stats)) + } + + private startQueryEmbedding( + agentId: string, + embedding: { providerId: string; modelId: string }, + query: string + ): Promise | null { + const key = `${agentId}::${embeddingFingerprint(embedding.providerId, embedding.modelId)}` + const now = Date.now() + const existing = this.queryEmbeddingInFlight.get(key) + // This is a tracked rate gate, not a provider-level hard concurrency cap: stale replacement + // does not abort the older provider request, it only lets a later turn try again. + if (existing && now - existing.startedAt < RECALL_QUERY_EMBEDDING_STALE_MS) return null + if (existing) { + logger.warn(`[Memory] stale query embedding replaced for ${agentId}`) + } + + const promise = this.deps.getEmbeddings(embedding.providerId, embedding.modelId, [query]) + const entry: QueryEmbeddingInFlight = { startedAt: now, promise } + this.queryEmbeddingInFlight.set(key, entry) + promise + .finally(() => { + if (this.queryEmbeddingInFlight.get(key) === entry) this.queryEmbeddingInFlight.delete(key) + }) + .catch(() => undefined) + return promise } - // Read-only search for the management surface: the same hybrid retrieval as recall, but it never - // records access, so browsing memories does not inflate access_count or skew archive-eligibility - // fairness. Re-queries the authoritative row for each hit and pairs it with its score; limit caps - // the result count only (it cannot widen the agent's configured topK). + // Read-only search for the management surface: it shares the hybrid retrieval core, but keeps the + // raw all-term keyword query and never records access, so browsing memories does not inflate + // access_count or skew archive-eligibility fairness. Re-queries the authoritative row for each hit + // and pairs it with its score; limit caps the result count only (it cannot widen the agent's + // configured topK). async searchMemories( agentId: string, query: string, @@ -1799,7 +1870,11 @@ export class MemoryPresenter implements MemoryRuntimePort { query: string, now: number, recordAccessHits: boolean, - trace: boolean = false + options: { + trace?: boolean + keywordQuery?: string + keywordMatchMode?: 'all' | 'any' + } = {} ): Promise { // The read path is gated on disposed too: after teardown begins it must neither reopen a vector // store nor record access on a database that is closing. @@ -1808,15 +1883,20 @@ export class MemoryPresenter implements MemoryRuntimePort { const { topK, rrfK, similarityThreshold, weights } = resolveRetrieval(config?.memoryRetrieval) const normalizedQuery = query.trim() if (!normalizedQuery) return [] + const normalizedKeywordQuery = (options.keywordQuery ?? normalizedQuery).trim() const candidateLimit = topK * 2 // Keyword path covers any status that still has content (embedded | fts_only | error). persona // is excluded (the self-model is injected separately, never recalled) and working (an internal // open-session cache row that must never feed back into recall). - const ftsRows = this.deps.repository - .search(agentId, normalizedQuery, candidateLimit) - .filter((row) => row.kind !== 'persona' && row.kind !== 'working') + const ftsRows = normalizedKeywordQuery + ? this.deps.repository + .search(agentId, normalizedKeywordQuery, candidateLimit, { + matchMode: options.keywordMatchMode ?? 'all' + }) + .filter((row) => row.kind !== 'persona' && row.kind !== 'working') + : [] // Vector path (embedded rows only). const vecMatches: { row: AgentMemoryRow; similarity: number }[] = [] @@ -1828,71 +1908,90 @@ export class MemoryPresenter implements MemoryRuntimePort { this.warmEmbeddingConnection(agentId, currentEmbedding) } else { try { - const vectors = await this.deps.getEmbeddings(embedding.providerId, embedding.modelId, [ + const queryEmbedding = this.startQueryEmbedding( + agentId, + currentEmbedding, normalizedQuery - ]) - // Teardown may have started during the embedding await: bail before opening the store so a - // late recall cannot reopen a sidecar the dispose close-loop has already passed. - if (!this.canReadAgentMemory(agentId)) return [] - const vector = vectors[0] - if (vector?.length) { - const fingerprint = embeddingFingerprint(embedding.providerId, embedding.modelId) - if (this.hasStaleEmbeddings(agentId, vector.length, fingerprint)) { - this.clearVectorStoreReady(agentId) - // The embedding model or dimension changed: rebuild vectors in the background and - // answer from FTS this turn instead of querying a store with stale dimensions. Skipped - // during teardown so no background write outlives the database connection. - if (this.canReadAgentMemory(agentId)) { - void this.reindexEmbeddings(agentId).catch((error) => { - logger.warn(`[Memory] reindex failed for ${agentId}: ${String(error)}`) - }) - } + ) + if (!queryEmbedding) { + logger.warn( + `[Memory] query embedding already in flight for ${agentId}; vector recall skipped this turn` + ) + } else { + const vectorsResult = await withSoftTimeout( + queryEmbedding, + RECALL_QUERY_EMBEDDING_TIMEOUT_MS + ) + if (vectorsResult.timedOut) { + logger.warn( + `[Memory] query embedding timed out for ${agentId}; vector recall skipped this turn` + ) } else { - const store = await this.getVectorStore(agentId, currentEmbedding, vector.length) - // Teardown may have begun while the store opened: bail before querying or reading rows. - // dispose awaits the per-agent open lock, so the store this call cached is closed there. + const vectors = vectorsResult.value + // Teardown may have started during the embedding await: bail before opening the store so a + // late recall cannot reopen a sidecar the dispose close-loop has already passed. if (!this.canReadAgentMemory(agentId)) return [] - if (store.isUsable()) { - this.markVectorStoreReady(agentId, currentEmbedding, vector.length) - const matches = await store.query(vector, { topK: candidateLimit }) - // ...and again after the query await, before any repository.getById on a closing DB. - if (!this.canReadAgentMemory(agentId)) return [] - for (const match of matches) { - const similarity = distanceToSimilarity(match.distance) - if (similarity < similarityThreshold) continue - const row = this.deps.repository.getById(match.memoryId) - // Skip persona even if an old/anomalous vector for it sits in the store: the - // self-model is injected separately, never recalled as a normal memory. working - // rows are never embedded, but skip them defensively too. Archived rows keep their - // vector but must stay out of recall until restored. - if ( - !row || - row.superseded_by || - row.kind === 'persona' || - row.kind === 'working' || - row.status === 'archived' || - row.status === 'conflicted' - ) - continue - vecMatches.push({ row, similarity }) + const vector = vectors[0] + if (vector?.length) { + const fingerprint = embeddingFingerprint(embedding.providerId, embedding.modelId) + if (this.hasStaleEmbeddings(agentId, vector.length, fingerprint)) { + this.clearVectorStoreReady(agentId) + // The embedding model or dimension changed: rebuild vectors in the background and + // answer from FTS this turn instead of querying a store with stale dimensions. Skipped + // during teardown so no background write outlives the database connection. + if (this.canReadAgentMemory(agentId)) { + void this.reindexEmbeddings(agentId).catch((error) => { + logger.warn(`[Memory] reindex failed for ${agentId}: ${String(error)}`) + }) + } + } else { + const store = await this.getVectorStore(agentId, currentEmbedding, vector.length) + // Teardown may have begun while the store opened: bail before querying or reading rows. + // dispose awaits the per-agent open lock, so the store this call cached is closed there. + if (!this.canReadAgentMemory(agentId)) return [] + if (store.isUsable()) { + this.markVectorStoreReady(agentId, currentEmbedding, vector.length) + const matches = await store.query(vector, { topK: candidateLimit }) + // ...and again after the query await, before any repository.getById on a closing DB. + if (!this.canReadAgentMemory(agentId)) return [] + for (const match of matches) { + const similarity = distanceToSimilarity(match.distance) + if (similarity < similarityThreshold) continue + const row = this.deps.repository.getById(match.memoryId) + // Skip persona even if an old/anomalous vector for it sits in the store: the + // self-model is injected separately, never recalled as a normal memory. working + // rows are never embedded, but skip them defensively too. Archived rows keep their + // vector but must stay out of recall until restored. + if ( + !row || + row.superseded_by || + row.kind === 'persona' || + row.kind === 'working' || + row.status === 'archived' || + row.status === 'conflicted' + ) + continue + vecMatches.push({ row, similarity }) + } + // The service embedded the query and the store is healthy: opportunistically embed + // rows deferred as fts_only (config added later) and re-drain any an earlier run left + // pending. Background, coalesced, and skipped while a reindex owns the requeue. + if (this.canReadAgentMemory(agentId) && !this.reindexing.has(agentId)) { + void this.backfillEmbeddings(agentId).catch((error) => { + logger.warn(`[Memory] backfill failed for ${agentId}: ${String(error)}`) + }) + } + } else if (this.canReadAgentMemory(agentId) && !this.reindexing.has(agentId)) { + this.clearVectorStoreReady(agentId) + // The on-disk sidecar carries a foreign/legacy identity we can never query (and there + // were no embedded rows to flag it as stale). Rebuild it under the current identity so + // the corpus stops failing closed; force the reset even if there is nothing to + // re-queue, since the unusable file itself is what blocks recovery. + void this.reindexEmbeddings(agentId, true).catch((error) => { + logger.warn(`[Memory] store rebuild failed for ${agentId}: ${String(error)}`) + }) + } } - // The service embedded the query and the store is healthy: opportunistically embed - // rows deferred as fts_only (config added later) and re-drain any an earlier run left - // pending. Background, coalesced, and skipped while a reindex owns the requeue. - if (this.canReadAgentMemory(agentId) && !this.reindexing.has(agentId)) { - void this.backfillEmbeddings(agentId).catch((error) => { - logger.warn(`[Memory] backfill failed for ${agentId}: ${String(error)}`) - }) - } - } else if (this.canReadAgentMemory(agentId) && !this.reindexing.has(agentId)) { - this.clearVectorStoreReady(agentId) - // The on-disk sidecar carries a foreign/legacy identity we can never query (and there - // were no embedded rows to flag it as stale). Rebuild it under the current identity so - // the corpus stops failing closed; force the reset even if there is nothing to - // re-queue, since the unusable file itself is what blocks recovery. - void this.reindexEmbeddings(agentId, true).catch((error) => { - logger.warn(`[Memory] store rebuild failed for ${agentId}: ${String(error)}`) - }) } } } @@ -1903,12 +2002,19 @@ export class MemoryPresenter implements MemoryRuntimePort { } } - const results = fuse(ftsRows, vecMatches, { topK, rrfK, weights, now, trace }) + const results = fuse(ftsRows, vecMatches, { + topK, + rrfK, + weights, + now, + trace: options.trace + }) // Re-check after the store/query awaits: never write access counters once teardown has begun. if (recordAccessHits && this.canReadAgentMemory(agentId)) { - for (const item of results) { - this.deps.repository.recordAccess(item.id, now) - } + this.deps.repository.recordAccessBatch( + results.map((item) => item.id), + now + ) } return results } @@ -1925,7 +2031,10 @@ export class MemoryPresenter implements MemoryRuntimePort { // path so the next open is served from L1. if (!working) this.scheduleWorkingRefresh(agentId) const recalled = query.trim() - ? await this.retrieve(agentId, query, Date.now(), true) + ? await this.retrieve(agentId, query, Date.now(), true, { + keywordQuery: this.buildAgentFacingRecallKeywordQuery(agentId, query), + keywordMatchMode: 'any' + }) : await this.recall(agentId, query) if (!persona && !working && recalled.length === 0) return null const tokenBudget = resolveInjectionTokenBudget(config?.memoryInjectionTokenBudget) @@ -2646,6 +2755,9 @@ export class MemoryPresenter implements MemoryRuntimePort { for (const key of this.vectorStoreDimensionFailures.keys()) { if (key.startsWith(`${agentId}::`)) this.vectorStoreDimensionFailures.delete(key) } + for (const key of this.queryEmbeddingInFlight.keys()) { + if (key.startsWith(`${agentId}::`)) this.queryEmbeddingInFlight.delete(key) + } this.vectorStoreReady.delete(agentId) } @@ -2703,6 +2815,7 @@ export class MemoryPresenter implements MemoryRuntimePort { this.vectorStoreWarmups.clear() this.vectorStoreDimensionFailures.clear() this.embeddingWarmups.clear() + this.queryEmbeddingInFlight.clear() this.vectorStoreLocks.clear() } diff --git a/src/main/presenter/memoryPresenter/recallKeyword.ts b/src/main/presenter/memoryPresenter/recallKeyword.ts new file mode 100644 index 000000000..ea52bdc22 --- /dev/null +++ b/src/main/presenter/memoryPresenter/recallKeyword.ts @@ -0,0 +1,114 @@ +import type { RecallKeywordTermStat } from './types' + +export type RecallKeywordCandidateKind = 'ascii' | 'code' | 'cjk' + +export interface RecallKeywordCandidate { + term: string + position: number + kind: RecallKeywordCandidateKind +} + +const RECALL_KEYWORD_MAX_CANDIDATES = 24 +const RECALL_KEYWORD_MAX_TERMS = 8 +const RECALL_KEYWORD_MIN_ASCII_TERM_LENGTH = 3 +const RECALL_KEYWORD_MIN_CODE_TERM_LENGTH = 2 +const RECALL_KEYWORD_CJK_WINDOW = 4 +const RECALL_KEYWORD_HIGH_FREQUENCY_MIN_ROWS = 4 +const RECALL_KEYWORD_HIGH_FREQUENCY_RATIO = 0.5 + +const CODE_EDGE_RE = /^[._:/@#+-]+|[._:/@#+-]+$/g +const CJK_SEQUENCE_RE = /^[\u3400-\u9fff\uf900-\ufaff\u3040-\u30ff\uac00-\ud7af]+$/u +const RECALL_KEYWORD_TOKEN_RE = + /[A-Za-z0-9_][A-Za-z0-9_.:/@#+-]*|[\u3400-\u9fff\uf900-\ufaff\u3040-\u30ff\uac00-\ud7af]+/gu + +function pushCandidate( + candidates: RecallKeywordCandidate[], + seen: Set, + term: string, + position: number, + kind: RecallKeywordCandidateKind +): void { + const normalized = term.trim().toLowerCase().replace(CODE_EDGE_RE, '') + if (!normalized || seen.has(normalized)) return + seen.add(normalized) + candidates.push({ term: normalized, position, kind }) +} + +function classifyAsciiTerm(term: string): RecallKeywordCandidateKind { + return /[._:/@#+-]|\d/u.test(term) ? 'code' : 'ascii' +} + +export function extractRecallKeywordCandidates(query: string): RecallKeywordCandidate[] { + const candidates: RecallKeywordCandidate[] = [] + const seen = new Set() + + for (const match of query.matchAll(RECALL_KEYWORD_TOKEN_RE)) { + if (candidates.length >= RECALL_KEYWORD_MAX_CANDIDATES) break + const sequence = match[0] + const start = match.index ?? 0 + if (CJK_SEQUENCE_RE.test(sequence)) { + if (sequence.length <= RECALL_KEYWORD_CJK_WINDOW) { + pushCandidate(candidates, seen, sequence, start, 'cjk') + } else { + for (let index = 0; index <= sequence.length - RECALL_KEYWORD_CJK_WINDOW; index += 1) { + if (candidates.length >= RECALL_KEYWORD_MAX_CANDIDATES) break + pushCandidate( + candidates, + seen, + sequence.slice(index, index + RECALL_KEYWORD_CJK_WINDOW), + start + index, + 'cjk' + ) + } + } + } else { + const rawTerm = sequence.replace(CODE_EDGE_RE, '') + const kind = classifyAsciiTerm(rawTerm) + const minLength = + kind === 'code' ? RECALL_KEYWORD_MIN_CODE_TERM_LENGTH : RECALL_KEYWORD_MIN_ASCII_TERM_LENGTH + if (rawTerm.length >= minLength) { + pushCandidate(candidates, seen, rawTerm, start, kind) + } + } + } + + return candidates +} + +export function selectRecallKeywordTerms( + candidates: RecallKeywordCandidate[], + stats: RecallKeywordTermStat[] +): string[] { + const byTerm = new Map(stats.map((stat) => [stat.term.toLowerCase(), stat])) + const scored = candidates + .map((candidate) => ({ candidate, stat: byTerm.get(candidate.term) })) + .filter( + (entry): entry is { candidate: RecallKeywordCandidate; stat: RecallKeywordTermStat } => + (entry.stat?.hitCount ?? 0) > 0 + ) + + if (!scored.length) return [] + + const lowFrequency = scored.filter( + ({ stat }) => + stat.totalRows < RECALL_KEYWORD_HIGH_FREQUENCY_MIN_ROWS || + stat.hitCount <= stat.totalRows * RECALL_KEYWORD_HIGH_FREQUENCY_RATIO + ) + const pool = lowFrequency.length ? lowFrequency : scored + const selected = [...pool] + .sort( + (left, right) => + left.stat.hitCount - right.stat.hitCount || + right.candidate.term.length - left.candidate.term.length || + left.candidate.position - right.candidate.position + ) + .slice(0, lowFrequency.length ? RECALL_KEYWORD_MAX_TERMS : 1) + + return selected + .sort((left, right) => left.candidate.position - right.candidate.position) + .map(({ candidate }) => candidate.term) +} + +export function buildRecallKeywordQuery(terms: string[]): string { + return terms.join(' ') +} diff --git a/src/main/presenter/memoryPresenter/types.ts b/src/main/presenter/memoryPresenter/types.ts index fd89dd6fa..7cbed33df 100644 --- a/src/main/presenter/memoryPresenter/types.ts +++ b/src/main/presenter/memoryPresenter/types.ts @@ -48,7 +48,13 @@ export interface MemoryRepositoryPort { setPersonaState(id: string, state: AgentMemoryPersonaState, supersededBy?: string | null): void setAnchor(id: string, anchored: boolean): void listPersonaVersions(agentId: string): AgentMemoryRow[] - search(agentId: string, query: string, limit?: number): AgentMemoryRow[] + search( + agentId: string, + query: string, + limit?: number, + options?: { matchMode?: 'all' | 'any' } + ): AgentMemoryRow[] + getRecallKeywordTermStats(agentId: string, terms: string[]): RecallKeywordTermStat[] listPendingEmbedding(limit?: number, agentId?: string): AgentMemoryRow[] updateStatus( id: string, @@ -75,6 +81,7 @@ export interface MemoryRepositoryPort { requeueForEmbedding(agentId: string, statuses: AgentMemoryStatus[]): number markSuperseded(id: string, supersededBy: string | null): void recordAccess(id: string, accessedAt?: number): void + recordAccessBatch(ids: string[], accessedAt?: number): void updateDecayScore(id: string, decayScore: number | null, consolidatedAt?: number | null): void updateContent( id: string, @@ -109,6 +116,12 @@ export interface MemoryRepositoryPort { listAgentIdsWithMemories(): string[] } +export interface RecallKeywordTermStat { + term: string + hitCount: number + totalRows: number +} + export interface MemoryAuditRepositoryPort { insert(input: AgentMemoryAuditInsertInput): AgentMemoryAuditRow listByAgent(agentId: string, options?: number | MemoryAuditListOptions): AgentMemoryAuditRow[] diff --git a/src/main/presenter/sqlitePresenter/tables/agentMemory.ts b/src/main/presenter/sqlitePresenter/tables/agentMemory.ts index b4c3a6c7b..e94fa798f 100644 --- a/src/main/presenter/sqlitePresenter/tables/agentMemory.ts +++ b/src/main/presenter/sqlitePresenter/tables/agentMemory.ts @@ -114,6 +114,8 @@ const AGENT_MEMORY_FTS_META_KEY = 'agent_memory_fts' const AGENT_MEMORY_FTS_META_VERSION = 1 type FtsCapability = { available: boolean; tokenizer: 'trigram' | 'unicode61' } +type SearchMatchMode = 'all' | 'any' +type RecallKeywordTermStat = { term: string; hitCount: number; totalRows: number } const AGENT_MEMORY_INDEX_SQL = ` CREATE INDEX IF NOT EXISTS idx_agent_memory_agent_kind @@ -604,12 +606,18 @@ export class AgentMemoryTable extends BaseTable { // in full so the result is never a subset of the old LIKE behavior — gating LIKE behind the cap // would silently drop high-importance rows whenever FTS5 alone filled it. Each path is bounded // by `limit`, so the union is bounded by `2 * limit`; downstream RRF reranks and trims. - search(agentId: string, query: string, limit: number = 20): AgentMemoryRow[] { + search( + agentId: string, + query: string, + limit: number = 20, + options: { matchMode?: SearchMatchMode } = {} + ): AgentMemoryRow[] { const normalized = query.trim() if (!normalized) { return [] } const cappedLimit = Math.min(Math.max(Math.floor(limit), 1), 100) + const matchMode = options.matchMode ?? 'all' const ordered: AgentMemoryRow[] = [] const seen = new Set() const collect = (rows: AgentMemoryRow[]): void => { @@ -620,18 +628,60 @@ export class AgentMemoryTable extends BaseTable { } } if (this.ftsReady) { - collect(this.searchFts(agentId, normalized, cappedLimit)) + collect(this.searchFts(agentId, normalized, cappedLimit, matchMode)) } - collect(this.searchLike(agentId, normalized, cappedLimit)) + collect(this.searchLike(agentId, normalized, cappedLimit, matchMode)) return ordered } - private searchFts(agentId: string, normalized: string, limit: number): AgentMemoryRow[] { + getRecallKeywordTermStats(agentId: string, terms: string[]): RecallKeywordTermStat[] { + const normalizedTerms = [...new Set(terms.map((term) => term.trim().toLowerCase()))].filter( + Boolean + ) + if (!normalizedTerms.length) return [] + const hitColumns = normalizedTerms + .map( + (_term, index) => + `SUM(CASE WHEN content LIKE ? ESCAPE '\\' THEN 1 ELSE 0 END) AS hit_${index}` + ) + .join(',\n ') + const row = this.db + .prepare( + `SELECT COUNT(*) AS totalRows, + ${hitColumns} + FROM agent_memory + WHERE agent_id = ? + AND superseded_by IS NULL + AND status != 'archived' + AND status != 'conflicted' + AND kind NOT IN ('persona', 'working')` + ) + .get(...normalizedTerms.map((term) => `%${escapeLikePattern(term)}%`), agentId) as Record< + string, + number | null + > + const totalRows = Number(row.totalRows ?? 0) + return normalizedTerms.map((term, index) => { + return { + term, + hitCount: Number(row[`hit_${index}`] ?? 0), + totalRows + } + }) + } + + private searchFts( + agentId: string, + normalized: string, + limit: number, + matchMode: SearchMatchMode + ): AgentMemoryRow[] { const terms = tokenizeSearchQuery(normalized) if (!terms.length) return [] - // Quote each token so user text cannot inject FTS5 operators; join with AND so multi-word - // searches match memories containing all terms rather than requiring one exact phrase. - const match = terms.map((term) => `"${term.replace(/"/g, '""')}"`).join(' AND ') + // Quote each token so user text cannot inject FTS5 operators; the caller chooses whether + // all terms or any term must match. + const operator = matchMode === 'any' ? ' OR ' : ' AND ' + const match = terms.map((term) => `"${term.replace(/"/g, '""')}"`).join(operator) try { return this.db .prepare( @@ -653,11 +703,17 @@ export class AgentMemoryTable extends BaseTable { } } - private searchLike(agentId: string, normalized: string, limit: number): AgentMemoryRow[] { + private searchLike( + agentId: string, + normalized: string, + limit: number, + matchMode: SearchMatchMode + ): AgentMemoryRow[] { const terms = tokenizeSearchQuery(normalized) if (!terms.length) return [] const clauses = terms.map(() => "content LIKE ? ESCAPE '\\'") const params = terms.map((term) => `%${escapeLikePattern(term)}%`) + const operator = matchMode === 'any' ? ' OR ' : ' AND ' return this.db .prepare( `SELECT * FROM agent_memory @@ -666,7 +722,7 @@ export class AgentMemoryTable extends BaseTable { AND status != 'archived' AND status != 'conflicted' AND kind != 'working' - AND ${clauses.join(' AND ')} + AND (${clauses.join(operator)}) ORDER BY importance DESC, created_at DESC LIMIT ?` ) @@ -796,6 +852,19 @@ export class AgentMemoryTable extends BaseTable { .run(accessedAt, id) } + recordAccessBatch(ids: string[], accessedAt: number = Date.now()): void { + const uniqueIds = [...new Set(ids.filter((id) => id.trim()))] + if (!uniqueIds.length) return + const placeholders = uniqueIds.map(() => '?').join(', ') + this.db + .prepare( + `UPDATE agent_memory + SET last_accessed = ?, access_count = access_count + 1 + WHERE id IN (${placeholders})` + ) + .run(accessedAt, ...uniqueIds) + } + // Omitting `consolidatedAt` (COALESCE keeps the prior value) leaves the LLM consolidation marker // untouched for callers that only refresh decay. updateDecayScore( diff --git a/src/renderer/settings/components/MemoryConfigPanel.vue b/src/renderer/settings/components/MemoryConfigPanel.vue index 52994155d..0bcb0c621 100644 --- a/src/renderer/settings/components/MemoryConfigPanel.vue +++ b/src/renderer/settings/components/MemoryConfigPanel.vue @@ -90,7 +90,14 @@ /> -

+

{{ t('settings.deepchatAgents.memoryEmbeddingHint') }}

@@ -174,7 +181,14 @@ /> -

+

{{ t('settings.memory.config.extractionModelHint') }}

diff --git a/test/main/presenter/agentMemoryTable.test.ts b/test/main/presenter/agentMemoryTable.test.ts index 253c61e74..8ad346881 100644 --- a/test/main/presenter/agentMemoryTable.test.ts +++ b/test/main/presenter/agentMemoryTable.test.ts @@ -648,6 +648,97 @@ describeIfSqlite('AgentMemoryTable', () => { } }) + it('supports OR keyword matching only when requested', () => { + const db = new DatabaseCtor(':memory:') + try { + const table = new AgentMemoryTableCtor(db) + table.createTable() + table.insert({ + id: 'm1', + agentId: 'deepchat', + kind: 'semantic', + content: 'redis setup' + }) + + expect(table.search('deepchat', 'please redis setup').map((row) => row.id)).toEqual([]) + expect( + table + .search('deepchat', 'please redis setup', 20, { matchMode: 'any' }) + .map((row) => row.id) + ).toEqual(['m1']) + } finally { + db.close() + } + }) + + it('counts recall keyword term stats over active recallable rows only', () => { + const db = new DatabaseCtor(':memory:') + try { + const table = new AgentMemoryTableCtor(db) + table.createTable() + table.insert({ id: 'm1', agentId: 'deepchat', kind: 'semantic', content: 'redis setup' }) + table.insert({ id: 'm2', agentId: 'deepchat', kind: 'semantic', content: 'please notes' }) + table.insert({ id: 'p1', agentId: 'deepchat', kind: 'persona', content: 'redis persona' }) + table.insert({ id: 'w1', agentId: 'deepchat', kind: 'working', content: 'redis working' }) + table.insert({ + id: 'a1', + agentId: 'deepchat', + kind: 'semantic', + content: 'redis archived', + status: 'archived' + }) + table.insert({ + id: 'c1', + agentId: 'deepchat', + kind: 'semantic', + content: 'redis conflicted', + status: 'conflicted' + }) + const old = table.insert({ + id: 'old', + agentId: 'deepchat', + kind: 'semantic', + content: 'redis old' + }) + const fresh = table.insert({ + id: 'fresh', + agentId: 'deepchat', + kind: 'semantic', + content: 'redis fresh' + }) + table.markSuperseded(old.id, fresh.id) + + expect( + table.getRecallKeywordTermStats('deepchat', ['redis', 'please', 'redis', 'missing']) + ).toEqual([ + { term: 'redis', hitCount: 2, totalRows: 3 }, + { term: 'please', hitCount: 1, totalRows: 3 }, + { term: 'missing', hitCount: 0, totalRows: 3 } + ]) + } finally { + db.close() + } + }) + + it('updates access counters in batch', () => { + const db = new DatabaseCtor(':memory:') + try { + const table = new AgentMemoryTableCtor(db) + table.createTable() + table.insert({ id: 'm1', agentId: 'deepchat', kind: 'semantic', content: 'a' }) + table.insert({ id: 'm2', agentId: 'deepchat', kind: 'semantic', content: 'b' }) + + table.recordAccessBatch(['m1', 'm2', 'm1'], 1234) + + expect(table.getById('m1')?.access_count).toBe(1) + expect(table.getById('m2')?.access_count).toBe(1) + expect(table.getById('m1')?.last_accessed).toBe(1234) + expect(table.getById('m2')?.last_accessed).toBe(1234) + } finally { + db.close() + } + }) + it('clears all memories for an agent', () => { const db = new DatabaseCtor(':memory:') try { diff --git a/test/main/presenter/fakes/memoryFakes.ts b/test/main/presenter/fakes/memoryFakes.ts index 3b055b1d3..ed478c013 100644 --- a/test/main/presenter/fakes/memoryFakes.ts +++ b/test/main/presenter/fakes/memoryFakes.ts @@ -169,8 +169,10 @@ export class FakeRepository implements MemoryRepositoryPort { .sort((a, b) => b.created_at - a.created_at) } - search(agentId: string, query: string, limit = 20) { - const q = query.toLowerCase() + search(agentId: string, query: string, limit = 20, options: { matchMode?: 'all' | 'any' } = {}) { + const terms = query.trim().toLowerCase().split(/\s+/u).filter(Boolean) + if (!terms.length) return [] + const matchMode = options.matchMode ?? 'all' return [...this.rows.values()] .filter( (row) => @@ -179,11 +181,33 @@ export class FakeRepository implements MemoryRepositoryPort { row.status !== 'archived' && row.status !== 'conflicted' && row.kind !== 'working' && - row.content.toLowerCase().includes(q) + (matchMode === 'any' + ? terms.some((term) => row.content.toLowerCase().includes(term)) + : terms.every((term) => row.content.toLowerCase().includes(term))) ) .slice(0, limit) } + getRecallKeywordTermStats(agentId: string, terms: string[]) { + const normalizedTerms = [...new Set(terms.map((term) => term.trim().toLowerCase()))].filter( + Boolean + ) + const rows = [...this.rows.values()].filter( + (row) => + row.agent_id === agentId && + !row.superseded_by && + row.status !== 'archived' && + row.status !== 'conflicted' && + row.kind !== 'persona' && + row.kind !== 'working' + ) + return normalizedTerms.map((term) => ({ + term, + hitCount: rows.filter((row) => row.content.toLowerCase().includes(term)).length, + totalRows: rows.length + })) + } + listPendingEmbedding(limit = 50, agentId?: string) { return [...this.rows.values()] .filter( @@ -275,6 +299,12 @@ export class FakeRepository implements MemoryRepositoryPort { } } + recordAccessBatch(ids: string[], accessedAt = 0) { + for (const id of new Set(ids)) { + this.recordAccess(id, accessedAt) + } + } + updateDecayScore(id: string, decayScore: number | null, consolidatedAt: number | null = null) { const row = this.rows.get(id) if (row) { diff --git a/test/main/presenter/memoryPresenter.test.ts b/test/main/presenter/memoryPresenter.test.ts index f4c5612da..72ffcb9c3 100644 --- a/test/main/presenter/memoryPresenter.test.ts +++ b/test/main/presenter/memoryPresenter.test.ts @@ -1597,6 +1597,268 @@ describe('MemoryPresenter management', () => { expect(querySpy).toHaveBeenCalledTimes(1) }) + it('agent-facing recall keywordizes long English messages for FTS-only recall', async () => { + const { presenter, repo } = makePresenter({ memoryEnabled: true }) + repo.insert({ + id: 'm1', + agentId: 'a', + kind: 'semantic', + content: 'redis setup', + status: 'fts_only' + }) + + const recalled = await presenter.recall('a', 'Could you please explain the redis setup again?') + + expect(recalled.map((item) => item.id)).toEqual(['m1']) + }) + + it('agent-facing recall filters high-frequency generic terms by corpus stats', async () => { + const { presenter, repo } = makePresenter({ memoryEnabled: true }) + repo.insert({ + id: 'm1', + agentId: 'a', + kind: 'semantic', + content: 'redis setup', + status: 'fts_only' + }) + repo.insert({ + id: 'm2', + agentId: 'a', + kind: 'semantic', + content: 'please review the dashboard notes', + status: 'fts_only' + }) + repo.insert({ + id: 'm3', + agentId: 'a', + kind: 'semantic', + content: 'please summarize the release notes', + status: 'fts_only' + }) + repo.insert({ + id: 'm4', + agentId: 'a', + kind: 'semantic', + content: 'please update the checklist', + status: 'fts_only' + }) + + const recalled = await presenter.recall('a', 'please redis setup') + + expect(recalled.map((item) => item.id)).toEqual(['m1']) + }) + + it('agent-facing recall keeps a single domain term even when it is frequent', async () => { + const { presenter, repo } = makePresenter({ memoryEnabled: true }) + for (const id of ['m1', 'm2', 'm3', 'm4']) { + repo.insert({ + id, + agentId: 'a', + kind: 'semantic', + content: `${id} redis note`, + status: 'fts_only' + }) + } + + const recalled = await presenter.recall('a', 'redis') + + expect(recalled.map((item) => item.id).sort()).toEqual(['m1', 'm2', 'm3', 'm4']) + }) + + it('agent-facing recall skips keyword search when no candidate hits but still tries vector recall', async () => { + const repo = new FakeRepository() + const store = new FakeVectorStore() + const getEmbeddings = vi.fn(async (_p: string, _m: string, texts: string[]) => + texts.map((text) => textToVector(text)) + ) + const presenter = new MemoryPresenter({ + repository: repo, + resolveAgentConfig: () => enabledConfig, + getEmbeddings, + getDimensions: embeddingDimensions, + createVectorStore: async () => store, + resetVectorStore: async () => undefined + }) + presenter.writeMemoriesSync([{ kind: 'semantic', content: 'redis setup' }], { agentId: 'a' }) + await presenter.processPendingEmbeddings('a') + const querySpy = vi.spyOn(store, 'query') + + await presenter.recall('a', 'zzzz qqqq') + + expect(getEmbeddings).toHaveBeenCalledWith('p', 'm', ['zzzz qqqq']) + expect(querySpy).toHaveBeenCalled() + }) + + it('agent-facing recall keywordizes CJK messages instead of requiring an entire sentence match', async () => { + const { presenter, repo } = makePresenter({ memoryEnabled: true }) + repo.insert({ + id: 'm1', + agentId: 'a', + kind: 'semantic', + content: '用户偏好简洁的中文回答问题', + status: 'fts_only' + }) + + const recalled = await presenter.recall('a', '请继续用中文回答这个问题，谢谢') + + expect(recalled.map((item) => item.id)).toEqual(['m1']) + }) + + it('warm recall soft-times out query embedding and keeps vector readiness for later turns', async () => { + vi.useFakeTimers() + try { + const repo = new FakeRepository() + const store = new FakeVectorStore() + let blockQueryEmbedding = false + const getEmbeddings = vi.fn((_p: string, _m: string, texts: string[]) => { + if (blockQueryEmbedding && texts[0] !== 'memory warmup') { + return new Promise(() => undefined) + } + return Promise.resolve(texts.map((text) => textToVector(text))) + }) + const createVectorStore = vi.fn(async () => store) + const presenter = new MemoryPresenter({ + repository: repo, + resolveAgentConfig: () => enabledConfig, + getEmbeddings, + getDimensions: embeddingDimensions, + createVectorStore, + resetVectorStore: async () => undefined + }) + const [memoryId] = presenter.writeMemoriesSync( + [{ kind: 'semantic', content: 'redis setup' }], + { + agentId: 'a' + } + ) + await presenter.processPendingEmbeddings('a') + expect( + (presenter as unknown as { vectorStoreReady: Map }).vectorStoreReady.has( + 'a' + ) + ).toBe(true) + + blockQueryEmbedding = true + const clearReadySpy = vi.spyOn( + presenter as unknown as { clearVectorStoreReady: (agentId: string) => void }, + 'clearVectorStoreReady' + ) + const backfillSpy = vi.spyOn(presenter, 'backfillEmbeddings') + const reindexSpy = vi.spyOn(presenter, 'reindexEmbeddings') + const recall = presenter.recall('a', 'Could you explain the redis setup again?') + + await vi.advanceTimersByTimeAsync(801) + const recalled = await recall + + expect(recalled.map((item) => item.id)).toEqual([memoryId]) + expect(clearReadySpy).not.toHaveBeenCalled() + expect(backfillSpy).not.toHaveBeenCalled() + expect(reindexSpy).not.toHaveBeenCalled() + expect( + (presenter as unknown as { vectorStoreReady: Map }).vectorStoreReady.has( + 'a' + ) + ).toBe(true) + } finally { + vi.useRealTimers() + } + }) + + it('skips duplicate query embeddings while a timed-out request is still in flight', async () => { + vi.useFakeTimers() + try { + const repo = new FakeRepository() + const store = new FakeVectorStore() + let blockQueryEmbedding = false + let queryEmbeddingCalls = 0 + const getEmbeddings = vi.fn((_p: string, _m: string, texts: string[]) => { + if (blockQueryEmbedding && texts[0] !== 'memory warmup') { + queryEmbeddingCalls += 1 + return new Promise(() => undefined) + } + return Promise.resolve(texts.map((text) => textToVector(text))) + }) + const presenter = new MemoryPresenter({ + repository: repo, + resolveAgentConfig: () => enabledConfig, + getEmbeddings, + getDimensions: embeddingDimensions, + createVectorStore: async () => store, + resetVectorStore: async () => undefined + }) + const [memoryId] = presenter.writeMemoriesSync( + [{ kind: 'semantic', content: 'redis setup' }], + { agentId: 'a' } + ) + await presenter.processPendingEmbeddings('a') + + blockQueryEmbedding = true + const first = presenter.recall('a', 'redis setup') + await vi.advanceTimersByTimeAsync(801) + expect((await first).map((item) => item.id)).toEqual([memoryId]) + expect(queryEmbeddingCalls).toBe(1) + + const second = await presenter.recall('a', 'redis setup') + + expect(second.map((item) => item.id)).toEqual([memoryId]) + expect(queryEmbeddingCalls).toBe(1) + } finally { + vi.useRealTimers() + } + }) + + it('replaces stale query embedding in-flight entries without late-settle deletion', async () => { + vi.useFakeTimers() + try { + const repo = new FakeRepository() + const store = new FakeVectorStore() + let blockQueryEmbedding = false + let queryEmbeddingCalls = 0 + const pendingQueryEmbeddings: Array<(vectors: number[][]) => void> = [] + const getEmbeddings = vi.fn((_p: string, _m: string, texts: string[]) => { + if (blockQueryEmbedding && texts[0] !== 'memory warmup') { + queryEmbeddingCalls += 1 + return new Promise((resolve) => pendingQueryEmbeddings.push(resolve)) + } + return Promise.resolve(texts.map((text) => textToVector(text))) + }) + const presenter = new MemoryPresenter({ + repository: repo, + resolveAgentConfig: () => enabledConfig, + getEmbeddings, + getDimensions: embeddingDimensions, + createVectorStore: async () => store, + resetVectorStore: async () => undefined + }) + const [memoryId] = presenter.writeMemoriesSync( + [{ kind: 'semantic', content: 'redis setup' }], + { agentId: 'a' } + ) + await presenter.processPendingEmbeddings('a') + + blockQueryEmbedding = true + const first = presenter.recall('a', 'redis setup') + await vi.advanceTimersByTimeAsync(801) + expect((await first).map((item) => item.id)).toEqual([memoryId]) + expect(queryEmbeddingCalls).toBe(1) + + await vi.advanceTimersByTimeAsync(30_001) + const second = presenter.recall('a', 'redis setup') + await vi.advanceTimersByTimeAsync(801) + expect((await second).map((item) => item.id)).toEqual([memoryId]) + expect(queryEmbeddingCalls).toBe(2) + + pendingQueryEmbeddings[0]?.([textToVector('redis setup')]) + await flushMicrotasks() + const third = presenter.recall('a', 'redis setup') + await vi.advanceTimersByTimeAsync(801) + expect((await third).map((item) => item.id)).toEqual([memoryId]) + expect(queryEmbeddingCalls).toBe(2) + } finally { + vi.useRealTimers() + } + }) + it('cools down repeated getDimensions failures while keeping cold recall on FTS', async () => { const repo = new FakeRepository() const getDimensions = vi.fn(async () => { @@ -4980,7 +5242,7 @@ describe('MemoryPresenter lifecycle revival (SDD-8)', () => { // A recall starts and parks inside getEmbeddings. blockRecall = true - const recordSpy = vi.spyOn(repo, 'recordAccess') + const recordSpy = vi.spyOn(repo, 'recordAccessBatch') const recall = presenter.recall('a', 'redis') await new Promise((r) => setTimeout(r, 0)) @@ -5130,7 +5392,7 @@ describe('MemoryPresenter lifecycle revival (SDD-8)', () => { blockCreate = true const getByIdSpy = vi.spyOn(repo, 'getById') - const recordSpy = vi.spyOn(repo, 'recordAccess') + const recordSpy = vi.spyOn(repo, 'recordAccessBatch') const backfillSpy = vi.spyOn(presenter, 'backfillEmbeddings') const reindexSpy = vi.spyOn(presenter, 'reindexEmbeddings') const closeSpy = vi.spyOn(store, 'close') @@ -5139,7 +5401,7 @@ describe('MemoryPresenter lifecycle revival (SDD-8)', () => { const results = await recall expect(results.some((item) => item.id === 'm1')).toBe(true) expect(getByIdSpy).not.toHaveBeenCalled() - expect(recordSpy).toHaveBeenCalledWith('m1', expect.any(Number)) + expect(recordSpy).toHaveBeenCalledWith(['m1'], expect.any(Number)) let disposed = false const disposePromise = presenter.dispose().then(() => { @@ -5202,7 +5464,7 @@ describe('MemoryPresenter lifecycle revival (SDD-8)', () => { blockQuery = true const getByIdSpy = vi.spyOn(repo, 'getById') - const recordSpy = vi.spyOn(repo, 'recordAccess') + const recordSpy = vi.spyOn(repo, 'recordAccessBatch') const backfillSpy = vi.spyOn(presenter, 'backfillEmbeddings') const recall = presenter.recall('a', 'redis') await new Promise((r) => setTimeout(r, 0)) // park inside store.query diff --git a/test/main/presenter/memorySearch.test.ts b/test/main/presenter/memorySearch.test.ts index fa9e2c629..1ad8037a8 100644 --- a/test/main/presenter/memorySearch.test.ts +++ b/test/main/presenter/memorySearch.test.ts @@ -29,7 +29,7 @@ describe('MemoryPresenter.searchMemories (read-only facade)', () => { it('never records access while recall does (browsing must not skew fairness)', async () => { const { presenter, repo } = makePresenter(enabledConfig) repo.insert(seed('the user prefers redis', 'm1')) - const accessSpy = vi.spyOn(repo, 'recordAccess') + const accessSpy = vi.spyOn(repo, 'recordAccessBatch') await presenter.searchMemories('deepchat', 'redis') expect(accessSpy).not.toHaveBeenCalled() @@ -39,6 +39,16 @@ describe('MemoryPresenter.searchMemories (read-only facade)', () => { expect(accessSpy).toHaveBeenCalled() }) + it('keeps management search on precise all-term keyword matching', async () => { + const { presenter, repo } = makePresenter({ memoryEnabled: true }) + repo.insert(seed('redis setup', 'm1')) + + expect(await presenter.searchMemories('deepchat', 'please redis setup')).toEqual([]) + expect( + (await presenter.recall('deepchat', 'please redis setup')).map((item) => item.id) + ).toEqual(['m1']) + }) + it('caps the result count to limit without widening topK', async () => { const { presenter, repo } = makePresenter(enabledConfig) repo.insert(seed('redis caching notes', 'm1')) diff --git a/test/main/presenter/recallKeyword.test.ts b/test/main/presenter/recallKeyword.test.ts new file mode 100644 index 000000000..c9e0250f4 --- /dev/null +++ b/test/main/presenter/recallKeyword.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it } from 'vitest' + +import { + extractRecallKeywordCandidates, + selectRecallKeywordTerms +} from '@/presenter/memoryPresenter/recallKeyword' +import type { RecallKeywordTermStat } from '@/presenter/memoryPresenter/types' + +function stats(entries: Array<[string, number, number]>): RecallKeywordTermStat[] { + return entries.map(([term, hitCount, totalRows]) => ({ term, hitCount, totalRows })) +} + +describe('recall keyword selection', () => { + it('extracts ascii, code-like, and CJK candidates without duplicates', () => { + const candidates = extractRecallKeywordCandidates( + 'Please retry api/v1.redis_error in 中文回答问题 and retry api/v1.redis_error' + ) + + expect(candidates.map((candidate) => [candidate.term, candidate.kind])).toEqual([ + ['please', 'ascii'], + ['retry', 'ascii'], + ['api/v1.redis_error', 'code'], + ['中文回答', 'cjk'], + ['文回答问', 'cjk'], + ['回答问题', 'cjk'], + ['and', 'ascii'] + ]) + }) + + it('caps candidate extraction before unbounded query text can expand work', () => { + const candidates = extractRecallKeywordCandidates( + Array.from({ length: 40 }, (_value, index) => `term${index}`).join(' ') + ) + + expect(candidates).toHaveLength(24) + expect(candidates.at(-1)?.term).toBe('term23') + }) + + it('preserves earlier CJK candidates before more than 24 ASCII tokens under the cap', () => { + const candidates = extractRecallKeywordCandidates( + `中文回答问题 ${Array.from({ length: 40 }, (_value, index) => `term${index}`).join(' ')}` + ) + + expect(candidates).toHaveLength(24) + expect(candidates.slice(0, 3).map((candidate) => candidate.term)).toEqual([ + '中文回答', + '文回答问', + '回答问题' + ]) + expect(candidates.some((candidate) => candidate.term === 'term23')).toBe(false) + }) + + it('selects lower-frequency terms but emits them in query order', () => { + const candidates = extractRecallKeywordCandidates('please redis setup dashboard') + + expect( + selectRecallKeywordTerms( + candidates, + stats([ + ['please', 8, 10], + ['redis', 1, 10], + ['setup', 2, 10], + ['dashboard', 1, 10] + ]) + ) + ).toEqual(['redis', 'setup', 'dashboard']) + }) + + it('falls back to the rarest single term when every hit is high-frequency', () => { + const candidates = extractRecallKeywordCandidates('redis setup cache') + + expect( + selectRecallKeywordTerms( + candidates, + stats([ + ['redis', 9, 10], + ['setup', 8, 10], + ['cache', 10, 10] + ]) + ) + ).toEqual(['setup']) + }) + + it('drops terms that do not hit the active corpus', () => { + const candidates = extractRecallKeywordCandidates('unknown redis') + + expect( + selectRecallKeywordTerms( + candidates, + stats([ + ['unknown', 0, 10], + ['redis', 1, 10] + ]) + ) + ).toEqual(['redis']) + }) +}) diff --git a/test/renderer/components/MemoryConfigPanel.test.ts b/test/renderer/components/MemoryConfigPanel.test.ts index a356071d5..d8a9e1d59 100644 --- a/test/renderer/components/MemoryConfigPanel.test.ts +++ b/test/renderer/components/MemoryConfigPanel.test.ts @@ -133,6 +133,18 @@ describe('MemoryConfigPanel override semantics (AC-2.1~2.5)', () => { expect('personaEvolutionEnabled' in config).toBe(false) }) + it('highlights degraded memory model hints when optional models are unset', async () => { + const { wrapper } = await setup({ memoryEnabled: true }, { memoryEnabled: true }) + + expect(wrapper.html()).toContain('settings.deepchatAgents.memoryEmbeddingHint') + expect(wrapper.html()).toContain('bg-amber-500/10') + + await openAdvancedSettings(wrapper) + + expect(wrapper.html()).toContain('settings.memory.config.extractionModelHint') + expect(wrapper.html()).toContain('bg-amber-500/10') + }) + it('writes the boolean override only after the switch is toggled', async () => { const { wrapper, updateDeepChatAgent } = await setup({}, { memoryEnabled: true })