diff --git a/AGENTS.md b/AGENTS.md index 45ccf77..6521d20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,6 +23,7 @@ This document is the LeapFlow engineering collaboration contract. It is not only - Design for generalization and universality; prefer reusable domain concepts over one-off special cases - Easy to extend, avoid hardcoding and hard rules - Industrial-grade robustness: every external call has timeout, retry, and fallback +- Structured error propagation: failures must flow as typed envelopes (`FailureEnvelope`) with source, category, recoverability, and side-effect state — never as ad-hoc dicts, bare strings, or unclassified exceptions - User experience is a first-class quality bar: optimize for clarity, ease of use, fast feedback, and graceful recovery - All comments and docstrings in English - Type annotations on all public APIs @@ -48,6 +49,10 @@ This document is the LeapFlow engineering collaboration contract. It is not only - **Graceful Degradation**: Every optional component (LLM, Hub) can be absent without crash - **Single Source of Truth**: DuckDB for persistence, EventBus for communication, Settings for configuration - **Inbound Signal Classification**: Platform events must be classified before they activate the agent. Message/callback events may enter Decide; signal/lifecycle events should be stored or routed without triggering LLM by default; ignored events must be explicit (e.g. self-message, duplicate, blocked scope). +- **Single Recovery Decision Point**: All agent loop errors (LLM, tool, system, security) enter one `RecoveryCoordinator`. No parallel decision paths, no scattered if/break logic. The pipeline is always: `FailureEnvelope` → `RecoveryDecision` → `StrategyOutcome` feedback. +- **Side-Effect Gating**: Recovery actions are gated by `SideEffectState`. Committed or partial side effects block automatic retry; only user-mediated or checkpoint-based resumption is permitted after state mutation. +- **Budget-Constrained Recovery**: Turn-level deadlines, per-category limits, and a global recovery budget prevent infinite retry loops. Every recovery action has an explicit cost; exhaustion triggers a clean halt or user escalation. +- **Recovery Strategy as Protocol**: Recovery strategies implement a `RecoveryStrategy` Protocol (`can_apply` + `decide`), registered by priority, composable, and extensible without modifying the coordinator. ## Path Tree, Configuration, and Secrets Rules @@ -81,6 +86,9 @@ This document is the LeapFlow engineering collaboration contract. It is not only - Every module must be importable standalone without side effects - No placeholder stubs — implement fully or do not add the code - ANSI output must check `sys.stdout.isatty()` before emitting escape codes +- For error recovery, route all failures through the `RecoveryCoordinator` — classify into a `FailureEnvelope`, receive a `RecoveryDecision` with an explainable `reason` and `strategy_key`, then feed the outcome back. Never handle errors with ad-hoc if/break in the loop body. +- Recovery strategies are standalone Protocol implementations with `can_apply()` + `decide()`. Add new strategies by registration, never by modifying the coordinator's decision logic. +- When automatic recovery exhausts its budget or encounters non-recoverable failures, emit a structured `InteractionRequest` (typed action, severity, suggested actions, timeout behavior, resumption key) — not raw text appended to conversation. ## Review Requirements @@ -103,6 +111,8 @@ This document is the LeapFlow engineering collaboration contract. It is not only - **Behavior contracts over snapshots**: assert invariants, not frozen values - **Mock at boundaries only**: mock external I/O (network, disk), never internal logic - **Change-scoped validation**: Run the most specific relevant tests first, then broaden only as needed: CLI/TUI changes require CLI/TUI tests; leapd changes require daemon RPC/lifecycle tests; storage or memory changes require persistence tests; gateway, IM, event-source, or approval changes require connector lifecycle, event normalization, routing, idempotency, self-message filtering, security/approval, and failure-recovery tests; skills, learning, perception, and copilot changes require their lifecycle or pipeline tests. +- **Recovery strategy isolation**: Each `RecoveryStrategy` must be testable in isolation — verify `can_apply` predicates, `decide` outputs, and side-effect-state gating independently of the coordinator and other strategies. +- **Budget boundary tests**: Verify that recovery budgets exhaust correctly (per-category, per-turn, deadline), that exhaustion produces a deterministic halt decision, and that cost accounting is exact. ## What to Avoid @@ -118,6 +128,10 @@ This document is the LeapFlow engineering collaboration contract. It is not only - Activating IM agents on all inbound messages by default, skipping self-message filtering, or allowing cross-chat/proactive sends before Progressive Trust and approval policies explicitly permit them. - Putting third-party app business code into platform core: do not add vendor scopes, lark-cli/SDK JSON parsing, auth command construction, console-specific recovery instructions, or resource-specific branching to gateway-wide modules such as action registries, capability ledgers, approval gates, or engine recovery paths. - Shortcut-style natural-language fitting and large intent-handler taxonomies; use stable runtime gates plus capability manifests instead. Rule-based keyword/action-verb/alias matching is prohibited by default and requires explicit human second confirmation before implementation when unavoidable. +- Scattered if/break/continue recovery decisions inside the agent loop body; all error handling enters the `RecoveryCoordinator` as a `FailureEnvelope` +- Magic retry counts or unbounded retry loops without budget constraints and deadline enforcement +- Feeding unstructured error text back to the LLM without classification, recoverability assessment, or side-effect awareness +- Multiple parallel error-handling paths for the same failure domain (LLM errors in one handler, tool errors in another, security errors in a third); use a unified classification and coordination pipeline - Bare `except:` clauses — always specify the exception type - `# TODO: implement` stubs — implement or don't commit diff --git a/src/leapflow/cli/tui_app/app.py b/src/leapflow/cli/tui_app/app.py index 2c42ea8..393ccee 100644 --- a/src/leapflow/cli/tui_app/app.py +++ b/src/leapflow/cli/tui_app/app.py @@ -54,7 +54,7 @@ from prompt_toolkit.widgets import TextArea from leapflow.cli.tui_app.approval_modal import ApprovalModal, request_is_expired -from leapflow.cli.tui_app.command import TuiCommand, TuiCommandStatus +from leapflow.cli.tui_app.command import TuiCommand, TuiCommandStatus, command_key from leapflow.cli.tui_app.input import build_completer from leapflow.cli.tui_app.paste import ( PASTE_FRAGMENT_WINDOW_S, @@ -159,6 +159,9 @@ def get_nowait(self) -> TuiCommand: def qsize(self) -> int: return len(self._items) + def contains_key(self, key: str) -> bool: + return any(command.command_key == key for command in self._items) + def snapshot(self) -> list[TuiCommand]: return list(self._items) @@ -283,8 +286,15 @@ def submit_text(self, text: str) -> TuiCommand: raise ValueError("Cannot submit an empty TUI command") if self._dispatch_control_text(normalized): return TuiCommand.create(command_id=0, text=normalized).mark_done() + key = command_key(normalized) command = TuiCommand.create(command_id=self._next_command_id, text=normalized) self._next_command_id += 1 + if self._is_duplicate_command_key(key): + skipped = command.mark_skipped("duplicate queued/running command skipped") + self._console.command_card(skipped) + self._sync_task_counts() + self._invalidate() + return skipped self._pending_input.put_nowait(command) self._sync_task_counts() if self._active_command is not None or self._pending_input.qsize() > 1 or self._queue_paused: @@ -422,6 +432,10 @@ def _terminal_command( return command.mark_blocked(reason) return command.mark_failed(reason) + def _is_duplicate_command_key(self, key: str) -> bool: + active = self._active_command + return bool((active is not None and active.command_key == key) or self._pending_input.contains_key(key)) + def _dispatch_control_text(self, text: str) -> bool: handler = self._on_control if handler is None: diff --git a/src/leapflow/cli/tui_app/command.py b/src/leapflow/cli/tui_app/command.py index 4734d6c..82c852a 100644 --- a/src/leapflow/cli/tui_app/command.py +++ b/src/leapflow/cli/tui_app/command.py @@ -7,6 +7,7 @@ from __future__ import annotations +import hashlib import time from dataclasses import dataclass, replace from enum import Enum @@ -25,6 +26,11 @@ def _truncate(text: str, limit: int) -> str: return text if len(text) <= limit else text[: limit - 1] + "…" +def command_key(text: str) -> str: + """Return a stable identity key for equivalent submitted commands.""" + return hashlib.sha256(_single_line(text).encode("utf-8")).hexdigest() + + class TuiCommandStatus(str, Enum): """Lifecycle states for a TUI-submitted command.""" @@ -45,6 +51,7 @@ class TuiCommand: text: str status: TuiCommandStatus created_at: float + command_key: str started_at: float = 0.0 finished_at: float = 0.0 error: str = "" @@ -57,6 +64,7 @@ def create(cls, *, command_id: int, text: str) -> "TuiCommand": text=text, status=TuiCommandStatus.QUEUED, created_at=time.monotonic(), + command_key=command_key(text), ) @property diff --git a/src/leapflow/cli/tui_app/stream.py b/src/leapflow/cli/tui_app/stream.py index 5fe5522..7b2621a 100644 --- a/src/leapflow/cli/tui_app/stream.py +++ b/src/leapflow/cli/tui_app/stream.py @@ -269,6 +269,8 @@ def _tool_icon(name: str, *, ok: bool = True) -> str: def _tool_result_detail(metadata: dict[str, Any] | None) -> str: if not metadata: return "" + if metadata.get("already_executed") or metadata.get("duplicate_suppressed") or metadata.get("execution_skipped"): + return "" if metadata.get("ok") is False: exit_code = metadata.get("exit_code") prefix = f"exit={exit_code} " if exit_code is not None else "" @@ -412,6 +414,11 @@ def tool_finished( tool_name = _metadata_text(metadata, "normalized_tool_name") or name or self._active_tool original_tool_name = _metadata_text(metadata, "original_tool_name") alias_detail = original_tool_name if original_tool_name and original_tool_name != tool_name else "" + if tool_name and metadata.get("ui_hidden"): + self._active_tool = "" + self._active_tool_detail = "" + self._tool_start_time = 0.0 + return if tool_name and self._tool_start_time > 0: duration = time.monotonic() - self._tool_start_time self._tool_history.append((tool_name, duration)) diff --git a/src/leapflow/config.py b/src/leapflow/config.py index 7b8745e..b3f5efa 100644 --- a/src/leapflow/config.py +++ b/src/leapflow/config.py @@ -349,6 +349,8 @@ class Settings: # ── Stream & Tool Robustness ── stale_stream_timeout_s: float = 180.0 # Idle timeout for streaming responses default_tool_timeout_s: float = 120.0 # Default per-tool execution timeout + daemon_request_ledger_ttl_s: float = 600.0 # Replay cache retention for completed engine requests + daemon_request_ledger_max_entries: int = 128 # Maximum completed engine requests kept for replay circuit_breaker_threshold: int = 5 # Consecutive failures before circuit opens circuit_breaker_cooldown_s: float = 60.0 # Circuit breaker cooldown period @@ -756,6 +758,8 @@ def _build_settings_from_env( # Stream & Tool Robustness stale_stream_timeout_s = float(os.getenv("LEAPFLOW_STALE_STREAM_TIMEOUT_S", "180.0")) default_tool_timeout_s = float(os.getenv("LEAPFLOW_DEFAULT_TOOL_TIMEOUT_S", "120.0")) + daemon_request_ledger_ttl_s = float(os.getenv("LEAPFLOW_DAEMON_REQUEST_LEDGER_TTL_S", "600.0")) + daemon_request_ledger_max_entries = int(os.getenv("LEAPFLOW_DAEMON_REQUEST_LEDGER_MAX_ENTRIES", "128")) circuit_breaker_threshold = int(os.getenv("LEAPFLOW_CIRCUIT_BREAKER_THRESHOLD", "5")) circuit_breaker_cooldown_s = float(os.getenv("LEAPFLOW_CIRCUIT_BREAKER_COOLDOWN_S", "60.0")) @@ -1018,6 +1022,8 @@ def _build_settings_from_env( # Stream & Tool Robustness stale_stream_timeout_s=stale_stream_timeout_s, default_tool_timeout_s=default_tool_timeout_s, + daemon_request_ledger_ttl_s=daemon_request_ledger_ttl_s, + daemon_request_ledger_max_entries=daemon_request_ledger_max_entries, circuit_breaker_threshold=circuit_breaker_threshold, circuit_breaker_cooldown_s=circuit_breaker_cooldown_s, # Signal Fusion diff --git a/src/leapflow/daemon/server.py b/src/leapflow/daemon/server.py index 99938ea..648b7a7 100644 --- a/src/leapflow/daemon/server.py +++ b/src/leapflow/daemon/server.py @@ -156,6 +156,8 @@ async def _dispatch(self, request: RpcRequest, writer: asyncio.StreamWriter) -> method = getattr(self._service, attr) params = dict(request.params or {}) + if request.method == "engine.chat": + params.setdefault("request_id", request.id) if request.method in ("engine.chat", "events.subscribe"): await self._dispatch_stream(request, method, params, writer) return diff --git a/src/leapflow/daemon/service.py b/src/leapflow/daemon/service.py index 15f10fa..a7c6521 100644 --- a/src/leapflow/daemon/service.py +++ b/src/leapflow/daemon/service.py @@ -2,6 +2,7 @@ from __future__ import annotations import asyncio +import inspect import logging import os import sys @@ -32,6 +33,10 @@ def __init__(self, settings: Any, *, mock_host: bool = False) -> None: self._client_leases: Callable[[], list[ClientLeaseSnapshot]] = lambda: [] self._approval_pending: dict[str, dict[str, Any]] = {} self._approval_event_queue: asyncio.Queue[StreamChunk] | None = None + self._active_engine_request_id: str = "" + self._engine_request_ledger: dict[str, dict[str, Any]] = {} + self._request_ledger_ttl_s = max(1.0, float(getattr(settings, "daemon_request_ledger_ttl_s", 600.0) or 600.0)) + self._request_ledger_max_entries = max(1, int(getattr(settings, "daemon_request_ledger_max_entries", 128) or 128)) from leapflow.daemon.notifications import NotificationBus self.notification_bus = NotificationBus() @@ -95,33 +100,93 @@ async def session_resume(self, session_id: str) -> dict[str, Any]: return {"found": found, "session_id": str(current or session_id)} async def engine_chat(self, message: str, **kwargs: Any) -> AsyncIterator[StreamChunk]: + request_id = str(kwargs.get("request_id") or uuid.uuid4().hex[:12]) async with self._engine_lock: - ctx = self.context - if ctx.reload_runtime_config_if_changed(): - self._settings = ctx.settings + self._prune_engine_request_ledger() + existing = self._engine_request_ledger.get(request_id) + if existing and existing.get("status") == "completed": + for chunk in existing.get("chunks", []): + if isinstance(chunk, StreamChunk): + metadata = dict(chunk.metadata or {}) + metadata["replayed_request"] = True + yield StreamChunk( + request_id=request_id, + content=chunk.content, + done=chunk.done, + event_type=chunk.event_type, + metadata=metadata, + ) + return + if existing and existing.get("status") == "running": yield StreamChunk( - request_id="", - content="Configuration reloaded in leapd.", + request_id=request_id, + content="Duplicate engine request is already running.", event_type="status", - metadata={ - **self._engine_context_metadata(getattr(ctx, "engine", None), ctx.settings), - "llm_model": getattr(ctx.settings, "llm_model", ""), - }, + metadata={"request_id": request_id, "duplicate_request": True}, ) - engine = getattr(ctx, "engine", None) - if engine is None: - raise RuntimeError("leapd engine is not initialized") - - enable_thinking = bool(kwargs.get("enable_thinking", False)) - approval_queue: asyncio.Queue[StreamChunk] = asyncio.Queue() - previous_queue = self._approval_event_queue - self._approval_event_queue = approval_queue + return + + request_record: dict[str, Any] = { + "status": "running", + "chunks": [], + "created_at": time.time(), + } + self._engine_request_ledger[request_id] = request_record + ctx = self.context try: - stream = engine.run_stream(message, enable_thinking=enable_thinking) - async for chunk in self._stream_engine_events(stream, approval_queue): + if ctx.reload_runtime_config_if_changed(): + self._settings = ctx.settings + chunk = StreamChunk( + request_id=request_id, + content="Configuration reloaded in leapd.", + event_type="status", + metadata={ + **self._engine_context_metadata(getattr(ctx, "engine", None), ctx.settings), + "llm_model": getattr(ctx.settings, "llm_model", ""), + "request_id": request_id, + }, + ) + request_record["chunks"].append(chunk) yield chunk - finally: - self._approval_event_queue = previous_queue + engine = getattr(ctx, "engine", None) + if engine is None: + raise RuntimeError("leapd engine is not initialized") + + enable_thinking = bool(kwargs.get("enable_thinking", False)) + approval_queue: asyncio.Queue[StreamChunk] = asyncio.Queue() + previous_queue = self._approval_event_queue + previous_request_id = self._active_engine_request_id + self._approval_event_queue = approval_queue + self._active_engine_request_id = request_id + try: + sig = inspect.signature(engine.run_stream) + if "request_id" in sig.parameters: + stream = engine.run_stream( + message, + enable_thinking=enable_thinking, + request_id=request_id, + ) + else: + stream = engine.run_stream( + message, + enable_thinking=enable_thinking, + ) + async for chunk in self._stream_engine_events( + stream, approval_queue, request_id=request_id, + ): + request_record["chunks"].append(chunk) + yield chunk + request_record["status"] = "completed" + request_record["completed_at"] = time.time() + self._prune_engine_request_ledger() + finally: + self._approval_event_queue = previous_queue + self._active_engine_request_id = previous_request_id + except Exception: + request_record["status"] = "failed" + request_record["completed_at"] = time.time() + self._prune_engine_request_ledger() + raise async def engine_cancel(self) -> bool: ctx = self.context @@ -376,6 +441,30 @@ async def gateway_send( ) -> dict[str, Any]: raise NotImplementedError("gateway.send is not available in this daemon phase") + def _prune_engine_request_ledger(self) -> None: + """Bound completed/failed engine request replay records by TTL and size.""" + now = time.time() + for request_id, record in list(self._engine_request_ledger.items()): + status = str(record.get("status") or "") + if status == "running": + continue + completed_at = float(record.get("completed_at") or record.get("created_at") or 0.0) + if now - completed_at > self._request_ledger_ttl_s: + self._engine_request_ledger.pop(request_id, None) + overflow = len(self._engine_request_ledger) - self._request_ledger_max_entries + if overflow <= 0: + return + evictable = sorted( + ( + (float(record.get("completed_at") or record.get("created_at") or 0.0), request_id) + for request_id, record in self._engine_request_ledger.items() + if str(record.get("status") or "") != "running" + ), + key=lambda item: item[0], + ) + for _timestamp, request_id in evictable[:overflow]: + self._engine_request_ledger.pop(request_id, None) + def _memory_entry_to_dict(self, entry: MemoryEntry) -> dict[str, Any]: return { "entry_id": entry.entry_id, @@ -519,6 +608,8 @@ async def _stream_engine_events( self, stream: AsyncIterator[object], approval_queue: asyncio.Queue[StreamChunk], + *, + request_id: str = "", ) -> AsyncIterator[StreamChunk]: engine_task: asyncio.Task[Any] | None = asyncio.create_task(anext(stream)) approval_task: asyncio.Task[StreamChunk] | None = asyncio.create_task(approval_queue.get()) @@ -537,7 +628,7 @@ async def _stream_engine_events( engine_task = None break stream_event = self._normalize_event(event) - yield self._chunk_from_event(stream_event) + yield self._chunk_from_event(stream_event, request_id=request_id) engine_task = asyncio.create_task(anext(stream)) finally: for task in (engine_task, approval_task): @@ -558,9 +649,10 @@ async def _request_approval(self, request: Any) -> str: if queue is None: return "deny" pending_id = str(getattr(request, "request_id", "") or uuid.uuid4().hex) + request_id = self._active_engine_request_id or pending_id payload = request.to_dict() payload["pending_id"] = pending_id - payload.setdefault("request_id", pending_id) + payload["request_id"] = request_id future: asyncio.Future[dict[str, Any]] = asyncio.get_running_loop().create_future() self._approval_pending[pending_id] = { "request": payload, @@ -569,10 +661,10 @@ async def _request_approval(self, request: Any) -> str: "created_at": time.time(), } await queue.put(StreamChunk( - request_id="", + request_id=request_id, content="Approval required", event_type="approval_request", - metadata={"approval": payload}, + metadata={"approval": payload, "request_id": request_id}, )) timeout_s = 120.0 if getattr(request, "expires_at", None): @@ -618,17 +710,19 @@ def _normalize_event(self, event: object) -> StreamEvent: return event return StreamEvent(type="chunk", content=str(event), metadata=None) - def _chunk_from_event(self, event: StreamEvent) -> StreamChunk: + def _chunk_from_event(self, event: StreamEvent, *, request_id: str = "") -> StreamChunk: ctx = self.context engine = getattr(ctx, "engine", None) metadata = dict(event.metadata or {}) session_id = getattr(engine, "_current_session_id", "") if engine else "" + if request_id: + metadata.setdefault("request_id", request_id) if session_id: metadata.setdefault("session_id", str(session_id)) if engine is not None: metadata.update(self._engine_context_metadata(engine, getattr(ctx, "settings", self._settings))) return StreamChunk( - request_id="", + request_id=request_id, content=event.content, done=False, event_type=event.type, diff --git a/src/leapflow/engine/context_control.py b/src/leapflow/engine/context_control.py index 8debe7e..e47a9d2 100644 --- a/src/leapflow/engine/context_control.py +++ b/src/leapflow/engine/context_control.py @@ -356,7 +356,6 @@ def _platform_action_evidence(self, arguments: Dict[str, Any], result: Dict[str, evidence["execution_note"] = str(result.get("execution_note") or "Done. Do not repeat.") if result.get("already_executed"): evidence["status"] = "ALREADY_EXECUTED" - evidence["execution_note"] = str(result.get("execution_note") or "") original = result.get("original_result") if isinstance(original, dict) and original.get("resource_id"): evidence["resource_id"] = str(original["resource_id"]) diff --git a/src/leapflow/engine/engine.py b/src/leapflow/engine/engine.py index 23b8c5e..7c22f9c 100644 --- a/src/leapflow/engine/engine.py +++ b/src/leapflow/engine/engine.py @@ -46,11 +46,20 @@ from leapflow.engine.stale_stream import StaleStreamError, stale_guarded_stream, build_continuation_prompt from leapflow.engine.turn_recovery import TurnRecoveryState from leapflow.engine.turn_usage import TurnUsageTracker +from leapflow.engine.recovery_coordinator import RecoveryCoordinator, RecoveryState +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.unified_classifier import UnifiedErrorClassifier +from leapflow.engine.failure_envelope import FailureEnvelope, FailureSource, Recoverability +from leapflow.engine.recovery_decision import RecoveryAction +from leapflow.engine.recovery_strategies import default_strategies +from leapflow.engine.recovery_audit import JsonlAuditSink, create_audit_entry +from leapflow.engine.recovery_checkpoint import RecoveryCheckpoint, InMemoryCheckpointStore from leapflow.engine.tool_concurrency import ( DefaultConcurrencyPolicy, ToolCall as ConcurrentToolCall, ToolConcurrencyPolicy, ) +from leapflow.engine.tool_execution import ToolExecutionLedger, execution_policy_for from leapflow.engine.graph_planner import GraphPlanner from leapflow.engine.scheduler import TaskScheduler from leapflow.engine.session import SessionController, SessionMode @@ -174,7 +183,13 @@ def _tool_result_metadata( for key in ("exit_code", "path", "lines", "truncated", "bytes_written"): if key in result: metadata[key] = result[key] - for key in ("error_type", "retryable", "resolution_status", "resolution_confidence"): + for key in ( + "error_type", "retryable", "resolution_status", "resolution_confidence", + "already_executed", "duplicate_suppressed", "execution_reused", "execution_skipped", + "counts_as_failure", "counts_as_tool_attempt", "ui_hidden", "skipped_reason", + "blocked_by_tool", "blocked_by_error", "execution_id", "idempotency_key", "execution_status", + "execution_policy", "tool_call_id", + ): if key in result: metadata[key] = result[key] # App Connector authorization failure metadata @@ -224,22 +239,6 @@ def _is_retryable_unknown_tool_result(result: Any) -> bool: ) -def _platform_action_fingerprint(tool_name: str, arguments: Mapping[str, Any]) -> str | None: - """Return a dedup fingerprint for platform_action calls; None for all other tools. - - Used to prevent duplicate side-effect actions (send, write) within one turn. - Two calls are considered duplicates when platform, action name, and payload - are all identical. - """ - if tool_name != "platform_action": - return None - platform = str(arguments.get("platform") or "") - action = str(arguments.get("action") or "") - payload = arguments.get("payload") or {} - payload_key = json.dumps(payload, sort_keys=True, ensure_ascii=False) - return f"{platform}:{action}:{payload_key}" - - def _has_completed_side_effect(results: List[Dict[str, Any]]) -> bool: """Return True if any result is a completed side-effect platform_action.""" for item in results: @@ -279,6 +278,68 @@ def _is_permission_hard_stop_payload(payload: Dict[str, Any]) -> bool: return is_permission_hard_stop_payload(payload) +_SIDE_EFFECT_STOP_POLICIES = frozenset({"external_side_effect", "mutating_once", "mutating_idempotent"}) +_SIDE_EFFECT_STOP_TOOLS = frozenset({ + "shell_run", + "scm_sync", + "platform_action", + "platform_connect", + "gateway_send", + "gateway_connect", + "hub_push", + "hub_pull", + "hub_sync", +}) + + +def _tool_result_counts_as_failure(payload: Dict[str, Any]) -> bool: + """Return whether a tool payload represents a real failed execution attempt.""" + if payload.get("counts_as_failure") is False: + return False + if _tool_result_is_control_signal(payload): + return False + return payload.get("ok") is False + + +def _tool_result_is_control_signal(payload: Dict[str, Any]) -> bool: + """Return whether a tool payload is execution control metadata, not an attempt result.""" + return bool(payload.get("already_executed") or payload.get("duplicate_suppressed") or payload.get("execution_skipped")) + + +def _tool_failure_text(payload: Dict[str, Any]) -> str: + """Return the most useful root-cause text from a failed tool payload.""" + for key in ("error", "stderr", "stdout", "message"): + value = payload.get(key) + if value: + return str(value) + return "unknown error" + + +def _should_stop_after_tool_result(tool_name: str, payload: Dict[str, Any]) -> bool: + """Return whether a failed side-effect result must stop the current tool batch.""" + if _is_permission_hard_stop_payload(payload): + return True + if not _tool_result_counts_as_failure(payload): + return False + policy = str(payload.get("execution_policy") or "") + normalized_name = str(tool_name or "").removeprefix("gp_") + return policy in _SIDE_EFFECT_STOP_POLICIES or normalized_name in _SIDE_EFFECT_STOP_TOOLS + + +def _skipped_after_failure_result(blocking_tool: str, blocking_result: Dict[str, Any]) -> Dict[str, Any]: + """Build a non-failure result for a tool skipped because an earlier side effect failed.""" + return { + "ok": True, + "execution_skipped": True, + "skipped_reason": "previous_tool_failed", + "blocked_by_tool": blocking_tool, + "blocked_by_error": _tool_failure_text(blocking_result), + "counts_as_failure": False, + "counts_as_tool_attempt": False, + "ui_hidden": True, + } + + def _permission_hard_stop_from_results(results: List[Dict[str, Any]]) -> Dict[str, Any] | None: """Return the first hard-stop permission failure from native tool results.""" for item in results: @@ -353,7 +414,7 @@ def _extract_recent_tool_failures(messages: List[Dict[str, Any]]) -> List[Dict[s payload = json.loads(content) except (json.JSONDecodeError, ValueError): continue - if not isinstance(payload, dict) or payload.get("ok", True): + if not isinstance(payload, dict) or not _tool_result_counts_as_failure(payload): continue failures.append(payload) if len(failures) >= 3: @@ -423,7 +484,7 @@ def _last_tool_failures_recovery_message(messages: List[Dict[str, Any]]) -> str: last = failures[0] failure_code = str(last.get("failure_code") or "") - error = str(last.get("error") or "") + error = str(last.get("error") or last.get("stderr") or last.get("stdout") or "") recovery_hint = str(last.get("recovery_hint") or "") available_actions: List[str] = list(last.get("available_action_names") or []) @@ -783,6 +844,11 @@ def __init__( # Session persistence (injected by CLI) self._conversation_store: Optional[Any] = None self._current_session_id: Optional[str] = None + self._current_turn_id: str = "" + self._current_command_id: str = "" + self._tool_execution_ledger = ToolExecutionLedger() + + self._current_request_id: str = "" # Memory context snapshot (frozen at session start for prefix cache stability) self._memory_context_snapshot: Optional[str] = None @@ -883,6 +949,12 @@ def __init__( # B4: Output sanitization (None = disabled) self._sanitizer: MessageSanitizer | None = None + # Recovery coordinator infrastructure + self._unified_classifier = UnifiedErrorClassifier(self._error_classifier) + self._recovery_coordinator = RecoveryCoordinator() # Re-created per turn + self._checkpoint_store = InMemoryCheckpointStore() + self._audit_sink = JsonlAuditSink() # In-memory; path-based if layout available + # ── Optional strategy setters (config-driven) ──────────────────────── def set_cache_strategy(self, strategy: CacheStrategy | None) -> None: @@ -996,14 +1068,11 @@ async def _handle_api_error( use_native_tools: bool = False, tools_kwarg: Optional[Dict[str, Any]] = None, ) -> Optional[str]: - """Unified API error recovery dispatcher. Returns 'continue' to retry, else None. - - Wires ALL TurnRecoveryState one-shot guards to their matching ErrorCategory: - - CONTEXT_OVERFLOW → try_compress - - IMAGE_TOO_LARGE → try_multimodal_strip - - should_fallback → try_provider_failover - - should_rotate_credential → try_credential_rotate - - FORMAT_ERROR with thinking → try_disable_thinking + """Legacy API error recovery dispatcher. Returns 'continue' to retry, else None. + + DEPRECATED: No longer called from main loops. Retained only for backward + compat with any external subclass overrides. All recovery now flows through + RecoveryCoordinator.evaluate(). """ if classified == ErrorCategory.CONTEXT_OVERFLOW and recovery.try_compress(): messages[:] = self._compressor.force_compress(messages) @@ -1091,6 +1160,46 @@ def _strip_images_from_messages(messages: list) -> None: else: messages[i] = {**msg, "content": "[images removed to reduce context]"} + def _execute_transform_decision( + self, + decision: "RecoveryDecision", + messages: list, + ) -> bool: + """Execute a TRANSFORM_AND_RETRY decision. Returns True if transform succeeded. + + Handles different transform strategies: + - context_compress: force-compress conversation history + - multimodal_strip: remove image content from messages + - native_to_text: disable native tool calling + - thinking_disable: disable thinking mode (handled externally) + """ + strategy_key = decision.strategy_key + # Determine specific phase from audit_metadata if available + phase = dict(decision.audit_metadata).get("phase", "") + + if strategy_key == "context_compress": + if phase == "multimodal_to_text": + self._strip_images_from_messages(messages) + else: + # Default: history_summarize and disclosure_shrink both use force_compress + messages[:] = self._compressor.force_compress(messages) + return True + + if strategy_key == "multimodal_strip": + self._strip_images_from_messages(messages) + return True + + if strategy_key == "native_to_text": + # Handled by caller via tools_kwarg mutation + return True + + if strategy_key == "thinking_disable": + # Handled by caller via enable_thinking flag + return True + + logger.warning("Unknown transform strategy: %s", strategy_key) + return True + def _check_guardrail( self, messages: List[Dict[str, Any]], @@ -1171,6 +1280,7 @@ def set_experience_store(self, store: Any) -> None: def set_conversation_store(self, store: Any) -> None: """Inject conversation persistence store.""" self._conversation_store = store + self._tool_execution_ledger.reset(store=store) def load_session(self, session_id: str) -> bool: """Resume a previous session by loading messages from DuckDB. @@ -1253,6 +1363,9 @@ def _begin_turn_context(self, user_text: str) -> None: self._last_disclosure_metadata = {} self._context_governance_controller.reset_turn_scope() self._current_task_contract = self._build_task_contract(user_text) + self._current_turn_id = self._current_task_contract.task_id + self._current_command_id = self._current_task_contract.task_id + self._tool_execution_ledger.reset(store=self._conversation_store) try: from leapflow.tools.gateway_tool import reset_platform_action_scope reset_platform_action_scope() @@ -1684,7 +1797,7 @@ async def run(self, user_text: str, *, enable_thinking: bool = False) -> str: return await self._unified_tool_loop(user_text, enable_thinking=enable_thinking) async def run_stream( - self, user_text: str, *, enable_thinking: bool = False + self, user_text: str, *, enable_thinking: bool = False, request_id: str = "" ) -> AsyncIterator[Union[str, StreamEvent]]: """Like run(), but yields text chunks for streamable responses. @@ -1695,6 +1808,7 @@ async def run_stream( StreamEvent(type="tool_call"): internal tool invocation (suppress display). """ self._session_turn_count += 1 + self._current_request_id = request_id logger.info("audit.user_input chars=%s", len(user_text)) self._begin_turn_context(user_text) self._emit_chat_event("user_message", {"content": user_text[:500]}) @@ -2153,6 +2267,14 @@ async def _unified_tool_loop( content = "" fatal_error: Optional[str] = None recovery = TurnRecoveryState() + # P3: Initialize recovery coordinator for this turn + recovery_budget = RecoveryBudget() + recovery_budget.start_deadline() + self._recovery_coordinator = RecoveryCoordinator( + strategies=default_strategies(), + budget=recovery_budget, + ) + self._recovery_coordinator.new_turn(turn_id=budget.used) use_native_tools = assembly.plan.native_tools result_budget = self._effective_tool_result_budget() unknown_tool_retry_used = False @@ -2205,21 +2327,84 @@ async def _unified_tool_loop( category_str = classified.value if hasattr(classified, 'value') else str(classified) recovery.record_api_error(category_str) - if await self._handle_api_error( - classified, rec, recovery, messages, budget, - use_native_tools=use_native_tools, tools_kwarg=tools_kwarg, - ) == "continue": - if classified == ErrorCategory.CONTEXT_OVERFLOW: + # Classify through unified coordinator and execute recovery + envelope = self._unified_classifier.classify_llm_error( + exc, provider=getattr(self._llm, 'provider', ''), + model=getattr(self._llm, 'model', ''), + ) + coordinator = self._recovery_coordinator + try: + decision = coordinator.evaluate(envelope) + except Exception as coord_exc: + logger.error("recovery_coordinator.evaluate() failed: %s", coord_exc) + fatal_error = f"Internal recovery error: {coord_exc}" + break + self._audit_sink.record(create_audit_entry( + envelope, decision, coordinator.budget, + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + )) + + # Execute decision via coordinator + if decision.action == RecoveryAction.RETRY_WITH_BACKOFF: + if decision.retry_semantics.backoff_config: + await asyncio.sleep( + jittered_backoff(budget.used, base=decision.retry_semantics.backoff_config.base_delay) + ) + continue + + elif decision.action == RecoveryAction.TRANSFORM_AND_RETRY: + # Handle native_to_text locally (needs local var mutation) + if decision.strategy_key == "native_to_text": + tools_kwarg = {} + use_native_tools = False + transform_ok = True + else: + transform_ok = self._execute_transform_decision(decision, messages) + if transform_ok: self._usage_tracker.mark_compression() + coordinator.on_strategy_outcome(decision.decision_id, transform_ok) + if not transform_ok: + fatal_error = f"Transform failed: {decision.reason}" + break continue - if classified in (ErrorCategory.FORMAT_ERROR,) and tools_kwarg and recovery.try_native_fallback(): - logger.info("Native tool calling failed, falling back to text mode") - tools_kwarg = {} - use_native_tools = False + + elif decision.action == RecoveryAction.FAILOVER: + if hasattr(self._llm, '_failover'): + self._llm._failover(f"recovery: {decision.reason}") + coordinator.on_strategy_outcome(decision.decision_id, True) continue - fatal_error = self._error_classifier.friendly_message(classified, str(exc)) - logger.error("unified_loop: unrecoverable %s: %s", category_str, exc) - break + + elif decision.action in (RecoveryAction.HALT_CLEAN, RecoveryAction.HALT_WITH_CHECKPOINT): + if decision.action == RecoveryAction.HALT_WITH_CHECKPOINT: + checkpoint = RecoveryCheckpoint( + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + failure_envelope_data={ + "message": envelope.message, + "category": envelope.category, + "failure_code": envelope.failure_code, + "source": envelope.source.value, + }, + messages_snapshot=list(messages), + context_data={ + "tools_kwarg_keys": list(tools_kwarg.keys()), + "use_native_tools": use_native_tools, + "budget_used": budget.used, + }, + ) + self._checkpoint_store.save(checkpoint) + fatal_error = decision.reason + self._audit_sink.update_outcome( + decision.decision_id, "failure", + reason="Terminal halt", + ) + break + + else: + # ASK_USER, SKIP_AND_CONTINUE, or unknown + fatal_error = decision.reason + break _clear_indicator() content = (resp.content or "").strip() @@ -2328,7 +2513,9 @@ async def _unified_tool_loop( "arguments_summary": json.dumps(tool_arguments, default=str, ensure_ascii=False)[:300] if tool_arguments else "", }) _show_progress("executing", tool_name) - result = await self._execute_general_tool(normalized_tool_call, TOOL_HANDLERS) + result = await self._execute_tool_with_ledger( + normalized_tool_call, TOOL_HANDLERS, tool_call_id=f"text-{budget.used}", + ) _clear_indicator() self._emit_chat_event("tool_result", { "tool_name": tool_name, @@ -2342,7 +2529,7 @@ async def _unified_tool_loop( observation=result if isinstance(result, dict) else {"result": str(result)}, ) - is_error = isinstance(result, dict) and not result.get("ok", True) + is_error = isinstance(result, dict) and _tool_result_counts_as_failure(result) if is_error: recovery.record_tool_failure() else: @@ -2352,7 +2539,11 @@ async def _unified_tool_loop( messages.append(build_user_message_text( f"Tool result ({tool_name}):\n{result_text}" )) - self._persist_message(session_id, "tool", result_text, tool_name=tool_name) + self._persist_message( + session_id, "tool", result_text, + tool_name=tool_name, tool_call_id=f"text-{budget.used}", + metadata=self._tool_execution_metadata(result), + ) if _is_permission_hard_stop_payload(result): logger.info( @@ -2362,6 +2553,28 @@ async def _unified_tool_loop( ) break + # Classify tool failures through coordinator for audit and decision + if ( + isinstance(result, dict) + and not result.get("ok", True) + and result.get("counts_as_failure") is not False + ): + tool_envelope = self._unified_classifier.classify_tool_result( + result, tool_name=tool_name, + execution_policy=result.get("execution_policy", "read_only"), + ) + if tool_envelope is not None: + tool_decision = self._recovery_coordinator.evaluate(tool_envelope) + self._audit_sink.record(create_audit_entry( + tool_envelope, tool_decision, self._recovery_coordinator.budget, + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + )) + if tool_decision.action in (RecoveryAction.HALT_CLEAN, RecoveryAction.HALT_WITH_CHECKPOINT): + fatal_error = tool_decision.reason + break + # Other actions: let LLM handle (append result to messages as before) + if _is_retryable_unknown_tool_result(result) and not unknown_tool_retry_used: unknown_tool_retry_used = True messages.append(build_user_message_text(_unknown_tool_retry_prompt(result))) @@ -2564,6 +2777,14 @@ async def _unified_tool_loop_stream( content = "" fatal_error: Optional[str] = None turn_recovery = TurnRecoveryState() + # Initialize recovery coordinator for stream turn + recovery_budget = RecoveryBudget() + recovery_budget.start_deadline() + self._recovery_coordinator = RecoveryCoordinator( + strategies=default_strategies(), + budget=recovery_budget, + ) + self._recovery_coordinator.new_turn(turn_id=budget.used) use_native_tools = assembly.plan.native_tools result_budget = self._effective_tool_result_budget() unknown_tool_retry_used = False @@ -2618,18 +2839,48 @@ async def _unified_tool_loop_stream( rec = self._error_classifier.get_recovery(classified) turn_recovery.record_api_error() - if await self._handle_api_error( - classified, rec, turn_recovery, messages, budget, - use_native_tools=use_native_tools, tools_kwarg=tools_kwarg, - ) == "continue": + # Classify through unified coordinator + envelope = self._unified_classifier.classify_llm_error( + exc, provider=getattr(self._llm, 'provider', ''), + model=getattr(self._llm, 'model', ''), + ) + coordinator = self._recovery_coordinator + try: + decision = coordinator.evaluate(envelope) + except Exception as coord_exc: + logger.error("recovery_coordinator.evaluate() failed: %s", coord_exc) + yield StreamEvent(type="error", content=f"Internal recovery error: {coord_exc}") + break + self._audit_sink.record(create_audit_entry( + envelope, decision, coordinator.budget, + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + )) + + if decision.action == RecoveryAction.RETRY_WITH_BACKOFF: + if decision.retry_semantics.backoff_config: + await asyncio.sleep( + jittered_backoff(budget.used, base=decision.retry_semantics.backoff_config.base_delay) + ) continue - if tools_kwarg and turn_recovery.try_native_fallback(): - logger.info("Native tool calling failed, falling back to text mode") - tools_kwarg = {} - use_native_tools = False + elif decision.action == RecoveryAction.TRANSFORM_AND_RETRY: + if decision.strategy_key == "native_to_text": + tools_kwarg = {} + use_native_tools = False + else: + self._execute_transform_decision(decision, messages) + coordinator.on_strategy_outcome(decision.decision_id, True) continue - yield StreamEvent(type="error", content=str(exc)) - break + elif decision.action == RecoveryAction.FAILOVER: + if hasattr(self._llm, '_failover'): + self._llm._failover(f"recovery: {decision.reason}") + coordinator.on_strategy_outcome(decision.decision_id, True) + continue + else: + # Terminal: HALT_CLEAN, HALT_WITH_CHECKPOINT, ASK_USER + fatal_error = decision.reason + yield StreamEvent(type="error", content=decision.reason) + break _clear_indicator() content = (resp.content or "").strip() @@ -2780,14 +3031,43 @@ async def _unified_tool_loop_stream( classified = self._error_classifier.classify(exc) rec = self._error_classifier.get_recovery(classified) turn_recovery.record_api_error() - if await self._handle_api_error( - classified, rec, turn_recovery, messages, budget, - ) == "continue": + # Classify through unified coordinator + envelope = self._unified_classifier.classify_llm_error( + exc, provider=getattr(self._llm, 'provider', ''), + model=getattr(self._llm, 'model', ''), + ) + coordinator = self._recovery_coordinator + try: + decision = coordinator.evaluate(envelope) + except Exception as coord_exc: + logger.error("recovery_coordinator.evaluate() failed: %s", coord_exc) + yield StreamEvent(type="error", content=f"Internal recovery error: {coord_exc}") + break + self._audit_sink.record(create_audit_entry( + envelope, decision, coordinator.budget, + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + )) + if decision.action == RecoveryAction.RETRY_WITH_BACKOFF: + if decision.retry_semantics.backoff_config: + await asyncio.sleep( + jittered_backoff(budget.used, base=decision.retry_semantics.backoff_config.base_delay) + ) continue - fatal_error = self._error_classifier.friendly_message(classified, str(exc)) - logger.error("unified_loop_stream: unrecoverable %s: %s", classified.value, exc) - yield StreamEvent(type="error", content=fatal_error) - break + elif decision.action == RecoveryAction.TRANSFORM_AND_RETRY: + self._execute_transform_decision(decision, messages) + coordinator.on_strategy_outcome(decision.decision_id, True) + continue + elif decision.action == RecoveryAction.FAILOVER: + if hasattr(self._llm, '_failover'): + self._llm._failover(f"recovery: {decision.reason}") + coordinator.on_strategy_outcome(decision.decision_id, True) + continue + else: + fatal_error = decision.reason + logger.error("unified_loop_stream: unrecoverable %s: %s", envelope.category, exc) + yield StreamEvent(type="error", content=decision.reason) + break content = "".join(content_parts).strip() if self._sanitizer: @@ -2812,14 +3092,43 @@ async def _unified_tool_loop_stream( classified = self._error_classifier.classify(exc) rec = self._error_classifier.get_recovery(classified) turn_recovery.record_api_error() - if await self._handle_api_error( - classified, rec, turn_recovery, messages, budget, - ) == "continue": + # Classify through unified coordinator + envelope = self._unified_classifier.classify_llm_error( + exc, provider=getattr(self._llm, 'provider', ''), + model=getattr(self._llm, 'model', ''), + ) + coordinator = self._recovery_coordinator + try: + decision = coordinator.evaluate(envelope) + except Exception as coord_exc: + logger.error("recovery_coordinator.evaluate() failed: %s", coord_exc) + yield StreamEvent(type="error", content=f"Internal recovery error: {coord_exc}") + break + self._audit_sink.record(create_audit_entry( + envelope, decision, coordinator.budget, + session_id=getattr(self, '_current_session_id', '') or '', + turn_id=budget.used, + )) + if decision.action == RecoveryAction.RETRY_WITH_BACKOFF: + if decision.retry_semantics.backoff_config: + await asyncio.sleep( + jittered_backoff(budget.used, base=decision.retry_semantics.backoff_config.base_delay) + ) continue - fatal_error = self._error_classifier.friendly_message(classified, str(exc)) - logger.error("unified_loop_stream: unrecoverable %s: %s", classified.value, exc) - yield StreamEvent(type="error", content=fatal_error) - break + elif decision.action == RecoveryAction.TRANSFORM_AND_RETRY: + self._execute_transform_decision(decision, messages) + coordinator.on_strategy_outcome(decision.decision_id, True) + continue + elif decision.action == RecoveryAction.FAILOVER: + if hasattr(self._llm, '_failover'): + self._llm._failover(f"recovery: {decision.reason}") + coordinator.on_strategy_outcome(decision.decision_id, True) + continue + else: + fatal_error = decision.reason + logger.error("unified_loop_stream: unrecoverable %s: %s", envelope.category, exc) + yield StreamEvent(type="error", content=decision.reason) + break _clear_indicator() content = (resp.content or "").strip() if self._sanitizer: @@ -2870,7 +3179,9 @@ async def _unified_tool_loop_stream( original_tool_name=original_tool_name, ), ) - result = await self._execute_general_tool(normalized_tool_call, TOOL_HANDLERS) + result = await self._execute_tool_with_ledger( + normalized_tool_call, TOOL_HANDLERS, tool_call_id=f"text-{budget.used}", + ) _clear_indicator() self._emit_chat_event("tool_result", { "tool_name": tool_name, @@ -2901,7 +3212,7 @@ async def _unified_tool_loop_stream( observation=result if isinstance(result, dict) else {"result": str(result)}, ) - is_error = isinstance(result, dict) and not result.get("ok", True) + is_error = isinstance(result, dict) and _tool_result_counts_as_failure(result) if is_error: turn_recovery.record_tool_failure() else: @@ -2912,7 +3223,11 @@ async def _unified_tool_loop_stream( messages.append(build_user_message_text( f"Tool result ({tool_name}):\n{result_text}" )) - self._persist_message(session_id, "tool", result_text, tool_name=tool_name) + self._persist_message( + session_id, "tool", result_text, + tool_name=tool_name, tool_call_id=f"text-{budget.used}", + metadata=self._tool_execution_metadata(result), + ) if _is_permission_hard_stop_payload(result): logger.info( @@ -3027,47 +3342,6 @@ async def _execute_tools_concurrent( executed: list[Dict[str, Any]] = [] original_names_by_id = {str(tc.id): str(tc.name) for tc in native_calls} - # Idempotency guard: skip duplicate platform_action calls within one turn. - # Prevents the model from accidentally repeating a side-effect action - # (e.g. im.send_message) when it copies the "first-try-failed → retry" - # pattern from a previous turn into the current turn's tool_calls list. - seen_action_fps: set[str] = set() - deduped: list = [] - for tc in native_calls: - fp = _platform_action_fingerprint(str(tc.name), dict(tc.arguments or {})) - if fp is not None and fp in seen_action_fps: - action_name = str((tc.arguments or {}).get("action") or tc.name) - skip_result: Dict[str, Any] = { - "ok": False, - "error": "idempotency_guard: duplicate platform_action skipped", - "reason": ( - f"'{action_name}' with identical payload was already queued " - "in this turn; LeapFlow prevents duplicate side-effects." - ), - "retryable": False, - } - messages.append({ - "role": "tool", - "tool_call_id": tc.id, - "content": json.dumps(skip_result, ensure_ascii=False), - }) - executed.append({ - "id": tc.id, - "name": "platform_action", - "original_tool_name": "platform_action", - "arguments": dict(tc.arguments or {}), - "result": skip_result, - }) - logger.info( - "idempotency_guard: skipped duplicate platform_action tc_id=%s action=%s", - tc.id, action_name, - ) - continue - if fp is not None: - seen_action_fps.add(fp) - deduped.append(tc) - native_calls = deduped - tc_wrappers = [ ConcurrentToolCall( id=tc.id, @@ -3087,7 +3361,9 @@ async def _execute_tools_concurrent( "arguments_summary": json.dumps(tc.arguments, default=str, ensure_ascii=False)[:300], }) _show_progress("executing", normalized_name, step=i + 1, total=len(native_calls)) - result = await self._execute_general_tool(tool_call_dict, handlers) + result = await self._execute_tool_with_ledger( + tool_call_dict, handlers, tool_call_id=str(tc.id), + ) _clear_indicator() self._emit_chat_event("tool_result", { "tool_name": normalized_name, @@ -3103,6 +3379,11 @@ async def _execute_tools_concurrent( result_payload = self._compact_tool_result(normalized_name, tc.arguments, result) result_text = json.dumps(result_payload, default=str, ensure_ascii=False)[:result_budget] messages.append({"role": "tool", "tool_call_id": tc.id, "content": result_text}) + self._persist_message( + self._current_session_id, "tool", result_text, + tool_name=normalized_name, tool_call_id=str(tc.id), + metadata=self._tool_execution_metadata(result), + ) executed.append({ "id": tc.id, "name": normalized_name, @@ -3110,9 +3391,19 @@ async def _execute_tools_concurrent( "arguments": tc.arguments, "result": result, }) - if isinstance(result, dict) and _is_permission_hard_stop_payload(result): + if isinstance(result, dict) and _should_stop_after_tool_result(normalized_name, result): + for skipped_tc in native_calls[i + 1:]: + skipped_call = _normalize_tool_call({"name": str(skipped_tc.name), "arguments": skipped_tc.arguments}) + skipped_name = str(skipped_call["name"]) + executed.append({ + "id": skipped_tc.id, + "name": skipped_name, + "original_tool_name": str(skipped_call.get("original_tool_name") or skipped_tc.name), + "arguments": skipped_tc.arguments, + "result": _skipped_after_failure_result(normalized_name, result), + }) logger.info( - "tool_concurrency: stopping remaining native tool calls after permission blocker from %s", + "tool_concurrency: stopping remaining native tool calls after failed side effect from %s", normalized_name, ) break @@ -3135,7 +3426,9 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: "original_tool_name": original_name, "normalized_tool_name": ctc.name, } - return await self._execute_general_tool(tool_call_dict, handlers) + return await self._execute_tool_with_ledger( + tool_call_dict, handlers, tool_call_id=str(ctc.id), + ) gather_results = await asyncio.gather( *[_run_one(ctc) for ctc in concurrent], @@ -3171,8 +3464,13 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: ) result_payload = self._compact_tool_result(ctc.name, ctc.arguments, result) result_text = json.dumps(result_payload, default=str, ensure_ascii=False)[:result_budget] - messages.append({"role": "tool", "tool_call_id": ctc.id, "content": result_text}) effective_result = error_result if isinstance(result, Exception) else result + messages.append({"role": "tool", "tool_call_id": ctc.id, "content": result_text}) + self._persist_message( + self._current_session_id, "tool", result_text, + tool_name=ctc.name, tool_call_id=str(ctc.id), + metadata=self._tool_execution_metadata(effective_result), + ) executed.append({ "id": ctc.id, "name": ctc.name, @@ -3180,9 +3478,18 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: "arguments": ctc.arguments, "result": effective_result, }) - if isinstance(effective_result, dict) and _is_permission_hard_stop_payload(effective_result): + if isinstance(effective_result, dict) and _should_stop_after_tool_result(ctc.name, effective_result): + for skipped_ctc in sequential: + skipped_original = original_names_by_id.get(str(skipped_ctc.id), skipped_ctc.name) + executed.append({ + "id": skipped_ctc.id, + "name": skipped_ctc.name, + "original_tool_name": skipped_original, + "arguments": skipped_ctc.arguments, + "result": _skipped_after_failure_result(ctc.name, effective_result), + }) logger.info( - "tool_concurrency: permission blocker returned from concurrent tool %s; skipping sequential group", + "tool_concurrency: failed side effect returned from concurrent tool %s; skipping sequential group", ctc.name, ) return executed @@ -3196,7 +3503,9 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: "original_tool_name": original_name, "normalized_tool_name": ctc.name, } - result = await self._execute_general_tool(tool_call_dict, handlers) + result = await self._execute_tool_with_ledger( + tool_call_dict, handlers, tool_call_id=str(ctc.id), + ) _clear_indicator() _print_tool_result(ctc.name, result, enabled=self._settings.verbose_progress) trace.record( @@ -3207,6 +3516,11 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: result_payload = self._compact_tool_result(ctc.name, ctc.arguments, result) result_text = json.dumps(result_payload, default=str, ensure_ascii=False)[:result_budget] messages.append({"role": "tool", "tool_call_id": ctc.id, "content": result_text}) + self._persist_message( + self._current_session_id, "tool", result_text, + tool_name=ctc.name, tool_call_id=str(ctc.id), + metadata=self._tool_execution_metadata(result), + ) executed.append({ "id": ctc.id, "name": ctc.name, @@ -3214,14 +3528,114 @@ async def _run_one(ctc: ConcurrentToolCall) -> Dict[str, Any]: "arguments": ctc.arguments, "result": result, }) - if isinstance(result, dict) and _is_permission_hard_stop_payload(result): + if isinstance(result, dict) and _should_stop_after_tool_result(ctc.name, result): + for skipped_ctc in sequential[i + 1:]: + skipped_original = original_names_by_id.get(str(skipped_ctc.id), skipped_ctc.name) + executed.append({ + "id": skipped_ctc.id, + "name": skipped_ctc.name, + "original_tool_name": skipped_original, + "arguments": skipped_ctc.arguments, + "result": _skipped_after_failure_result(ctc.name, result), + }) logger.info( - "tool_concurrency: stopping sequential native tool calls after permission blocker from %s", + "tool_concurrency: stopping sequential native tool calls after failed side effect from %s", ctc.name, ) break return executed + async def _execute_tool_with_ledger( + self, + tool_call: Dict[str, Any], + handlers: Dict[str, Any], + *, + tool_call_id: str = "", + ) -> Dict[str, Any]: + """Execute a tool through the unified idempotency ledger.""" + original_name = str(tool_call.get("original_tool_name") or tool_call.get("name", "")) + proposed_name = str(tool_call.get("name", "")) + args = dict(tool_call.get("arguments") or {}) + registry = _default_tool_registry() + resolution = registry.resolve(proposed_name, args) + if not resolution.auto_executable or resolution.normalized_name is None: + return await self._execute_general_tool(tool_call, handlers) + + tool_name = resolution.normalized_name + spec = registry.specs.get(tool_name) + policy = execution_policy_for(tool_name, spec) + session_id = self._current_session_id or "ephemeral" + turn_id = self._current_turn_id or f"turn-{self._session_turn_count}" + command_id = self._current_command_id or turn_id + normalized_call = { + **tool_call, + "name": tool_name, + "arguments": args, + "original_tool_name": original_name, + "normalized_tool_name": tool_name, + } + record, existing = self._tool_execution_ledger.reserve( + session_id=session_id, + turn_id=turn_id, + command_id=command_id, + tool_call_id=tool_call_id, + tool_name=tool_name, + arguments=args, + policy=policy, + ) + if existing is not None: + if existing.status == "running": + existing = await self._tool_execution_ledger.wait_for_completion( + existing, + timeout_s=self._tool_timeouts.get(tool_name, self._default_tool_timeout_s), + ) + duplicate = ToolExecutionLedger.duplicate_result(existing) + duplicate.update({ + "tool_name": tool_name, + "tool_call_id": tool_call_id, + "execution_policy": existing.policy, + }) + logger.info( + "tool_idempotency: skipped duplicate tool=%s policy=%s key=%s", + tool_name, existing.policy, existing.idempotency_key[:12], + ) + return duplicate + + try: + result = await self._execute_general_tool(normalized_call, handlers) + except Exception as exc: + failed_result: Dict[str, Any] = { + "ok": False, + "error": f"{type(exc).__name__}: {exc}", + "retryable": True, + "execution_id": record.execution_id, + "idempotency_key": record.idempotency_key, + "execution_policy": policy, + "tool_call_id": tool_call_id, + } + self._tool_execution_ledger.complete(record, failed_result) + raise + if isinstance(result, dict): + result_for_ledger: Dict[str, Any] = { + **result, + "execution_id": record.execution_id, + "idempotency_key": record.idempotency_key, + "execution_policy": policy, + "tool_call_id": tool_call_id, + } + else: + result_for_ledger = { + "ok": True, + "result": result, + "execution_id": record.execution_id, + "idempotency_key": record.idempotency_key, + "execution_policy": policy, + "tool_call_id": tool_call_id, + } + completed = self._tool_execution_ledger.complete(record, result_for_ledger) + result_for_ledger["execution_status"] = completed.status + return result_for_ledger + async def _execute_general_tool( self, tool_call: Dict[str, Any], handlers: Dict[str, Any] ) -> Dict[str, Any]: @@ -3422,10 +3836,28 @@ def _ensure_session(self, user_text: str) -> Optional[str]: logger.debug("session.ensure failed", exc_info=True) return None + @staticmethod + def _tool_execution_metadata(result: Any) -> Dict[str, Any]: + """Extract tool execution audit metadata for transcript rows.""" + if not isinstance(result, dict): + return {} + metadata: Dict[str, Any] = {} + for key in ( + "execution_id", "idempotency_key", "execution_status", "execution_policy", + "already_executed", "duplicate_suppressed", "execution_reused", "execution_skipped", + "counts_as_failure", "counts_as_tool_attempt", "ui_hidden", "skipped_reason", + "blocked_by_tool", "blocked_by_error", "tool_call_id", + ): + if key in result: + metadata[key] = result[key] + return metadata + def _persist_message( self, session_id: Optional[str], role: str, content: str, *, tool_name: Optional[str] = None, + tool_call_id: Optional[str] = None, tool_calls: Optional[list] = None, + metadata: Optional[Dict[str, Any]] = None, ) -> None: """Persist a message to conversation store (fire-and-forget).""" if not session_id or not self._conversation_store: @@ -3433,7 +3865,8 @@ def _persist_message( try: self._conversation_store.append_message( session_id, role, content[:8000], - tool_name=tool_name, tool_calls=tool_calls, + tool_name=tool_name, tool_call_id=tool_call_id, + tool_calls=tool_calls, metadata=metadata, ) except Exception: logger.debug("session.persist_message failed", exc_info=True) @@ -3532,9 +3965,12 @@ def _count_consecutive_tool_failures(messages: List[Dict[str, Any]]) -> int: continue try: parsed = json.loads(content) - if isinstance(parsed, dict) and parsed.get("ok") is False: - count += 1 - continue + if isinstance(parsed, dict): + if _tool_result_counts_as_failure(parsed): + count += 1 + continue + if parsed.get("counts_as_failure") is False or _tool_result_is_control_signal(parsed): + continue except (json.JSONDecodeError, ValueError): pass # Non-JSON or ok!=False — treat as success, reset @@ -4059,7 +4495,9 @@ async def _bridge_fn() -> Any: if a_type == "tool": from leapflow.tools.registry_bootstrap import TOOL_HANDLERS tool_call_dict = {"name": name, "arguments": payload} - result = await self._execute_general_tool(tool_call_dict, TOOL_HANDLERS) + result = await self._execute_tool_with_ledger( + tool_call_dict, TOOL_HANDLERS, tool_call_id=f"action-{name}", + ) logger.info("audit.tool name=%s ok=%s", name, result.get("ok")) return result diff --git a/src/leapflow/engine/failure_envelope.py b/src/leapflow/engine/failure_envelope.py new file mode 100644 index 0000000..4e49cfd --- /dev/null +++ b/src/leapflow/engine/failure_envelope.py @@ -0,0 +1,154 @@ +"""Structured failure representation for the recovery subsystem. + +FailureEnvelope wraps every failure encountered in the agent loop with +enough metadata to drive automated recovery decisions: source, category, +recoverability, side-effect state, and context. +""" +from __future__ import annotations + +import time +import uuid +from dataclasses import dataclass, field +from enum import Enum +from typing import Any + + +class FailureSource(Enum): + """Origin subsystem of a failure.""" + + LLM = "llm" + TOOL = "tool" + SYSTEM = "system" + SECURITY = "security" + + +class Recoverability(Enum): + """How a failure can potentially be recovered.""" + + AUTO_RETRY = "auto_retry" + AUTO_RECOVER = "auto_recover" + USER_FIXABLE = "user_fixable" + ADMIN_REQUIRED = "admin_required" + NON_RECOVERABLE = "non_recoverable" + + +class SideEffectState(Enum): + """Observed side-effect state at the point of failure.""" + + NONE = "none" + COMMITTED = "committed" + PARTIAL = "partial" + UNKNOWN = "unknown" + + +@dataclass(frozen=True) +class FailureContext: + """Execution context at the moment of failure. + + Uses tuple of pairs for arguments to remain hashable in a frozen dataclass. + """ + + tool_name: str = "" + arguments: tuple[tuple[str, Any], ...] = () + execution_id: str = "" + trace_id: str = "" + turn_id: int = 0 + session_id: str = "" + provider: str = "" + model: str = "" + attempt_number: int = 0 + elapsed_ms: float = 0.0 + + @classmethod + def from_dict_args( + cls, + *, + tool_name: str = "", + arguments: dict[str, Any] | None = None, + execution_id: str = "", + trace_id: str = "", + turn_id: int = 0, + session_id: str = "", + provider: str = "", + model: str = "", + attempt_number: int = 0, + elapsed_ms: float = 0.0, + ) -> FailureContext: + """Factory that converts a dict of arguments into frozen-compatible tuple form.""" + args_tuple = tuple(sorted((arguments or {}).items())) + return cls( + tool_name=tool_name, + arguments=args_tuple, + execution_id=execution_id, + trace_id=trace_id, + turn_id=turn_id, + session_id=session_id, + provider=provider, + model=model, + attempt_number=attempt_number, + elapsed_ms=elapsed_ms, + ) + + @property + def arguments_dict(self) -> dict[str, Any]: + """Reconstruct arguments as a dict for downstream consumers.""" + return dict(self.arguments) + + +@dataclass(frozen=True) +class RecoveryHint: + """Provider- or system-generated hint for how to resolve a failure.""" + + hint_text: str = "" + suggested_command: str = "" + documentation_url: str = "" + + +@dataclass(frozen=True) +class FailureEnvelope: + """Immutable, self-describing failure record for recovery coordination. + + Every failure in the agent loop is wrapped in this envelope before being + passed to the RecoveryCoordinator for decision-making. + """ + + envelope_id: str + source: FailureSource + category: str + failure_class: str + failure_code: str + message: str + recoverability: Recoverability + side_effect_state: SideEffectState + context: FailureContext + timestamp: float + provider_hint: RecoveryHint | None = None + + @classmethod + def create( + cls, + *, + source: FailureSource, + category: str, + failure_class: str, + failure_code: str, + message: str, + recoverability: Recoverability, + side_effect_state: SideEffectState = SideEffectState.NONE, + context: FailureContext | None = None, + provider_hint: RecoveryHint | None = None, + ) -> FailureEnvelope: + """Convenience factory that auto-generates envelope_id and timestamp.""" + return cls( + envelope_id=uuid.uuid4().hex, + source=source, + category=category, + failure_class=failure_class, + failure_code=failure_code, + message=message, + recoverability=recoverability, + side_effect_state=side_effect_state, + context=context or FailureContext(), + timestamp=time.time(), + provider_hint=provider_hint, + ) diff --git a/src/leapflow/engine/interaction_request.py b/src/leapflow/engine/interaction_request.py new file mode 100644 index 0000000..40f9dca --- /dev/null +++ b/src/leapflow/engine/interaction_request.py @@ -0,0 +1,132 @@ +"""Interaction request types for the agent loop recovery subsystem. + +When automated recovery is insufficient (e.g., permissions, credentials, +ambiguous intent), the recovery coordinator emits an InteractionRequest +that the TUI or gateway surfaces to the user for resolution. +""" +from __future__ import annotations + +import uuid +from dataclasses import dataclass +from enum import Enum +from typing import Any + + +class InteractionType(Enum): + """Classification of what the agent needs from the user.""" + + APPROVAL = "approval" + CLARIFICATION = "clarification" + PARAMETER_CONFIRMATION = "parameter_confirmation" + RECOVERY_GUIDANCE = "recovery_guidance" + CREDENTIAL_SETUP = "credential_setup" + PERMISSION_GRANT = "permission_grant" + RETRY_CHOICE = "retry_choice" + CONFLICT_RESOLUTION = "conflict_resolution" + CONTINUE_AFTER_FIX = "continue_after_fix" + DEGRADED_MODE_CONFIRM = "degraded_mode_confirm" + + +class Severity(Enum): + """Visual and priority severity for an interaction request.""" + + INFO = "info" + WARNING = "warning" + ERROR = "error" + CRITICAL = "critical" + + +class TimeoutBehavior(Enum): + """What happens when an interaction request times out without user response.""" + + CANCEL = "cancel" + DEGRADE = "degrade" + PERSIST = "persist" + + +@dataclass(frozen=True) +class SuggestedAction: + """A suggested action the user can take to resolve an interaction request.""" + + label: str + command: str = "" + description: str = "" + is_default: bool = False + + +@dataclass(frozen=True) +class InteractionRequest: + """Immutable interaction request emitted by the recovery coordinator. + + Represents a point where automated recovery cannot proceed without + user input. The TUI or gateway layer surfaces this request and waits + for user response before resuming the agent loop. + """ + + request_id: str + interaction_type: InteractionType + severity: Severity + title: str + description: str + suggested_actions: tuple[SuggestedAction, ...] = () + context: tuple[tuple[str, Any], ...] = () + resumption_key: str = "" + expires_at: float | None = None + timeout_behavior: TimeoutBehavior = TimeoutBehavior.PERSIST + + @classmethod + def create( + cls, + *, + interaction_type: InteractionType, + severity: Severity, + title: str, + description: str, + suggested_actions: tuple[SuggestedAction, ...] = (), + context: dict[str, Any] | None = None, + resumption_key: str = "", + expires_at: float | None = None, + timeout_behavior: TimeoutBehavior = TimeoutBehavior.PERSIST, + ) -> InteractionRequest: + """Factory that auto-generates request_id and normalizes context.""" + ctx_tuple = tuple(sorted((context or {}).items())) + return cls( + request_id=uuid.uuid4().hex, + interaction_type=interaction_type, + severity=severity, + title=title, + description=description, + suggested_actions=suggested_actions, + context=ctx_tuple, + resumption_key=resumption_key, + expires_at=expires_at, + timeout_behavior=timeout_behavior, + ) + + @property + def context_dict(self) -> dict[str, Any]: + """Reconstruct context as a dict for downstream consumers.""" + return dict(self.context) + + def to_json(self) -> dict[str, Any]: + """Produce a JSON-serializable representation of this request.""" + return { + "request_id": self.request_id, + "interaction_type": self.interaction_type.value, + "severity": self.severity.value, + "title": self.title, + "description": self.description, + "suggested_actions": [ + { + "label": a.label, + "command": a.command, + "description": a.description, + "is_default": a.is_default, + } + for a in self.suggested_actions + ], + "context": self.context_dict, + "resumption_key": self.resumption_key, + "expires_at": self.expires_at, + "timeout_behavior": self.timeout_behavior.value, + } diff --git a/src/leapflow/engine/oneshot_guard.py b/src/leapflow/engine/oneshot_guard.py new file mode 100644 index 0000000..aeea3d3 --- /dev/null +++ b/src/leapflow/engine/oneshot_guard.py @@ -0,0 +1,67 @@ +"""One-shot guard — ensures each strategy key fires at most once per lifetime. + +Replaces the pattern of 9+ boolean flags (tried_compress, tried_failover, etc.) +with a generic, auditable guard that tracks which strategies have been attempted +and when they were consumed. +""" +from __future__ import annotations + +import time + + +class OneShotGuard: + """Generalized one-shot guard replacing TurnRecoveryState's boolean flags. + + Each strategy key can fire at most once per guard lifetime. + Provides an audit trail of which strategies were attempted and when. + Supports per-turn reset via new_turn() while preserving audit history. + """ + + def __init__(self) -> None: + self._used: dict[str, float] = {} + self._history: list[dict[str, float]] = [] + + def is_available(self, key: str) -> bool: + """Check whether a strategy key has not been used yet.""" + return key not in self._used + + def mark_used(self, key: str) -> None: + """Mark a strategy key as consumed. Idempotent — re-marking is a no-op.""" + if key not in self._used: + self._used[key] = time.time() + + def used_strategies(self) -> list[str]: + """Return the list of strategy keys that have been used, in usage order.""" + return sorted(self._used.keys(), key=lambda k: self._used[k]) + + def usage_history(self) -> dict[str, float]: + """Return a mapping of strategy key → timestamp when it was consumed.""" + return dict(self._used) + + def new_turn(self) -> None: + """Reset guard state for a new turn. Called at turn boundary. + + Previous turn's used strategies remain in history for audit, + but are no longer blocking. + """ + if self._used: + self._history.append(dict(self._used)) + self._used.clear() + + def reset(self) -> None: + """Clear all usage records. Intended for testing and turn resets.""" + self._used.clear() + self._history.clear() + + @property + def turn_history(self) -> list[dict[str, float]]: + """Return previous turns' usage records for audit purposes.""" + return list(self._history) + + def __len__(self) -> int: + """Number of strategy keys that have been consumed.""" + return len(self._used) + + def __contains__(self, key: str) -> bool: + """Support `key in guard` syntax.""" + return key in self._used diff --git a/src/leapflow/engine/recovery_audit.py b/src/leapflow/engine/recovery_audit.py new file mode 100644 index 0000000..898f2e1 --- /dev/null +++ b/src/leapflow/engine/recovery_audit.py @@ -0,0 +1,172 @@ +"""Recovery audit system — structured logging of all recovery decisions. + +Every RecoveryCoordinator.evaluate() call produces an audit entry written +to a JSONL file for post-hoc analysis, debugging, and learning. +""" +from __future__ import annotations + +import json +import logging +import time +from collections import deque +from dataclasses import dataclass, asdict +from pathlib import Path +from typing import Any, Protocol + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class RecoveryAuditEntry: + """Immutable audit record for one recovery decision.""" + + timestamp: float + session_id: str + turn_id: int + + # Input (from FailureEnvelope) + envelope_id: str + failure_source: str + failure_category: str + failure_code: str + recoverability: str + + # Decision (from RecoveryDecision) + decision_id: str + strategy_key: str + action: str + reason: str + budget_cost: int = 0 + + # Budget snapshot + budget_consumed: int = 0 + budget_remaining: int = 0 + + # Outcome (filled async after execution) + outcome: str = "" # "success" | "failure" | "timeout" | "" + outcome_reason: str = "" + elapsed_ms: float = 0.0 + + def to_json_dict(self) -> dict[str, Any]: + """Serialize to JSON-safe dict. Excludes only trailing optional fields when empty.""" + d = asdict(self) + # Only strip the optional outcome fields when they are at default/empty + for key in ("outcome", "outcome_reason"): + if d.get(key) == "": + del d[key] + if d.get("elapsed_ms") == 0.0: + del d["elapsed_ms"] + return d + + +class RecoveryAuditSink(Protocol): + """Protocol for audit sinks. Allows different backends.""" + + def record(self, entry: RecoveryAuditEntry) -> None: + """Write an audit entry.""" + ... + + def update_outcome(self, decision_id: str, outcome: str, + reason: str = "", elapsed_ms: float = 0.0) -> None: + """Update the outcome of a previously recorded decision.""" + ... + + +class JsonlAuditSink: + """JSONL file-based audit sink. + + Writes one JSON line per entry to a .jsonl file. + Thread-safe via append mode. + """ + + _MAX_IN_MEMORY = 1000 # Default configurable cap + + def __init__(self, path: Path | str | None = None, max_in_memory: int = 1000): + self._path = Path(path) if path else None + self._entries: deque[RecoveryAuditEntry] = deque(maxlen=max_in_memory) + + def record(self, entry: RecoveryAuditEntry) -> None: + """Write entry to JSONL file and in-memory buffer.""" + self._entries.append(entry) + if self._path: + try: + self._path.parent.mkdir(parents=True, exist_ok=True) + with self._path.open("a", encoding="utf-8") as f: + f.write(json.dumps(entry.to_json_dict(), ensure_ascii=False)) + f.write("\n") + except OSError as exc: + logger.warning("Failed to write audit entry: %s", exc) + + def update_outcome(self, decision_id: str, outcome: str, + reason: str = "", elapsed_ms: float = 0.0) -> None: + """Update outcome by writing an update record.""" + update = { + "type": "outcome_update", + "decision_id": decision_id, + "outcome": outcome, + "outcome_reason": reason, + "elapsed_ms": elapsed_ms, + "timestamp": time.time(), + } + if self._path: + try: + with self._path.open("a", encoding="utf-8") as f: + f.write(json.dumps(update, ensure_ascii=False)) + f.write("\n") + except OSError as exc: + logger.warning("Failed to write audit outcome: %s", exc) + + @property + def entries(self) -> list[RecoveryAuditEntry]: + """In-memory entries (for testing).""" + return list(self._entries) + + def summary(self) -> dict[str, Any]: + """Generate summary statistics from in-memory entries.""" + if not self._entries: + return {"total": 0} + total = len(self._entries) + by_strategy: dict[str, int] = {} + by_action: dict[str, int] = {} + for e in self._entries: + by_strategy[e.strategy_key] = by_strategy.get(e.strategy_key, 0) + 1 + by_action[e.action] = by_action.get(e.action, 0) + 1 + return { + "total": total, + "by_strategy": by_strategy, + "by_action": by_action, + } + + +def create_audit_entry( + envelope: "FailureEnvelope", + decision: "RecoveryDecision", + budget: "RecoveryBudget", + session_id: str = "", + turn_id: int = 0, +) -> RecoveryAuditEntry: + """Factory function to create an audit entry from coordinator outputs. + + Handles attribute access safely for enum values and budget internals. + """ + from leapflow.engine.failure_envelope import FailureEnvelope # noqa: F811 + from leapflow.engine.recovery_decision import RecoveryDecision # noqa: F811 + from leapflow.engine.recovery_budget import RecoveryBudget # noqa: F811 + + return RecoveryAuditEntry( + timestamp=time.time(), + session_id=session_id, + turn_id=turn_id, + envelope_id=envelope.envelope_id, + failure_source=envelope.source.value if hasattr(envelope.source, 'value') else str(envelope.source), + failure_category=envelope.category, + failure_code=envelope.failure_code, + recoverability=envelope.recoverability.value if hasattr(envelope.recoverability, 'value') else str(envelope.recoverability), + decision_id=decision.decision_id, + strategy_key=decision.strategy_key, + action=decision.action.value if hasattr(decision.action, 'value') else str(decision.action), + reason=decision.reason, + budget_cost=decision.budget_cost, + budget_consumed=budget._consumed if hasattr(budget, '_consumed') else 0, + budget_remaining=budget.remaining() if hasattr(budget, 'remaining') else 0, + ) diff --git a/src/leapflow/engine/recovery_budget.py b/src/leapflow/engine/recovery_budget.py new file mode 100644 index 0000000..2ea1dae --- /dev/null +++ b/src/leapflow/engine/recovery_budget.py @@ -0,0 +1,144 @@ +"""Recovery budget — global constraint system for recovery attempts within a turn. + +The budget prevents infinite recovery loops by enforcing hard caps on retries, +transforms, failovers, credential rotations, and wall-clock time within a +single agent loop turn. +""" +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Any + + +@dataclass +class RecoveryBudget: + """Global constraint system for recovery attempts within a turn. + + Provides per-turn and per-category limits plus a wall-clock deadline. + Strategies consult the budget before deciding whether to attempt recovery. + """ + + max_retries_per_turn: int = 8 + max_retry_per_category: int = 3 + max_transform_attempts: int = 2 + max_failovers: int = 2 + max_credential_rotations: int = 3 + turn_deadline_s: float = 300.0 + total_recovery_actions: int = 12 + + # Internal accounting — not meant for external configuration + _consumed: int = field(default=0, init=False, repr=False) + _category_consumed: dict[str, int] = field(default_factory=dict, init=False, repr=False) + _transforms_used: int = field(default=0, init=False, repr=False) + _failovers_used: int = field(default=0, init=False, repr=False) + _rotations_used: int = field(default=0, init=False, repr=False) + _deadline_start: float = field(default=0.0, init=False, repr=False) + + def start_deadline(self) -> None: + """Called at turn start to anchor the wall-clock deadline.""" + self._deadline_start = time.monotonic() + + def new_turn(self) -> None: + """Reset all per-turn accounting for a new turn. + + Configuration limits (max_retries_per_turn, etc.) remain unchanged; + only the consumption counters and deadline are reset. + """ + self._consumed = 0 + self._category_consumed.clear() + self._transforms_used = 0 + self._failovers_used = 0 + self._rotations_used = 0 + self.start_deadline() + + def can_afford(self, cost: int, category: str = "") -> bool: + """Check whether spending `cost` actions is within budget. + + Checks global budget, per-category budget, and deadline. + """ + if self.is_deadline_exceeded(): + return False + if self._consumed + cost > self.total_recovery_actions: + return False + if category: + cat_used = self._category_consumed.get(category, 0) + if cat_used + cost > self.max_retry_per_category: + return False + return True + + def consume(self, cost: int, category: str = "") -> None: + """Record that `cost` recovery actions were consumed. + + Raises ValueError if the budget is exceeded — callers should check + can_afford first. + """ + if not self.can_afford(cost, category): + raise ValueError( + f"Recovery budget exceeded: consumed={self._consumed}, " + f"cost={cost}, category={category!r}, " + f"category_used={self._category_consumed.get(category, 0)}" + ) + self._consumed += cost + if category: + self._category_consumed[category] = ( + self._category_consumed.get(category, 0) + cost + ) + + def consume_transform(self) -> None: + """Record a transform attempt (context compression, payload reduction).""" + self._transforms_used += 1 + + def consume_failover(self) -> None: + """Record a failover attempt (model switch, endpoint rotation).""" + self._failovers_used += 1 + + def consume_rotation(self) -> None: + """Record a credential rotation attempt.""" + self._rotations_used += 1 + + def can_transform(self) -> bool: + """Whether transform budget remains.""" + return self._transforms_used < self.max_transform_attempts + + def can_failover(self) -> bool: + """Whether failover budget remains.""" + return self._failovers_used < self.max_failovers + + def can_rotate(self) -> bool: + """Whether credential rotation budget remains.""" + return self._rotations_used < self.max_credential_rotations + + def is_deadline_exceeded(self) -> bool: + """Whether the wall-clock deadline for this turn has been exceeded.""" + if self._deadline_start == 0.0: + return False + elapsed = time.monotonic() - self._deadline_start + return elapsed > self.turn_deadline_s + + def remaining(self) -> int: + """Remaining global recovery action budget.""" + return max(0, self.total_recovery_actions - self._consumed) + + def category_remaining(self, category: str) -> int: + """Remaining budget for a specific error category.""" + used = self._category_consumed.get(category, 0) + return max(0, self.max_retry_per_category - used) + + def summary(self) -> dict[str, Any]: + """Return an audit-friendly summary of budget consumption.""" + return { + "consumed": self._consumed, + "remaining": self.remaining(), + "total_budget": self.total_recovery_actions, + "category_consumed": dict(self._category_consumed), + "transforms_used": self._transforms_used, + "failovers_used": self._failovers_used, + "rotations_used": self._rotations_used, + "deadline_exceeded": self.is_deadline_exceeded(), + "elapsed_s": ( + round(time.monotonic() - self._deadline_start, 2) + if self._deadline_start > 0 + else 0.0 + ), + } diff --git a/src/leapflow/engine/recovery_checkpoint.py b/src/leapflow/engine/recovery_checkpoint.py new file mode 100644 index 0000000..1a0dac0 --- /dev/null +++ b/src/leapflow/engine/recovery_checkpoint.py @@ -0,0 +1,380 @@ +"""Recovery checkpoint system for cross-turn state persistence and safe resumption. + +Enables the agent loop to save execution state when halting with HALT_WITH_CHECKPOINT, +and resume safely in a subsequent turn after user action or external fix. +""" +from __future__ import annotations + +import time +import uuid +from dataclasses import dataclass, field +from enum import Enum +from typing import Any, Protocol, runtime_checkable + + +class CheckpointState(Enum): + """Lifecycle states of a recovery checkpoint.""" + + PENDING = "pending" # Created, waiting for resume + CONSUMED = "consumed" # Successfully resumed (CAS consumed) + EXPIRED = "expired" # TTL exceeded, auto-cleaned + CANCELLED = "cancelled" # User cancelled recovery + + +@dataclass(frozen=True) +class Precondition: + """A verifiable condition that must hold for safe resumption. + + Examples: session_id match, workspace_root unchanged, credential hash valid. + """ + + key: str # e.g. "session_id", "workspace_root", "credential_hash" + expected_value: str # Value at checkpoint creation time + description: str = "" # Human-readable explanation + critical: bool = True # If True, drift blocks resume; if False, warn only + + +@dataclass(frozen=True) +class StepRecord: + """Record of a completed execution step.""" + + step_id: str + tool_name: str = "" + action: str = "" + result_summary: str = "" + timestamp: float = 0.0 + has_side_effect: bool = False + + +@dataclass +class RecoveryCheckpoint: + """Persistent checkpoint for cross-turn recovery. + + Created when RecoveryCoordinator returns HALT_WITH_CHECKPOINT. + Enables the loop to resume from a known-safe state after user fixes the issue. + + Design decisions: + - Mutable (state transitions: PENDING -> CONSUMED/EXPIRED/CANCELLED) + - version field for optimistic locking (CAS semantics) + - TTL-based expiry with configurable default + - preconditions verified before resume to detect state drift + """ + + checkpoint_id: str = field(default_factory=lambda: str(uuid.uuid4())) + session_id: str = "" + turn_id: int = 0 + version: int = 1 # Optimistic lock version + created_at: float = field(default_factory=time.time) + ttl_seconds: float = 3600.0 # Default 1 hour + state: CheckpointState = CheckpointState.PENDING + + # Failure context + failure_envelope_data: dict[str, Any] = field(default_factory=dict) + interaction_request_id: str = "" + + # Execution state snapshot + completed_steps: tuple[StepRecord, ...] = () + pending_steps: tuple[str, ...] = () # Descriptions of what remains + messages_snapshot: list[dict[str, Any]] = field(default_factory=list) + + # Preconditions for safe resume + preconditions: tuple[Precondition, ...] = () + + # Context for resumption + context_data: dict[str, Any] = field(default_factory=dict) + + # Recovery tracking + resume_attempts: int = 0 + max_resume_attempts: int = 3 + consumed_at: float | None = None + + @property + def is_expired(self) -> bool: + """Whether the checkpoint has exceeded its TTL.""" + return time.time() > (self.created_at + self.ttl_seconds) + + @property + def is_consumable(self) -> bool: + """Whether this checkpoint can still be consumed (resumed).""" + return ( + self.state == CheckpointState.PENDING + and not self.is_expired + and self.resume_attempts < self.max_resume_attempts + ) + + @property + def expires_at(self) -> float: + """Absolute timestamp when the checkpoint expires.""" + return self.created_at + self.ttl_seconds + + +@dataclass(frozen=True) +class ResumeResult: + """Result of attempting to resume from a checkpoint.""" + + success: bool + reason: str = "" + messages: list[dict[str, Any]] = field(default_factory=list) + context_data: dict[str, Any] = field(default_factory=dict) + drift_warnings: tuple[str, ...] = () # Non-critical precondition drifts + + +@runtime_checkable +class CheckpointStore(Protocol): + """Protocol for checkpoint persistence backends. + + Implementations must provide CAS (Compare-And-Swap) semantics + for load_and_consume to prevent concurrent resumption. + """ + + def save(self, checkpoint: RecoveryCheckpoint) -> None: + """Persist a checkpoint. Overwrites if same checkpoint_id exists.""" + ... + + def load(self, checkpoint_id: str) -> RecoveryCheckpoint | None: + """Load a checkpoint by ID. Returns None if not found or expired.""" + ... + + def consume(self, checkpoint_id: str) -> bool: + """Atomically mark a PENDING checkpoint as CONSUMED. Returns True if successful.""" + ... + + def load_and_consume(self, checkpoint_id: str) -> RecoveryCheckpoint | None: + """Atomically load and mark as CONSUMED (CAS semantics). + + Returns the checkpoint if successfully consumed, None if: + - Not found + - Already consumed + - Expired + - Max resume attempts exceeded + + This must be atomic — concurrent calls for the same ID must + result in exactly one success. + """ + ... + + def list_pending(self, session_id: str = "") -> list[RecoveryCheckpoint]: + """List all pending (resumable) checkpoints, optionally filtered by session.""" + ... + + def cancel(self, checkpoint_id: str) -> bool: + """Mark a checkpoint as cancelled. Returns True if it was pending.""" + ... + + def cleanup_expired(self) -> int: + """Remove expired checkpoints. Returns count of removed entries.""" + ... + + +class InMemoryCheckpointStore: + """In-memory checkpoint store for testing and single-session use. + + Thread-safe via simple dict operations (GIL-protected for CPython). + For production multi-process use, implement DuckDB-backed store. + """ + + def __init__(self) -> None: + self._checkpoints: dict[str, RecoveryCheckpoint] = {} + + def save(self, checkpoint: RecoveryCheckpoint) -> None: + """Persist a checkpoint. Overwrites if same checkpoint_id exists.""" + self._checkpoints[checkpoint.checkpoint_id] = checkpoint + + def load(self, checkpoint_id: str) -> RecoveryCheckpoint | None: + """Load a checkpoint by ID. Returns None if not found or expired.""" + cp = self._checkpoints.get(checkpoint_id) + if cp is None: + return None + if cp.is_expired: + self._checkpoints.pop(checkpoint_id, None) + return None + return cp + + def consume(self, checkpoint_id: str) -> bool: + """Atomically mark a PENDING checkpoint as CONSUMED. Returns True if successful.""" + cp = self._checkpoints.get(checkpoint_id) + if cp is None: + return False + if cp.is_expired: + cp.state = CheckpointState.EXPIRED + return False + if cp.state != CheckpointState.PENDING: + return False + if cp.resume_attempts >= cp.max_resume_attempts: + return False + cp.state = CheckpointState.CONSUMED + cp.consumed_at = time.time() + cp.version += 1 + return True + + def load_and_consume(self, checkpoint_id: str) -> RecoveryCheckpoint | None: + """Atomically load and mark as CONSUMED (CAS semantics). + + Returns the checkpoint if successfully consumed, None if not consumable. + Increments resume_attempts on each call regardless of success. + """ + cp = self._checkpoints.get(checkpoint_id) + if cp is None: + return None + + # Auto-expire if past TTL + if cp.is_expired: + cp.state = CheckpointState.EXPIRED + return None + + # Must be PENDING and within attempt limits + if cp.state != CheckpointState.PENDING: + return None + if cp.resume_attempts >= cp.max_resume_attempts: + return None + + # CAS: atomically transition to CONSUMED and bump version + cp.resume_attempts += 1 + cp.state = CheckpointState.CONSUMED + cp.consumed_at = time.time() + cp.version += 1 + return cp + + def list_pending(self, session_id: str = "") -> list[RecoveryCheckpoint]: + """List all pending (resumable) checkpoints, optionally filtered by session.""" + result: list[RecoveryCheckpoint] = [] + for cp in self._checkpoints.values(): + # Auto-expire stale entries + if cp.is_expired and cp.state == CheckpointState.PENDING: + cp.state = CheckpointState.EXPIRED + continue + if cp.state != CheckpointState.PENDING: + continue + if session_id and cp.session_id != session_id: + continue + result.append(cp) + return result + + def cancel(self, checkpoint_id: str) -> bool: + """Mark a checkpoint as cancelled. Returns True if it was pending.""" + cp = self._checkpoints.get(checkpoint_id) + if cp is None: + return False + if cp.state != CheckpointState.PENDING: + return False + cp.state = CheckpointState.CANCELLED + cp.version += 1 + return True + + def cleanup_expired(self) -> int: + """Remove expired checkpoints. Returns count of removed entries.""" + expired_ids: list[str] = [] + for cid, cp in self._checkpoints.items(): + if cp.is_expired: + expired_ids.append(cid) + for cid in expired_ids: + del self._checkpoints[cid] + return len(expired_ids) + + +@runtime_checkable +class DriftVerifier(Protocol): + """Protocol for custom drift verification logic.""" + + def get_current_value(self, key: str) -> str: + """Retrieve the current value for a precondition key.""" + ... + + +class DefaultDriftVerifier: + """Default drift verifier — returns empty string for all keys.""" + + def get_current_value(self, key: str) -> str: + """Returns empty string (no environment introspection by default).""" + return "" + + +class CheckpointResumer: + """Orchestrates safe resumption from a checkpoint. + + Responsibilities: + 1. Load checkpoint via CAS (prevents double-resume) + 2. Verify all preconditions (detect state drift) + 3. Build resume context (messages + injection) + 4. Return ResumeResult for the engine loop to act on + """ + + def __init__( + self, + store: CheckpointStore, + drift_verifier: DriftVerifier | None = None, + ) -> None: + self._store = store + self._drift_verifier = drift_verifier or DefaultDriftVerifier() + + def resume( + self, + checkpoint_id: str, + current_context: dict[str, Any] | None = None, + ) -> ResumeResult: + """Attempt to resume from a checkpoint. + + Args: + checkpoint_id: The checkpoint to resume from + current_context: Current environment values for drift detection + (e.g. {"session_id": "...", "workspace_root": "..."}) + """ + # 1. Load checkpoint (without consuming) + checkpoint = self._store.load(checkpoint_id) + if checkpoint is None or not checkpoint.is_consumable: + return ResumeResult( + success=False, + reason="Checkpoint not available (expired, consumed, or not found)", + ) + + # 2. Verify preconditions + context = current_context or {} + drift_errors: list[str] = [] + drift_warnings: list[str] = [] + + for precond in checkpoint.preconditions: + actual = context.get(precond.key, "") + if str(actual) != precond.expected_value: + msg = f"{precond.key}: expected '{precond.expected_value}', got '{actual}'" + if precond.critical: + drift_errors.append(msg) + else: + drift_warnings.append(msg) + + if drift_errors: + # Preconditions failed: increment resume_attempts, keep PENDING + checkpoint.resume_attempts += 1 + if checkpoint.resume_attempts >= checkpoint.max_resume_attempts: + checkpoint.state = CheckpointState.EXPIRED + self._store.save(checkpoint) + return ResumeResult( + success=False, + reason=f"Critical precondition drift: {'; '.join(drift_errors)}", + drift_warnings=tuple(drift_warnings), + ) + + # 3. Preconditions passed — atomically consume + if not self._store.consume(checkpoint_id): + return ResumeResult( + success=False, + reason="Checkpoint not available (expired, consumed, or not found)", + ) + + # 4. Build resume messages + messages = list(checkpoint.messages_snapshot) + failure_msg = checkpoint.failure_envelope_data.get("message", "unknown") + messages.append({ + "role": "system", + "content": ( + f"[Recovery] Resuming from checkpoint. " + f"Previous failure: {failure_msg}. " + f"User has resolved the issue. Continue from where you left off." + ), + }) + + return ResumeResult( + success=True, + messages=messages, + context_data=checkpoint.context_data, + drift_warnings=tuple(drift_warnings), + ) diff --git a/src/leapflow/engine/recovery_coordinator.py b/src/leapflow/engine/recovery_coordinator.py new file mode 100644 index 0000000..5858bf9 --- /dev/null +++ b/src/leapflow/engine/recovery_coordinator.py @@ -0,0 +1,336 @@ +"""Recovery coordinator — unified recovery decision entry point for the agent loop. + +Replaces the scattered if/elif chains in _handle_api_error() and the inline +break/continue decisions in _unified_tool_loop() with a strategy-based, +budget-aware, auditable coordinator. +""" +from __future__ import annotations + +import logging +import time +import uuid +from dataclasses import dataclass, field +from typing import Any, Protocol, runtime_checkable + +from leapflow.engine.failure_envelope import FailureEnvelope, Recoverability +from leapflow.engine.oneshot_guard import OneShotGuard +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + +logger = logging.getLogger(__name__) + + +@dataclass +class RecoveryState: + """Shared mutable state visible to all strategies during a turn. + + Strategies may read and update this state to coordinate across multiple + failures within the same turn (e.g. tracking consecutive failures to + escalate from retry to halt). + """ + + consecutive_failures: int = 0 + consecutive_api_errors: int = 0 + last_error_category: str = "" + compress_phase_index: int = 0 + last_failure_timestamp: float = 0.0 + total_decisions: int = 0 + current_turn_id: int = 0 + + +@runtime_checkable +class RecoveryStrategy(Protocol): + """Protocol for pluggable recovery strategies. + + Strategies are registered with the coordinator and evaluated in priority + order. Each strategy encapsulates domain logic for a specific failure + pattern (e.g. "retry on rate limit", "compress on context overflow"). + """ + + @property + def key(self) -> str: + """Unique identifier for this strategy (used by OneShotGuard).""" + ... + + @property + def priority(self) -> int: + """Lower number = higher priority. Evaluated first.""" + ... + + @property + def repeatable(self) -> bool: + """Whether this strategy can fire multiple times per turn. + + Repeatable strategies manage their own deduplication (e.g. phase index) + and are NOT guarded by the OneShotGuard. Non-repeatable strategies + fire at most once per turn and are automatically one-shot guarded. + """ + ... + + @property + def applicable_sources(self) -> frozenset[str]: + """Set of FailureSource values this strategy handles. Empty = all.""" + ... + + @property + def applicable_categories(self) -> frozenset[str]: + """Set of error category strings this strategy handles. Empty = all.""" + ... + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Whether this strategy can handle the given failure envelope.""" + ... + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Produce a recovery decision for the given failure.""" + ... + + +class RecoveryCoordinator: + """Unified recovery decision entry point for the agent loop. + + Orchestrates strategy evaluation, one-shot enforcement, budget tracking, + and audit logging. The agent loop calls evaluate() on each failure and + receives a deterministic RecoveryDecision. + """ + + def __init__( + self, + strategies: list[RecoveryStrategy] | None = None, + budget: RecoveryBudget | None = None, + ) -> None: + self._strategies: list[RecoveryStrategy] = sorted( + strategies or [], key=lambda s: s.priority + ) + self._budget = budget or RecoveryBudget() + self._guard = OneShotGuard() + self._state = RecoveryState() + self._audit: list[dict[str, Any]] = [] + + def evaluate(self, envelope: FailureEnvelope) -> RecoveryDecision: + """Main entry point: evaluate a failure and return a recovery decision. + + Pipeline: + 1. Check budget deadline — if exceeded, emit terminal decision. + 2. Iterate strategies by priority. + 3. For each: check source/category match → one-shot guard → can_apply → decide. + 4. If none applies: return terminal decision. + """ + # Deadline check + if self._budget.is_deadline_exceeded(): + reason = ( + f"Turn deadline exceeded ({self._budget.turn_deadline_s}s). " + f"Strategies attempted: {', '.join(self._guard.used_strategies()) or 'none'}." + ) + decision = self._make_terminal(envelope, reason=reason) + self._record_audit(decision, strategy_key="") + return decision + + # Global budget exhaustion + if self._budget.remaining() == 0: + reason = ( + f"Recovery budget exhausted ({self._budget._consumed}/{self._budget.total_recovery_actions}). " + f"Strategies attempted: {', '.join(self._guard.used_strategies()) or 'none'}. " + f"Last error: {envelope.category}/{envelope.failure_code}." + ) + decision = self._make_terminal(envelope, reason=reason) + self._record_audit(decision, strategy_key="") + return decision + + # Non-recoverable failures short-circuit + if envelope.recoverability == Recoverability.NON_RECOVERABLE: + reason = ( + f"Failure is non-recoverable: {envelope.message[:100]}. " + f"Category: {envelope.category}, code: {envelope.failure_code}." + ) + decision = self._make_terminal(envelope, reason=reason) + self._record_audit(decision, strategy_key="") + return decision + + # Strategy evaluation + for strategy in self._strategies: + if not self._matches_source(strategy, envelope): + continue + if not self._matches_category(strategy, envelope): + continue + # One-shot guard: only check for non-repeatable strategies + if not strategy.repeatable and not self._guard.is_available(strategy.key): + continue + if not strategy.can_apply(envelope, self._state, self._budget): + continue + + # Budget pre-check for the strategy's expected cost + decision = strategy.decide(envelope, self._state) + if decision.budget_cost > 0: + if not self._budget.can_afford(decision.budget_cost, envelope.category): + # Rollback any state mutations from decide() + self._rollback_state_changes(decision, strategy) + continue + + # Commit the decision + if decision.budget_cost > 0: + self._budget.consume(decision.budget_cost, envelope.category) + + # Commit state changes AFTER decision is accepted + self._commit_state_changes(decision, strategy) + + # Track type-specific budget counters + self._consume_type_budget(decision) + + # Mark one-shot guard for non-repeatable strategies + if not strategy.repeatable: + self._guard.mark_used(strategy.key) + + self._update_state(envelope, decision) + self._record_audit(decision, strategy_key=strategy.key) + return decision + + # No strategy matched + decision = self.terminal_decision(envelope) + self._record_audit(decision, strategy_key="") + return decision + + def on_strategy_outcome(self, decision_id: str, success: bool) -> None: + """Record the outcome of executing a recovery decision. + + Updates consecutive failure counters and appends to audit log. + """ + if success: + self._state.consecutive_failures = 0 + self._state.consecutive_api_errors = 0 + else: + self._state.consecutive_failures += 1 + + self._audit.append({ + "event": "strategy_outcome", + "decision_id": decision_id, + "success": success, + "timestamp": time.time(), + "consecutive_failures": self._state.consecutive_failures, + }) + + def record_success(self) -> None: + """Record a successful operation (resets failure counts).""" + self._state.consecutive_failures = 0 + self._state.consecutive_api_errors = 0 + self._state.last_error_category = "" + + def terminal_decision(self, envelope: FailureEnvelope) -> RecoveryDecision: + """Generate a terminal (halt) decision when no strategy applies.""" + return self._make_terminal( + envelope, + reason="No applicable recovery strategy found", + ) + + @property + def audit_log(self) -> list[dict[str, Any]]: + """Return the decision audit trail for this turn.""" + return list(self._audit) + + @property + def state(self) -> RecoveryState: + """Access shared recovery state.""" + return self._state + + @property + def budget(self) -> RecoveryBudget: + """Access the recovery budget.""" + return self._budget + + @property + def guard(self) -> OneShotGuard: + """Access the one-shot guard (primarily for testing/introspection).""" + return self._guard + + def new_turn(self, turn_id: int = 0) -> None: + """Reset per-turn state for a new turn. Called at turn start.""" + self._guard.new_turn() + self._budget.new_turn() + self._state.compress_phase_index = 0 + self._state.consecutive_failures = 0 + self._state.consecutive_api_errors = 0 + self._state.current_turn_id = turn_id + + # ------------------------------------------------------------------ + # Private helpers + # ------------------------------------------------------------------ + + @staticmethod + def _matches_source(strategy: RecoveryStrategy, envelope: FailureEnvelope) -> bool: + """Check if the strategy accepts the envelope's failure source.""" + sources = strategy.applicable_sources + if not sources: + return True + return envelope.source.value in sources + + @staticmethod + def _matches_category(strategy: RecoveryStrategy, envelope: FailureEnvelope) -> bool: + """Check if the strategy accepts the envelope's error category.""" + categories = strategy.applicable_categories + if not categories: + return True + return envelope.category in categories + + def _update_state(self, envelope: FailureEnvelope, decision: RecoveryDecision) -> None: + """Update shared state after a decision is committed.""" + self._state.last_error_category = envelope.category + self._state.last_failure_timestamp = envelope.timestamp + self._state.total_decisions += 1 + if envelope.source.value == "llm": + self._state.consecutive_api_errors += 1 + + def _commit_state_changes(self, decision: RecoveryDecision, strategy: RecoveryStrategy) -> None: + """Commit strategy-specific state changes after a decision is accepted.""" + # ContextCompressStrategy phase advancement + if strategy.key == "context_compress": + phase_index = decision.audit_metadata_dict.get("phase_index") + if phase_index is not None: + self._state.compress_phase_index = phase_index + 1 + + def _rollback_state_changes(self, decision: RecoveryDecision, strategy: RecoveryStrategy) -> None: + """Rollback state mutations from decide() when a decision is rejected.""" + # ContextCompressStrategy phase rollback + if strategy.key == "context_compress": + phase_index = decision.audit_metadata_dict.get("phase_index") + if phase_index is not None: + self._state.compress_phase_index = phase_index + + def _make_terminal(self, envelope: FailureEnvelope, *, reason: str) -> RecoveryDecision: + """Create a HALT_CLEAN terminal decision.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.HALT_CLEAN, + reason=reason, + strategy_key="", + retry_semantics=RetrySemantics(consumes_retry_budget=False), + budget_cost=0, + ) + + def _consume_type_budget(self, decision: RecoveryDecision) -> None: + """Consume type-specific budget counters based on action type.""" + if decision.action == RecoveryAction.TRANSFORM_AND_RETRY: + self._budget.consume_transform() + elif decision.action == RecoveryAction.FAILOVER: + if "credential" in decision.strategy_key: + self._budget.consume_rotation() + else: + self._budget.consume_failover() + + def _record_audit(self, decision: RecoveryDecision, *, strategy_key: str) -> None: + """Append a decision record to the audit log.""" + self._audit.append({ + "event": "recovery_decision", + "decision_id": decision.decision_id, + "envelope_id": decision.envelope.envelope_id, + "action": decision.action.value, + "strategy_key": strategy_key, + "reason": decision.reason, + "budget_cost": decision.budget_cost, + "budget_remaining": self._budget.remaining(), + "timestamp": time.time(), + }) diff --git a/src/leapflow/engine/recovery_decision.py b/src/leapflow/engine/recovery_decision.py new file mode 100644 index 0000000..e6114ae --- /dev/null +++ b/src/leapflow/engine/recovery_decision.py @@ -0,0 +1,123 @@ +"""Recovery decision types for the agent loop recovery subsystem. + +A RecoveryDecision encapsulates what the coordinator decided to do about a +FailureEnvelope: which action to take, the retry semantics, the strategy +that produced the decision, and audit metadata. +""" +from __future__ import annotations + +import uuid +from dataclasses import dataclass, field +from enum import Enum +from typing import TYPE_CHECKING, Any + +from leapflow.engine.failure_envelope import FailureEnvelope + +if TYPE_CHECKING: + from leapflow.engine.interaction_request import InteractionRequest + + +class RecoveryAction(Enum): + """Possible recovery actions the coordinator can prescribe.""" + + RETRY_WITH_BACKOFF = "retry_with_backoff" + TRANSFORM_AND_RETRY = "transform_and_retry" + FAILOVER = "failover" + HALT_CLEAN = "halt_clean" + HALT_WITH_CHECKPOINT = "halt_with_checkpoint" + ASK_USER = "ask_user" + SKIP_AND_CONTINUE = "skip_and_continue" + + +@dataclass(frozen=True) +class BackoffConfig: + """Backoff algorithm parameters for retry decisions.""" + + base_delay: float = 1.0 + max_delay: float = 60.0 + jitter_ratio: float = 0.5 + algorithm: str = "decorrelated_jitter" + + +@dataclass(frozen=True) +class RetrySemantics: + """Describes how a retry interacts with the recovery budget.""" + + consumes_retry_budget: bool = True + resets_retry_count: bool = False + backoff_config: BackoffConfig | None = None + + +@dataclass(frozen=True) +class RecoveryDecision: + """Immutable record of a recovery decision. + + Produced by RecoveryCoordinator.evaluate() and consumed by the agent loop + to determine the next control-flow action (retry, halt, ask, skip). + """ + + decision_id: str + envelope: FailureEnvelope + action: RecoveryAction + reason: str + strategy_key: str + retry_semantics: RetrySemantics = field(default_factory=RetrySemantics) + budget_cost: int = 0 + audit_metadata: tuple[tuple[str, Any], ...] = () + interaction: InteractionRequest | None = None + transform_description: str = "" + failover_target: str = "" + + @classmethod + def create( + cls, + *, + envelope: FailureEnvelope, + action: RecoveryAction, + reason: str, + strategy_key: str, + retry_semantics: RetrySemantics | None = None, + budget_cost: int = 0, + audit_metadata: dict[str, Any] | None = None, + interaction: InteractionRequest | None = None, + transform_description: str = "", + failover_target: str = "", + ) -> RecoveryDecision: + """Factory that auto-generates decision_id and normalizes audit_metadata.""" + meta_tuple = tuple(sorted((audit_metadata or {}).items())) + return cls( + decision_id=uuid.uuid4().hex, + envelope=envelope, + action=action, + reason=reason, + strategy_key=strategy_key, + retry_semantics=retry_semantics or RetrySemantics(), + budget_cost=budget_cost, + audit_metadata=meta_tuple, + interaction=interaction, + transform_description=transform_description, + failover_target=failover_target, + ) + + @property + def audit_metadata_dict(self) -> dict[str, Any]: + """Reconstruct audit_metadata as a dict for serialization.""" + return dict(self.audit_metadata) + + @property + def is_terminal(self) -> bool: + """Whether this decision ends the current turn.""" + return self.action in ( + RecoveryAction.HALT_CLEAN, + RecoveryAction.HALT_WITH_CHECKPOINT, + RecoveryAction.ASK_USER, + ) + + @property + def is_retry(self) -> bool: + """Whether this decision involves retrying the failed operation.""" + return self.action in ( + RecoveryAction.RETRY_WITH_BACKOFF, + RecoveryAction.TRANSFORM_AND_RETRY, + RecoveryAction.FAILOVER, + ) diff --git a/src/leapflow/engine/recovery_strategies/__init__.py b/src/leapflow/engine/recovery_strategies/__init__.py new file mode 100644 index 0000000..8ddee8e --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/__init__.py @@ -0,0 +1,42 @@ +"""Built-in recovery strategies for the agent loop recovery coordinator. + +Each strategy implements the RecoveryStrategy Protocol and encapsulates +a single-responsibility recovery pattern (context compression, failover, +credential rotation, etc.). +""" +from __future__ import annotations + +from leapflow.engine.recovery_strategies.context_compress import ContextCompressStrategy +from leapflow.engine.recovery_strategies.credential_rotate import CredentialRotateStrategy +from leapflow.engine.recovery_strategies.jittered_retry import JitteredRetryStrategy +from leapflow.engine.recovery_strategies.multimodal_strip import MultimodalStripStrategy +from leapflow.engine.recovery_strategies.native_to_text import NativeToTextFallbackStrategy +from leapflow.engine.recovery_strategies.provider_failover import ProviderFailoverStrategy +from leapflow.engine.recovery_strategies.thinking_disable import ThinkingDisableStrategy +from leapflow.engine.recovery_strategies.tool_schema_expand import ToolSchemaExpandStrategy + +__all__ = [ + "ContextCompressStrategy", + "CredentialRotateStrategy", + "JitteredRetryStrategy", + "MultimodalStripStrategy", + "NativeToTextFallbackStrategy", + "ProviderFailoverStrategy", + "ThinkingDisableStrategy", + "ToolSchemaExpandStrategy", + "default_strategies", +] + + +def default_strategies() -> list: + """Return all built-in strategies in priority order (lowest priority number first).""" + return [ + ContextCompressStrategy(), + MultimodalStripStrategy(), + ProviderFailoverStrategy(), + CredentialRotateStrategy(), + ThinkingDisableStrategy(), + NativeToTextFallbackStrategy(), + ToolSchemaExpandStrategy(), + JitteredRetryStrategy(), + ] diff --git a/src/leapflow/engine/recovery_strategies/context_compress.py b/src/leapflow/engine/recovery_strategies/context_compress.py new file mode 100644 index 0000000..ca73558 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/context_compress.py @@ -0,0 +1,76 @@ +"""Context compression recovery strategy. + +Handles context overflow errors by progressively compressing the conversation +context through three phases: history summarization, multimodal-to-text +conversion, and disclosure shrinkage. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + +_PHASES = ( + "history_summarize", + "multimodal_to_text", + "disclosure_shrink", +) + + +class ContextCompressStrategy: + """Progressive context compression for context overflow errors. + + Three compression phases are applied in order: + 1. history_summarize — Summarize older conversation history + 2. multimodal_to_text — Convert multimodal content to text descriptions + 3. disclosure_shrink — Reduce progressive disclosure payload size + """ + + @property + def key(self) -> str: + return "context_compress" + + @property + def priority(self) -> int: + return 10 + + @property + def repeatable(self) -> bool: + return True + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"context_overflow", "payload_too_large"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable if there are remaining compression phases.""" + return state.compress_phase_index < len(_PHASES) + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return TRANSFORM_AND_RETRY with the current compression phase.""" + phase_idx = state.compress_phase_index + phase_name = _PHASES[phase_idx] + + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason=f"Context overflow: applying compression phase '{phase_name}' " + f"({phase_idx + 1}/{len(_PHASES)})", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=False, + resets_retry_count=False, + ), + budget_cost=0, + audit_metadata={"phase": phase_name, "phase_index": phase_idx}, + ) diff --git a/src/leapflow/engine/recovery_strategies/credential_rotate.py b/src/leapflow/engine/recovery_strategies/credential_rotate.py new file mode 100644 index 0000000..3b19877 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/credential_rotate.py @@ -0,0 +1,66 @@ +"""Credential rotation recovery strategy. + +Handles auth errors and rate limiting by rotating to alternate credentials +(API keys, tokens) from the credential pool. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class CredentialRotateStrategy: + """Rotate credentials on authentication or rate-limit failures. + + When an API key is invalid, expired, or rate-limited, this strategy + attempts to switch to alternate credentials from the configured pool. + Credential rotation is a form of failover at the authentication level. + """ + + @property + def key(self) -> str: + return "credential_rotate" + + @property + def priority(self) -> int: + return 25 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"auth_error", "rate_limited", "billing"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable when credential rotation budget remains.""" + if budget is not None and not budget.can_rotate(): + return False + return True + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return FAILOVER decision for credential rotation.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.FAILOVER, + reason=f"Credential issue ({envelope.category}): rotating to alternate credentials", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=True, + resets_retry_count=True, + ), + budget_cost=1, + audit_metadata={"trigger_category": envelope.category}, + ) diff --git a/src/leapflow/engine/recovery_strategies/jittered_retry.py b/src/leapflow/engine/recovery_strategies/jittered_retry.py new file mode 100644 index 0000000..6187381 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/jittered_retry.py @@ -0,0 +1,107 @@ +"""Jittered retry recovery strategy. + +The lowest-priority catch-all retry strategy for transient failures. +Applies decorrelated jitter backoff with category-specific delay parameters. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope, Recoverability +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + BackoffConfig, + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + +# Category-specific backoff configurations +_BACKOFF_CONFIGS: dict[str, BackoffConfig] = { + "rate_limited": BackoffConfig( + base_delay=5.0, + max_delay=120.0, + jitter_ratio=0.5, + algorithm="decorrelated_jitter", + ), + "transient": BackoffConfig( + base_delay=1.0, + max_delay=60.0, + jitter_ratio=0.5, + algorithm="decorrelated_jitter", + ), + "overloaded": BackoffConfig( + base_delay=1.0, + max_delay=60.0, + jitter_ratio=0.5, + algorithm="decorrelated_jitter", + ), + "tool_timeout": BackoffConfig( + base_delay=2.0, + max_delay=30.0, + jitter_ratio=0.5, + algorithm="decorrelated_jitter", + ), +} + +_DEFAULT_BACKOFF = BackoffConfig( + base_delay=1.0, + max_delay=60.0, + jitter_ratio=0.5, + algorithm="decorrelated_jitter", +) + + +class JitteredRetryStrategy: + """Catch-all retry with decorrelated jitter backoff for transient failures. + + This is the lowest-priority strategy that handles any AUTO_RETRY-recoverable + failure with exponential backoff and jitter. It consumes retry budget and + uses category-specific delay parameters. + """ + + @property + def key(self) -> str: + return "jittered_retry" + + @property + def priority(self) -> int: + return 100 + + @property + def repeatable(self) -> bool: + return True + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm", "tool", "system"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset() # Empty = wildcard, matches all categories + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable for AUTO_RETRY failures when budget allows.""" + return envelope.recoverability == Recoverability.AUTO_RETRY + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return RETRY_WITH_BACKOFF with category-appropriate backoff config.""" + backoff = _BACKOFF_CONFIGS.get(envelope.category, _DEFAULT_BACKOFF) + + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.RETRY_WITH_BACKOFF, + reason=f"Transient failure ({envelope.category}): retrying with jittered backoff", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=True, + resets_retry_count=False, + backoff_config=backoff, + ), + budget_cost=1, + audit_metadata={ + "category": envelope.category, + "base_delay": backoff.base_delay, + "max_delay": backoff.max_delay, + }, + ) diff --git a/src/leapflow/engine/recovery_strategies/multimodal_strip.py b/src/leapflow/engine/recovery_strategies/multimodal_strip.py new file mode 100644 index 0000000..9c4d819 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/multimodal_strip.py @@ -0,0 +1,63 @@ +"""Multimodal strip recovery strategy. + +Handles image-too-large errors by stripping multimodal content and converting +to text descriptions before retry. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class MultimodalStripStrategy: + """Strip multimodal content when images are too large for the provider. + + Converts images and other multimodal content to text descriptions, + allowing the request to proceed within provider size limits. + """ + + @property + def key(self) -> str: + return "multimodal_strip" + + @property + def priority(self) -> int: + return 15 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"image_too_large"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable when the failure message indicates an image size issue.""" + msg = envelope.message.lower() + return "image" in msg or "large" in msg or "size" in msg or envelope.category == "image_too_large" + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return TRANSFORM_AND_RETRY to strip multimodal content.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Image too large: stripping multimodal content to text descriptions", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=False, + resets_retry_count=False, + ), + budget_cost=0, + ) diff --git a/src/leapflow/engine/recovery_strategies/native_to_text.py b/src/leapflow/engine/recovery_strategies/native_to_text.py new file mode 100644 index 0000000..9ae2368 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/native_to_text.py @@ -0,0 +1,75 @@ +"""Native-to-text fallback recovery strategy. + +Handles format errors by falling back from native tool calling mode to +text-based tool calling when the native mode produces parsing conflicts. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class NativeToTextFallbackStrategy: + """Fall back from native tool calling to text-based tool calling. + + When native (function calling) mode produces format errors due to + provider limitations or model incompatibilities, this strategy switches + to text-mode tool calling where tool calls are parsed from LLM text output. + """ + + @property + def key(self) -> str: + return "native_to_text" + + @property + def priority(self) -> int: + return 35 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"format_error"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable when native tool mode is indicated by failure context. + + Checks for indicators that native tool calling mode is active: + - Message mentions 'tool_call', 'function_call', or 'native' + - The failure indicates a tool calling parse/format issue + """ + msg = envelope.message.lower() + return ( + "tool_call" in msg + or "function_call" in msg + or "native" in msg + or "tool_use" in msg + or "tools" in msg + ) + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return TRANSFORM_AND_RETRY to switch from native to text tool mode.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Fall back from native tool calling to text mode", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=False, + resets_retry_count=False, + ), + budget_cost=0, + ) diff --git a/src/leapflow/engine/recovery_strategies/provider_failover.py b/src/leapflow/engine/recovery_strategies/provider_failover.py new file mode 100644 index 0000000..e39ae8a --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/provider_failover.py @@ -0,0 +1,66 @@ +"""Provider failover recovery strategy. + +Handles permanent provider failures (billing, auth permanent, overloaded, +model not found, content blocked) by switching to an alternate provider/model. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class ProviderFailoverStrategy: + """Failover to alternate provider/model on non-transient provider errors. + + Triggers when the current provider is unable to serve requests due to + billing, permanent auth failures, capacity issues, model unavailability, + or content policy violations. + """ + + @property + def key(self) -> str: + return "provider_failover" + + @property + def priority(self) -> int: + return 20 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"billing", "auth_permanent", "overloaded", "model_not_found", "content_blocked"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable if failover budget remains.""" + if budget is not None and not budget.can_failover(): + return False + return True + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return FAILOVER decision to switch provider/model.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.FAILOVER, + reason=f"Provider failure ({envelope.category}): failing over to alternate provider/model", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=True, + resets_retry_count=True, + ), + budget_cost=1, + audit_metadata={"trigger_category": envelope.category}, + ) diff --git a/src/leapflow/engine/recovery_strategies/thinking_disable.py b/src/leapflow/engine/recovery_strategies/thinking_disable.py new file mode 100644 index 0000000..8aeec93 --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/thinking_disable.py @@ -0,0 +1,63 @@ +"""Thinking mode disable recovery strategy. + +Handles format errors by disabling the LLM's thinking/reasoning mode +which can conflict with certain output format expectations. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class ThinkingDisableStrategy: + """Disable thinking mode to resolve format conflicts. + + Some LLM providers produce format errors when thinking/reasoning mode + is active alongside structured output requirements. This strategy + disables thinking mode as a transform-and-retry action. + """ + + @property + def key(self) -> str: + return "thinking_disable" + + @property + def priority(self) -> int: + return 30 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"llm"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"format_error"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Always applicable -- the one-shot guard handles dedup at coordinator level.""" + return True + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return TRANSFORM_AND_RETRY to disable thinking mode.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Disable thinking mode to resolve format conflict", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=False, + resets_retry_count=False, + ), + budget_cost=0, + ) diff --git a/src/leapflow/engine/recovery_strategies/tool_schema_expand.py b/src/leapflow/engine/recovery_strategies/tool_schema_expand.py new file mode 100644 index 0000000..ff7ed7d --- /dev/null +++ b/src/leapflow/engine/recovery_strategies/tool_schema_expand.py @@ -0,0 +1,64 @@ +"""Tool schema expansion recovery strategy. + +Handles unknown tool errors by expanding the tool schema to include +additional tools that the LLM attempted to call but were not in scope. +""" +from __future__ import annotations + +from leapflow.engine.failure_envelope import FailureEnvelope, Recoverability +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import RecoveryState +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +class ToolSchemaExpandStrategy: + """Expand tool schema when an unknown tool call is auto-recoverable. + + When the LLM produces a tool call for a tool not in the current schema + but marked as retryable (indicating the tool exists but wasn't disclosed), + this strategy expands the schema and retries. + """ + + @property + def key(self) -> str: + return "tool_schema_expand" + + @property + def priority(self) -> int: + return 40 + + @property + def repeatable(self) -> bool: + return False + + @property + def applicable_sources(self) -> frozenset[str]: + return frozenset({"tool"}) + + @property + def applicable_categories(self) -> frozenset[str]: + return frozenset({"tool_unknown"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + """Applicable when the tool error is marked as auto-recoverable.""" + return envelope.recoverability == Recoverability.AUTO_RECOVER + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + """Return TRANSFORM_AND_RETRY to expand the tool schema.""" + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Expand tool schema for unknown tool recovery", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=False, + resets_retry_count=False, + ), + budget_cost=0, + audit_metadata={"tool_name": envelope.context.tool_name}, + ) diff --git a/src/leapflow/engine/tool_concurrency.py b/src/leapflow/engine/tool_concurrency.py index dad6960..6b1e280 100644 --- a/src/leapflow/engine/tool_concurrency.py +++ b/src/leapflow/engine/tool_concurrency.py @@ -68,6 +68,7 @@ def partition( # rest of the turn). _DEFAULT_NEVER_PARALLEL: FrozenSet[str] = frozenset({ "gp_shell_run", "shell_run", + "gp_scm_sync", "scm_sync", "clarify", "gp_clarify", "delegate_task", "gp_delegate_task", "memory_add", "gp_memory_add", diff --git a/src/leapflow/engine/tool_execution.py b/src/leapflow/engine/tool_execution.py new file mode 100644 index 0000000..39db700 --- /dev/null +++ b/src/leapflow/engine/tool_execution.py @@ -0,0 +1,263 @@ +"""Tool execution identity, policy, and idempotency ledger.""" +from __future__ import annotations + +import asyncio +import hashlib +import json +import time +import uuid +from dataclasses import dataclass, replace +from typing import Any, Literal, Mapping + +ExecutionPolicy = Literal["read_only", "mutating_idempotent", "mutating_once", "external_side_effect"] +ExecutionStatus = Literal["reserved", "running", "completed", "failed_retryable", "failed_final"] + +_EXTERNAL_TOOLS = frozenset({ + "shell_run", + "scm_sync", + "gateway_send", + "gateway_connect", + "platform_action", + "platform_connect", + "hub_push", + "hub_pull", + "hub_sync", +}) + + +def canonical_json(value: Any) -> str: + """Return deterministic JSON for execution identity keys.""" + return json.dumps(value or {}, sort_keys=True, ensure_ascii=False, default=str, separators=(",", ":")) + + +def execution_policy_for(tool_name: str, spec: Any | None = None) -> ExecutionPolicy: + """Classify a tool into an idempotency policy using registry metadata.""" + name = str(tool_name or "").removeprefix("gp_") + risk_level = str(getattr(spec, "risk_level", "") or "") + mutates_state = bool(getattr(spec, "mutates_state", False)) + idempotency_scope = str(getattr(spec, "idempotency_scope", "") or "") + effect_scope = str(getattr(spec, "effect_scope", "") or "") + if risk_level == "read_only" and not mutates_state: + return "read_only" + if name in _EXTERNAL_TOOLS or risk_level == "external" or effect_scope == "external": + return "external_side_effect" + if idempotency_scope == "session": + return "mutating_once" + return "mutating_idempotent" + + +def build_idempotency_key( + *, + session_id: str, + turn_id: str, + tool_name: str, + arguments: Mapping[str, Any] | None, + policy: ExecutionPolicy, +) -> str: + """Build a stable system-owned idempotency key for a tool execution.""" + scope = "turn" if policy in {"read_only", "mutating_idempotent"} else "session" + payload = { + "session_id": session_id, + "turn_id": turn_id if scope == "turn" else "", + "scope": scope, + "tool_name": str(tool_name or "").removeprefix("gp_"), + "arguments": arguments or {}, + "policy": policy, + } + return hashlib.sha256(canonical_json(payload).encode("utf-8")).hexdigest() + + +@dataclass(frozen=True) +class ToolExecutionRecord: + """Immutable execution ledger row.""" + + execution_id: str + session_id: str + turn_id: str + command_id: str + tool_call_id: str + tool_name: str + idempotency_key: str + arguments: dict[str, Any] + policy: ExecutionPolicy + status: ExecutionStatus + result: Any = None + created_at: float = 0.0 + completed_at: float = 0.0 + + @classmethod + def create( + cls, + *, + session_id: str, + turn_id: str, + command_id: str, + tool_call_id: str, + tool_name: str, + idempotency_key: str, + arguments: Mapping[str, Any] | None, + policy: ExecutionPolicy, + ) -> "ToolExecutionRecord": + return cls( + execution_id=uuid.uuid4().hex, + session_id=session_id, + turn_id=turn_id, + command_id=command_id, + tool_call_id=tool_call_id, + tool_name=tool_name, + idempotency_key=idempotency_key, + arguments=dict(arguments or {}), + policy=policy, + status="running", + created_at=time.time(), + ) + + def mark_completed(self, result: Any) -> "ToolExecutionRecord": + return replace(self, status="completed", result=result, completed_at=time.time()) + + def mark_failed(self, result: Any, *, retryable: bool) -> "ToolExecutionRecord": + return replace( + self, + status="failed_retryable" if retryable else "failed_final", + result=result, + completed_at=time.time(), + ) + + +class ToolExecutionLedger: + """Turn-scope idempotency ledger with optional durable backing store.""" + + def __init__(self, *, store: Any | None = None) -> None: + self._records: dict[str, ToolExecutionRecord] = {} + self._inflight: dict[str, asyncio.Future[ToolExecutionRecord]] = {} + self._store = store + + def reset(self, *, store: Any | None = None) -> None: + self._records.clear() + self._inflight.clear() + self._store = store + + def reserve( + self, + *, + session_id: str, + turn_id: str, + command_id: str, + tool_call_id: str, + tool_name: str, + arguments: Mapping[str, Any] | None, + policy: ExecutionPolicy, + ) -> tuple[ToolExecutionRecord, ToolExecutionRecord | None]: + """Reserve an execution or return the original record for duplicates.""" + key = build_idempotency_key( + session_id=session_id, + turn_id=turn_id, + tool_name=tool_name, + arguments=arguments, + policy=policy, + ) + if policy != "read_only": + existing = self._records.get(key) or self._get_durable(session_id, key) + if existing is not None: + return existing, existing + record = ToolExecutionRecord.create( + session_id=session_id, + turn_id=turn_id, + command_id=command_id, + tool_call_id=tool_call_id, + tool_name=tool_name, + idempotency_key=key, + arguments=arguments, + policy=policy, + ) + self._records[key] = record + if policy != "read_only": + self._inflight[key] = asyncio.get_running_loop().create_future() + self._reserve_durable(record) + return record, None + + async def wait_for_completion( + self, + record: ToolExecutionRecord, + *, + timeout_s: float, + ) -> ToolExecutionRecord: + """Wait for an in-flight duplicate's original execution to finish.""" + future = self._inflight.get(record.idempotency_key) + if future is None: + return self._records.get(record.idempotency_key, record) + try: + return await asyncio.wait_for(asyncio.shield(future), timeout=max(1.0, timeout_s)) + except TimeoutError: + current = self._records.get(record.idempotency_key, record) + return current.mark_failed( + { + "ok": False, + "error": "Original tool execution is still running.", + "retryable": True, + "already_executed": True, + "duplicate_suppressed": True, + "counts_as_failure": False, + "counts_as_tool_attempt": False, + "ui_hidden": True, + }, + retryable=True, + ) + + def complete(self, record: ToolExecutionRecord, result: Any) -> ToolExecutionRecord: + retryable = bool(result.get("retryable")) if isinstance(result, dict) else False + ok = bool(result.get("ok", True)) if isinstance(result, dict) else True + updated = record.mark_completed(result) if ok else record.mark_failed(result, retryable=retryable) + self._records[record.idempotency_key] = updated + future = self._inflight.pop(record.idempotency_key, None) + if future is not None and not future.done(): + future.set_result(updated) + self._complete_durable(updated) + return updated + + @staticmethod + def duplicate_result(record: ToolExecutionRecord) -> dict[str, Any]: + """Return a structured payload for a suppressed duplicate side effect.""" + completed = record.status == "completed" + payload: dict[str, Any] = { + "ok": completed, + "already_executed": True, + "duplicate_suppressed": True, + "execution_reused": completed, + "execution_skipped": not completed, + "counts_as_failure": False, + "counts_as_tool_attempt": False, + "ui_hidden": True, + "execution_id": record.execution_id, + "idempotency_key": record.idempotency_key, + "execution_status": record.status, + "original_result": record.result, + "retryable": record.status == "failed_retryable", + } + if not completed: + payload["error"] = "An identical side-effect attempt is already recorded. Review the original result before retrying." + return payload + + def _get_durable(self, session_id: str, key: str) -> ToolExecutionRecord | None: + if self._store is None or not hasattr(self._store, "get_tool_execution_by_key"): + return None + try: + return self._store.get_tool_execution_by_key(session_id, key) + except Exception: + return None + + def _reserve_durable(self, record: ToolExecutionRecord) -> None: + if self._store is None or not hasattr(self._store, "reserve_tool_execution"): + return + try: + self._store.reserve_tool_execution(record) + except Exception: + pass + + def _complete_durable(self, record: ToolExecutionRecord) -> None: + if self._store is None or not hasattr(self._store, "complete_tool_execution"): + return + try: + self._store.complete_tool_execution(record) + except Exception: + pass diff --git a/src/leapflow/engine/unified_classifier.py b/src/leapflow/engine/unified_classifier.py new file mode 100644 index 0000000..c9c55c4 --- /dev/null +++ b/src/leapflow/engine/unified_classifier.py @@ -0,0 +1,298 @@ +"""Unified error classifier bridging LLM, tool, and system failures into FailureEnvelope. + +Provides a single classification entry point that produces FailureEnvelope instances +from any error source. Wraps the existing ErrorClassifier for LLM errors and adds +structured classification for tool results and system exceptions. +""" +from __future__ import annotations + +import logging +from typing import Any + +from leapflow.engine.error_classifier import ErrorCategory, ErrorClassifier +from leapflow.engine.failure_envelope import ( + FailureContext, + FailureEnvelope, + FailureSource, + Recoverability, + RecoveryHint, + SideEffectState, +) + +logger = logging.getLogger(__name__) + +# Default mapping from ErrorCategory to (Recoverability, default_failure_class) +_DEFAULT_MAPPINGS: dict[str, tuple[Recoverability, str]] = { + ErrorCategory.TRANSIENT.value: (Recoverability.AUTO_RETRY, "transient"), + ErrorCategory.RATE_LIMITED.value: (Recoverability.AUTO_RETRY, "rate_limited"), + ErrorCategory.OVERLOADED.value: (Recoverability.AUTO_RETRY, "overloaded"), + ErrorCategory.CONTEXT_OVERFLOW.value: (Recoverability.AUTO_RECOVER, "context_overflow"), + ErrorCategory.PAYLOAD_TOO_LARGE.value: (Recoverability.AUTO_RECOVER, "payload_too_large"), + ErrorCategory.FORMAT_ERROR.value: (Recoverability.AUTO_RECOVER, "format_error"), + ErrorCategory.TOOL_FAILURE.value: (Recoverability.USER_FIXABLE, "tool_failure"), + ErrorCategory.AUTH_ERROR.value: (Recoverability.AUTO_RETRY, "auth_error"), + ErrorCategory.AUTH_PERMANENT.value: (Recoverability.NON_RECOVERABLE, "auth_permanent"), + ErrorCategory.BILLING.value: (Recoverability.USER_FIXABLE, "billing"), + ErrorCategory.CONTENT_BLOCKED.value: (Recoverability.NON_RECOVERABLE, "content_blocked"), + ErrorCategory.MODEL_NOT_FOUND.value: (Recoverability.USER_FIXABLE, "model_not_found"), + ErrorCategory.IMAGE_TOO_LARGE.value: (Recoverability.AUTO_RECOVER, "image_too_large"), + ErrorCategory.SSL_ERROR.value: (Recoverability.NON_RECOVERABLE, "ssl_error"), + ErrorCategory.PERMANENT.value: (Recoverability.NON_RECOVERABLE, "permanent"), +} + + +class RecoverabilityRegistry: + """Extensible registry for category -> recoverability mapping. + + Allows plugins and configuration to register new error categories + without modifying this module. + """ + + def __init__(self) -> None: + self._mappings: dict[str, tuple[Recoverability, str]] = dict(_DEFAULT_MAPPINGS) + + def register(self, category: str, recoverability: Recoverability, + failure_class: str = "") -> None: + """Register or override a category mapping.""" + self._mappings[category] = (recoverability, failure_class or category) + + def get(self, category: str) -> tuple[Recoverability, str]: + """Get recoverability for a category, defaulting to NON_RECOVERABLE.""" + return self._mappings.get(category, (Recoverability.NON_RECOVERABLE, "unknown")) + + def categories(self) -> list[str]: + """List all registered categories.""" + return list(self._mappings.keys()) + +# Permission failure classes that indicate non-recoverable tool permission errors +_PERMISSION_FAILURE_CLASSES = frozenset({"authorization", "scope_denied"}) +_PERMISSION_FAILURE_CODES = frozenset({"access_denied", "missing_scope", "platform_degraded"}) + + +class UnifiedErrorClassifier: + """Unified classification entry point producing FailureEnvelope from any error source. + + Bridges: + - LLM/API exceptions -> FailureEnvelope (wraps existing ErrorClassifier) + - Tool result dicts -> FailureEnvelope (new structured classification) + - System exceptions -> FailureEnvelope (new) + """ + + def __init__(self, error_classifier: Any = None, + registry: RecoverabilityRegistry | None = None) -> None: + """Accept existing ErrorClassifier instance for LLM error classification. + + Args: + error_classifier: An ErrorClassifier instance. If None, a default is created. + registry: Optional RecoverabilityRegistry for extensible category mapping. + """ + if error_classifier is not None and isinstance(error_classifier, ErrorClassifier): + self._classifier: ErrorClassifier = error_classifier + else: + self._classifier = ErrorClassifier() + self._registry = registry or RecoverabilityRegistry() + + def classify_llm_error( + self, + exc: Exception, + *, + provider: str = "", + model: str = "", + ) -> FailureEnvelope: + """Classify an LLM/API exception into a FailureEnvelope. + + Uses the existing ErrorClassifier internally to determine the category, + then maps to the appropriate recoverability and constructs a FailureEnvelope. + """ + category = self._classifier.classify(exc) + category_str = category.value + + recoverability, failure_class = self._registry.get(category_str) + + # Build recovery hint from the classifier's friendly message + friendly_msg = self._classifier.friendly_message(category, str(exc)[:200]) + hint = RecoveryHint(hint_text=friendly_msg) if friendly_msg else None + + return FailureEnvelope.create( + source=FailureSource.LLM, + category=category_str, + failure_class=failure_class, + failure_code=f"llm_{category_str}", + message=str(exc)[:500], + recoverability=recoverability, + side_effect_state=SideEffectState.NONE, + context=FailureContext.from_dict_args( + tool_name="", + arguments={"provider": provider, "model": model} if provider or model else None, + ), + provider_hint=hint, + ) + + def classify_tool_result( + self, + result: dict[str, Any], + *, + tool_name: str = "", + execution_policy: str = "read_only", + ) -> FailureEnvelope | None: + """Classify a tool result dict. Returns None if result is not a failure. + + Non-failure conditions (returns None): + - ok=True + - duplicate_suppressed=True + - counts_as_failure=False + """ + # Non-failure fast paths + if result.get("ok", True) is True: + return None + if result.get("duplicate_suppressed", False): + return None + if result.get("counts_as_failure") is False: + return None + + # Extract structured failure fields + failure_class = str(result.get("failure_class") or "") + failure_code = str(result.get("failure_code") or "") + error_msg = str(result.get("error") or "") + error_type = str(result.get("error_type") or "") + retryable = bool(result.get("retryable") if result.get("retryable") is not None else True) + + # Determine side-effect state based on execution policy + side_effect_state = self._side_effect_state_from_policy(execution_policy) + + # Classification rules (priority order) + + # 1. Permission failures + if failure_class in _PERMISSION_FAILURE_CLASSES or failure_code in _PERMISSION_FAILURE_CODES: + return FailureEnvelope.create( + source=FailureSource.TOOL, + category="tool_permission", + failure_class=failure_class or "authorization", + failure_code=failure_code or "permission_denied", + message=error_msg or "Permission denied", + recoverability=Recoverability.NON_RECOVERABLE, + side_effect_state=side_effect_state, + context=FailureContext.from_dict_args(tool_name=tool_name), + ) + + # 2. Unknown tool with retryable flag + if error_type == "unknown_tool" and retryable: + return FailureEnvelope.create( + source=FailureSource.TOOL, + category="tool_unknown", + failure_class="unknown_tool", + failure_code=failure_code or "tool_not_found", + message=error_msg or f"Unknown tool: {tool_name}", + recoverability=Recoverability.AUTO_RECOVER, + side_effect_state=SideEffectState.NONE, + context=FailureContext.from_dict_args(tool_name=tool_name), + ) + + # 3. Timeout detection + error_lower = error_msg.lower() + if "timeout" in error_lower or "timed out" in error_lower: + timeout_recoverability = ( + Recoverability.AUTO_RETRY + if execution_policy == "read_only" + else Recoverability.USER_FIXABLE + ) + return FailureEnvelope.create( + source=FailureSource.TOOL, + category="tool_timeout", + failure_class="timeout", + failure_code=failure_code or "execution_timeout", + message=error_msg, + recoverability=timeout_recoverability, + side_effect_state=side_effect_state, + context=FailureContext.from_dict_args(tool_name=tool_name), + ) + + # 4. Generic tool failure + recoverability = ( + Recoverability.AUTO_RETRY if retryable else Recoverability.USER_FIXABLE + ) + return FailureEnvelope.create( + source=FailureSource.TOOL, + category="tool_failure", + failure_class=failure_class or "tool_error", + failure_code=failure_code or "execution_failed", + message=error_msg or "Tool execution failed", + recoverability=recoverability, + side_effect_state=side_effect_state, + context=FailureContext.from_dict_args(tool_name=tool_name), + ) + + def classify_system_error(self, exc: Exception) -> FailureEnvelope: + """Classify a system-level exception (resource, timeout, etc.).""" + msg = str(exc).lower() + exc_type = type(exc).__name__ + + # Timeout errors + if isinstance(exc, (TimeoutError,)) or "timeout" in msg or "timed out" in msg: + return FailureEnvelope.create( + source=FailureSource.SYSTEM, + category="system_timeout", + failure_class="timeout", + failure_code="system_timeout", + message=str(exc)[:500], + recoverability=Recoverability.AUTO_RETRY, + side_effect_state=SideEffectState.NONE, + ) + + # Memory errors + if isinstance(exc, MemoryError): + return FailureEnvelope.create( + source=FailureSource.SYSTEM, + category="system_resource", + failure_class="memory_error", + failure_code="out_of_memory", + message=str(exc)[:500], + recoverability=Recoverability.NON_RECOVERABLE, + side_effect_state=SideEffectState.UNKNOWN, + ) + + # OS/IO errors + if isinstance(exc, OSError): + return FailureEnvelope.create( + source=FailureSource.SYSTEM, + category="system_io", + failure_class="os_error", + failure_code=f"errno_{getattr(exc, 'errno', 'unknown')}", + message=str(exc)[:500], + recoverability=Recoverability.AUTO_RETRY, + side_effect_state=SideEffectState.UNKNOWN, + ) + + # Connection errors (subclass of OSError but check explicitly) + if "connection" in msg or "refused" in msg or "reset" in msg: + return FailureEnvelope.create( + source=FailureSource.SYSTEM, + category="system_network", + failure_class="connection_error", + failure_code="connection_failed", + message=str(exc)[:500], + recoverability=Recoverability.AUTO_RETRY, + side_effect_state=SideEffectState.NONE, + ) + + # Generic system error + return FailureEnvelope.create( + source=FailureSource.SYSTEM, + category="system_unknown", + failure_class=exc_type.lower(), + failure_code="unclassified", + message=str(exc)[:500], + recoverability=Recoverability.USER_FIXABLE, + side_effect_state=SideEffectState.UNKNOWN, + ) + + @staticmethod + def _side_effect_state_from_policy(policy: str) -> SideEffectState: + """Map execution policy to the appropriate side-effect state for failures.""" + if policy == "read_only": + return SideEffectState.NONE + if policy == "external_side_effect": + return SideEffectState.UNKNOWN + if policy in ("mutating_idempotent", "mutating_once"): + return SideEffectState.PARTIAL + return SideEffectState.UNKNOWN diff --git a/src/leapflow/storage/conversation_store.py b/src/leapflow/storage/conversation_store.py index f21eca4..1532736 100644 --- a/src/leapflow/storage/conversation_store.py +++ b/src/leapflow/storage/conversation_store.py @@ -20,7 +20,10 @@ import uuid from dataclasses import dataclass, field from pathlib import Path -from typing import Any, Dict, List, Optional, Protocol, Union, runtime_checkable +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Protocol, Union, runtime_checkable + +if TYPE_CHECKING: + from leapflow.engine.tool_execution import ToolExecutionRecord logger = logging.getLogger(__name__) @@ -79,6 +82,9 @@ def create_session(self, session_id: str, *, title: str = "", **kwargs: Any) -> def get_session(self, session_id: str) -> Optional[ConversationSession]: ... def list_sessions(self, *, limit: int = 20, active_only: bool = True) -> List[ConversationSession]: ... def append_message(self, session_id: str, role: str, content: str, **kwargs: Any) -> ConversationMessage: ... + def reserve_tool_execution(self, record: "ToolExecutionRecord") -> None: ... + def complete_tool_execution(self, record: "ToolExecutionRecord") -> None: ... + def get_tool_execution_by_key(self, session_id: str, idempotency_key: str) -> Optional["ToolExecutionRecord"]: ... def get_messages(self, session_id: str, *, limit: int = 100, active_only: bool = True) -> List[ConversationMessage]: ... def search_messages(self, query: str, *, limit: int = 10) -> List[ConversationSearchResult]: ... def close(self) -> None: ... @@ -140,6 +146,32 @@ def _initialize_schema(self) -> None: metadata_json VARCHAR DEFAULT '{}' ) """) + self._conn.execute(""" + CREATE TABLE IF NOT EXISTS tool_executions ( + execution_id VARCHAR PRIMARY KEY, + session_id VARCHAR NOT NULL, + turn_id VARCHAR NOT NULL, + command_id VARCHAR DEFAULT '', + tool_call_id VARCHAR DEFAULT '', + tool_name VARCHAR NOT NULL, + idempotency_key VARCHAR NOT NULL, + arguments_json VARCHAR DEFAULT '{}', + policy VARCHAR DEFAULT 'read_only', + status VARCHAR NOT NULL, + result_json VARCHAR, + created_at DOUBLE DEFAULT 0.0, + completed_at DOUBLE DEFAULT 0.0, + UNIQUE(session_id, idempotency_key) + ) + """) + self._conn.execute(""" + CREATE INDEX IF NOT EXISTS idx_tool_executions_session + ON tool_executions (session_id, idempotency_key) + """) + self._conn.execute(""" + CREATE INDEX IF NOT EXISTS idx_tool_executions_turn + ON tool_executions (turn_id, created_at) + """) self._conn.execute(""" CREATE INDEX IF NOT EXISTS idx_conv_messages_session ON conversation_messages (session_id, created_at) @@ -278,6 +310,75 @@ def append_message( token_count=token_count, metadata=metadata or {}, ) + def reserve_tool_execution(self, record: "ToolExecutionRecord") -> None: + """Reserve a tool execution idempotently.""" + self._execute_write( + """ + INSERT INTO tool_executions + (execution_id, session_id, turn_id, command_id, tool_call_id, + tool_name, idempotency_key, arguments_json, policy, status, + result_json, created_at, completed_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT (session_id, idempotency_key) DO NOTHING + """, + [ + record.execution_id, + record.session_id, + record.turn_id, + record.command_id, + record.tool_call_id, + record.tool_name, + record.idempotency_key, + json.dumps(record.arguments, ensure_ascii=False, default=str), + record.policy, + record.status, + json.dumps(record.result, ensure_ascii=False, default=str) if record.result is not None else None, + record.created_at, + record.completed_at, + ], + ) + + def complete_tool_execution(self, record: "ToolExecutionRecord") -> None: + """Store the final status and result for a reserved tool execution.""" + self._execute_write( + """ + UPDATE tool_executions SET + status = ?, + result_json = ?, + completed_at = ?, + tool_call_id = COALESCE(NULLIF(?, ''), tool_call_id) + WHERE session_id = ? AND idempotency_key = ? + """, + [ + record.status, + json.dumps(record.result, ensure_ascii=False, default=str) if record.result is not None else None, + record.completed_at, + record.tool_call_id, + record.session_id, + record.idempotency_key, + ], + ) + + def get_tool_execution_by_key( + self, + session_id: str, + idempotency_key: str, + ) -> Optional["ToolExecutionRecord"]: + """Return a stored tool execution by session and idempotency key.""" + row = self._conn.execute( + """ + SELECT execution_id, session_id, turn_id, command_id, tool_call_id, + tool_name, idempotency_key, arguments_json, policy, status, + result_json, created_at, completed_at + FROM tool_executions + WHERE session_id = ? AND idempotency_key = ? + """, + [session_id, idempotency_key], + ).fetchone() + if row is None: + return None + return self._row_to_tool_execution(row) + def get_messages( self, session_id: str, @@ -555,6 +656,38 @@ def _row_to_session(self, row: tuple) -> ConversationSession: metadata=meta, ) + def _row_to_tool_execution(self, row: tuple) -> "ToolExecutionRecord": + from leapflow.engine.tool_execution import ToolExecutionRecord + + arguments: dict[str, Any] = {} + result: Any = None + try: + arguments_raw = row[7] + parsed_arguments = json.loads(arguments_raw) if arguments_raw else {} + if isinstance(parsed_arguments, dict): + arguments = parsed_arguments + except (json.JSONDecodeError, TypeError): + arguments = {} + try: + result = json.loads(row[10]) if row[10] else None + except (json.JSONDecodeError, TypeError): + result = row[10] + return ToolExecutionRecord( + execution_id=row[0] or "", + session_id=row[1] or "", + turn_id=row[2] or "", + command_id=row[3] or "", + tool_call_id=row[4] or "", + tool_name=row[5] or "", + idempotency_key=row[6] or "", + arguments=arguments, + policy=row[8] or "read_only", + status=row[9] or "reserved", + result=result, + created_at=row[11] or 0.0, + completed_at=row[12] or 0.0, + ) + def _row_to_message(self, row: tuple) -> ConversationMessage: meta = {} try: diff --git a/src/leapflow/tools/gateway_tool.py b/src/leapflow/tools/gateway_tool.py index 2558db8..edb3834 100644 --- a/src/leapflow/tools/gateway_tool.py +++ b/src/leapflow/tools/gateway_tool.py @@ -21,22 +21,18 @@ _approval_gate: Any = None _pending_app_onboarding: Dict[str, Any] | None = None -# Task-scoped side-effect dedup: tracks completed (platform, action, payload) -# fingerprints within the current user turn. Reset at each engine.run() via -# reset_platform_action_scope(). Prevents cross-turn duplicate execution of -# send/write/execute actions regardless of LLM behavior. -_task_completed_actions: Dict[str, Dict[str, Any]] = {} - +# Platform action idempotency is enforced centrally by the engine ToolExecutionLedger. +# The reset hook remains for callers that still mark turn boundaries. _SIDE_EFFECT_KINDS = frozenset({"send", "write", "execute"}) def reset_platform_action_scope() -> None: - """Reset the task-scoped action dedup state. Called at each turn boundary.""" - _task_completed_actions.clear() + """Compatibility hook for turn boundaries; engine ledger owns deduplication.""" + return None def _action_fingerprint(platform: str, action: str, payload: Mapping[str, Any]) -> str: - """Stable fingerprint for dedup: hash of (platform, action, sorted payload).""" + """Return the legacy fingerprint format for diagnostics only.""" raw = f"{platform}:{action}:{_json.dumps(payload, sort_keys=True, ensure_ascii=False)}" return hashlib.sha256(raw.encode()).hexdigest()[:24] @@ -906,27 +902,6 @@ async def platform_action_handler(params: Dict[str, Any]) -> Dict[str, Any]: if not validation.ok: return _structured_validation_error(spec, validation, params) - # Task-scoped side-effect dedup: block re-execution of identical - # send/write/execute actions within the same user turn. - fp = _action_fingerprint(platform, action_name, payload) - if spec.effect in _SIDE_EFFECT_KINDS and fp in _task_completed_actions: - logger.info( - "side_effect_dedup: blocked duplicate %s.%s fp=%s", - platform, action_name, fp, - ) - original = _task_completed_actions[fp] - return { - "ok": True, - "already_executed": True, - "execution_note": ( - f"This exact action ({platform}.{action_name}) was already executed " - "successfully in this task. Do not re-invoke. Report the original " - "result to the user." - ), - "original_result": original, - "retryable": False, - } - feasibility = _gateway_server_ref.check_platform_action_feasibility( platform, action_name, payload ) @@ -1033,13 +1008,6 @@ async def platform_action_handler(params: Dict[str, Any]) -> Dict[str, Any]: result = await _gateway_server_ref.execute_platform_action(platform, action_name, payload) - # Register successful side-effect actions for dedup. - if result.get("ok") and spec.effect in _SIDE_EFFECT_KINDS: - _task_completed_actions[fp] = { - k: v for k, v in result.items() - if k in ("ok", "resource_id", "data", "action", "platform") - } - return result diff --git a/src/leapflow/tools/name_resolver.py b/src/leapflow/tools/name_resolver.py index 4f9d36d..0ddf24a 100644 --- a/src/leapflow/tools/name_resolver.py +++ b/src/leapflow/tools/name_resolver.py @@ -108,6 +108,9 @@ class ToolSpec: parameters: frozenset[str] = field(default_factory=frozenset) required: frozenset[str] = field(default_factory=frozenset) risk_level: RiskLevel = "read_only" + mutates_state: bool = False + effect_scope: str = "local" + idempotency_scope: str = "turn" @dataclass(frozen=True) @@ -178,8 +181,8 @@ def from_definitions( Each entry declares a human-verified 1:1 semantic equivalence. Keys are normalized via ``tool_lookup_key`` before storage. """ - bridge_mutates = { - str(tool.get("name", "")).removeprefix("gp_"): bool(tool.get("mutates_state", False)) + bridge_meta = { + str(tool.get("name", "")).removeprefix("gp_"): dict(tool) for tool in bridge_tools } specs: dict[str, ToolSpec] = {} @@ -191,19 +194,32 @@ def from_definitions( parameters_schema = function.get("parameters", {}) or {} properties = parameters_schema.get("properties", {}) or {} required = parameters_schema.get("required", []) or [] + metadata = function.get("x_leapflow", {}) or definition.get("x_leapflow", {}) or {} + bridge_tool = bridge_meta.get(name, {}) + mutates_state = bool(bridge_tool.get("mutates_state", metadata.get("mutates_state", False))) + risk_level = _infer_risk_level(name, mutates_state) specs[name] = ToolSpec( name=name, description=str(function.get("description") or definition.get("description") or ""), parameters=frozenset(str(key) for key in properties.keys()), required=frozenset(str(key) for key in required), - risk_level=_infer_risk_level(name, bridge_mutates.get(name, False)), + risk_level=risk_level, + mutates_state=mutates_state, + effect_scope=str(metadata.get("effect_scope") or ("external" if risk_level == "external" else "local")), + idempotency_scope=str(metadata.get("idempotency_scope") or ("session" if risk_level == "external" else "turn")), ) for name in handlers.keys(): canonical = str(name).removeprefix("gp_") if canonical and canonical not in specs: + bridge_tool = bridge_meta.get(canonical, {}) + mutates_state = bool(bridge_tool.get("mutates_state", False)) + risk_level = _infer_risk_level(canonical, mutates_state) specs[canonical] = ToolSpec( name=canonical, - risk_level=_infer_risk_level(canonical, bridge_mutates.get(canonical, False)), + risk_level=risk_level, + mutates_state=mutates_state, + effect_scope="external" if risk_level == "external" else "local", + idempotency_scope="session" if risk_level == "external" else "turn", ) validated_aliases: dict[str, str] = {} diff --git a/src/leapflow/tools/registry_bootstrap.py b/src/leapflow/tools/registry_bootstrap.py index ddb178f..8c87d7d 100644 --- a/src/leapflow/tools/registry_bootstrap.py +++ b/src/leapflow/tools/registry_bootstrap.py @@ -9,6 +9,7 @@ from typing import Any, Dict, List from leapflow.tools.file_operations import file_list, file_read, file_write +from leapflow.tools.scm_tools import scm_sync from leapflow.tools.shell_tools import shell_run from leapflow.tools.system_tools import env_info, time_get from leapflow.tools.text_tools import text_replace, text_search @@ -108,6 +109,45 @@ }, }, }, + { + "type": "function", + "function": { + "name": "scm_sync", + "description": ( + "Run a typed git SCM action. Use this instead of shell_run for git pull/push/status. " + "For 'pull origin main then push', set action='pull_then_push', remote='origin', " + "pull_ref='main', and omit push_ref so LeapFlow pushes the current local branch." + ), + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["status", "pull", "push", "pull_then_push"], + "description": "Structured SCM action to run.", + }, + "cwd": {"type": "string", "description": "Repository working directory (optional)."}, + "remote": {"type": "string", "description": "Git remote, default origin."}, + "pull_ref": {"type": "string", "description": "Remote ref to pull, e.g. main."}, + "push_ref": { + "type": "string", + "description": "Ref to push. Omit or use current_branch to push the current local branch.", + }, + "timeout": {"type": "number", "description": "Timeout in seconds (default/max 120)."}, + }, + "required": ["action"], + }, + "x_leapflow": { + "category": "scm", + "risk_level": "high", + "schema_cost": "high", + "requires_approval": True, + "effect_scope": "external", + "idempotency_scope": "session", + "summary": "Typed git status/pull/push with explicit current-branch push semantics.", + }, + }, + }, { "type": "function", "function": { @@ -334,6 +374,20 @@ "handler": shell_run, "mutates_state": True, }, + { + "name": "gp_scm_sync", + "description": "Run typed git status/pull/push actions with explicit current-branch push semantics.", + "parameters": { + "action": "string (required) — status|pull|push|pull_then_push", + "cwd": "string (optional) — repository working directory", + "remote": "string (optional) — git remote, default origin", + "pull_ref": "string (optional) — ref to pull, e.g. main", + "push_ref": "string (optional) — ref to push; default current_branch", + "timeout": "number (optional) — timeout in seconds (default/max 120)", + }, + "handler": scm_sync, + "mutates_state": True, + }, { "name": "gp_time_get", "description": "Get current date and time.", diff --git a/src/leapflow/tools/scm_tools.py b/src/leapflow/tools/scm_tools.py new file mode 100644 index 0000000..3363eca --- /dev/null +++ b/src/leapflow/tools/scm_tools.py @@ -0,0 +1,241 @@ +"""Typed source-control tools. + +The SCM tool intentionally models git operations as structured actions instead +of asking the model to synthesize raw shell commands. This keeps ref semantics +explicit: pulling from ``origin/main`` does not imply pushing to ``origin/main``. +""" +from __future__ import annotations + +import asyncio +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Awaitable, Callable, Dict, Sequence + +from leapflow.security.redact import redact_sensitive_text + +_MAX_OUTPUT_CHARS = 10_000 +_DEFAULT_TIMEOUT_S = 120.0 +_ALLOWED_ACTIONS = frozenset({"status", "pull", "push", "pull_then_push"}) + + +@dataclass(frozen=True) +class GitCommandResult: + """Result from one git command invocation.""" + + returncode: int + stdout: str + stderr: str + + +GitRunner = Callable[[Sequence[str], Path, float], Awaitable[GitCommandResult]] + + +def _clip_output(value: str, limit: int = _MAX_OUTPUT_CHARS) -> str: + text = redact_sensitive_text(str(value or ""), force=True) + if len(text) <= limit: + return text + return text[: limit - 1] + "…" + + +def _workspace(cwd: Any) -> Path: + raw = str(cwd or ".").strip() or "." + return Path(raw).expanduser().resolve() + + +def _safe_ref(value: Any, *, field: str) -> str: + ref = str(value or "").strip() + if not ref: + return "" + if ref.startswith("-") or any(ch.isspace() for ch in ref) or ".." in ref: + raise ValueError(f"Invalid git ref for {field}: {ref}") + return ref + + +async def _run_git(args: Sequence[str], cwd: Path, timeout_s: float) -> GitCommandResult: + proc = await asyncio.create_subprocess_exec( + "git", + *args, + cwd=str(cwd), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout_s) + except TimeoutError: + proc.kill() + await proc.communicate() + return GitCommandResult(returncode=124, stdout="", stderr=f"git command timed out after {timeout_s:.0f}s") + return GitCommandResult( + returncode=proc.returncode, + stdout=_clip_output(stdout.decode("utf-8", errors="replace")), + stderr=_clip_output(stderr.decode("utf-8", errors="replace")), + ) + + +def _step_payload(step: str, args: Sequence[str], result: GitCommandResult) -> Dict[str, Any]: + return { + "step": step, + "command": "git " + " ".join(args), + "returncode": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + "ok": result.returncode == 0, + } + + +def _failure_payload( + *, + action: str, + cwd: Path, + step: str, + args: Sequence[str], + result: GitCommandResult, + completed_steps: list[Dict[str, Any]], + current_branch: str = "", +) -> Dict[str, Any]: + error = result.stderr.strip() or result.stdout.strip() or f"git {step} failed with exit code {result.returncode}" + return { + "ok": False, + "tool": "scm_sync", + "scm": "git", + "action": action, + "cwd": str(cwd), + "current_branch": current_branch, + "failed_step": step, + "failure_code": f"git_{step}_failed", + "returncode": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + "error": error, + "command": "git " + " ".join(args), + "completed_steps": completed_steps, + "retryable": False, + } + + +async def _current_branch(cwd: Path, timeout_s: float, runner: GitRunner) -> tuple[str, Dict[str, Any] | None]: + args = ("branch", "--show-current") + result = await runner(args, cwd, timeout_s) + if result.returncode != 0: + return "", _failure_payload( + action="resolve_current_branch", + cwd=cwd, + step="branch", + args=args, + result=result, + completed_steps=[], + ) + branch = result.stdout.strip() + if not branch: + detached = GitCommandResult(returncode=1, stdout=result.stdout, stderr="Current HEAD is detached; explicit push_ref is required.") + return "", _failure_payload( + action="resolve_current_branch", + cwd=cwd, + step="branch", + args=args, + result=detached, + completed_steps=[], + ) + return branch, None + + +async def scm_sync(params: Dict[str, Any], runner: GitRunner | None = None) -> Dict[str, Any]: + """Run a structured git sync action. + + Parameters + ---------- + action: + ``status``, ``pull``, ``push``, or ``pull_then_push``. + remote: + Git remote used for pull/push. Defaults to ``origin``. + pull_ref: + Ref to pull from. For the common request "pull origin main then push", + this is ``main``. + push_ref: + Ref to push. Defaults to the literal current local branch, not + ``pull_ref``. Use ``current_branch`` or omit it to keep that behavior. + """ + action = str(params.get("action") or "pull_then_push").strip().lower() + if action not in _ALLOWED_ACTIONS: + return {"ok": False, "error": f"Unsupported SCM action: {action}", "failure_code": "unsupported_scm_action"} + + cwd = _workspace(params.get("cwd")) + if not cwd.exists(): + return {"ok": False, "error": f"Working directory does not exist: {cwd}", "failure_code": "path_not_found", "cwd": str(cwd)} + if not cwd.is_dir(): + return {"ok": False, "error": f"Working directory is not a directory: {cwd}", "failure_code": "path_not_directory", "cwd": str(cwd)} + + timeout_s = min(float(params.get("timeout") or _DEFAULT_TIMEOUT_S), _DEFAULT_TIMEOUT_S) + run = runner or _run_git + remote = _safe_ref(params.get("remote") or "origin", field="remote") + pull_ref = _safe_ref(params.get("pull_ref") or "", field="pull_ref") + push_ref = _safe_ref(params.get("push_ref") or "current_branch", field="push_ref") + completed_steps: list[Dict[str, Any]] = [] + + current_branch = "" + if action in {"push", "pull_then_push"} and push_ref in {"", "current", "current_branch"}: + current_branch, branch_failure = await _current_branch(cwd, timeout_s, run) + if branch_failure is not None: + branch_failure["action"] = action + return branch_failure + push_ref = current_branch + + if action == "status": + args = ("status", "--short", "--branch") + result = await run(args, cwd, timeout_s) + return _failure_payload(action=action, cwd=cwd, step="status", args=args, result=result, completed_steps=[]) if result.returncode != 0 else { + "ok": True, + "tool": "scm_sync", + "scm": "git", + "action": action, + "cwd": str(cwd), + "stdout": result.stdout, + "stderr": result.stderr, + "completed_steps": [_step_payload("status", args, result)], + } + + if action in {"pull", "pull_then_push"}: + pull_args = ("pull", remote, pull_ref) if pull_ref else ("pull", remote) + pull_result = await run(pull_args, cwd, timeout_s) + if pull_result.returncode != 0: + return _failure_payload( + action=action, + cwd=cwd, + step="pull", + args=pull_args, + result=pull_result, + completed_steps=completed_steps, + current_branch=current_branch, + ) + completed_steps.append(_step_payload("pull", pull_args, pull_result)) + + if action in {"push", "pull_then_push"}: + push_args = ("push", remote, push_ref) + push_result = await run(push_args, cwd, timeout_s) + if push_result.returncode != 0: + return _failure_payload( + action=action, + cwd=cwd, + step="push", + args=push_args, + result=push_result, + completed_steps=completed_steps, + current_branch=current_branch, + ) + completed_steps.append(_step_payload("push", push_args, push_result)) + + return { + "ok": True, + "tool": "scm_sync", + "scm": "git", + "action": action, + "cwd": str(cwd), + "remote": remote, + "pull_ref": pull_ref, + "push_ref": push_ref, + "current_branch": current_branch, + "completed": True, + "completed_steps": completed_steps, + "stdout": "\n".join(step.get("stdout", "") for step in completed_steps if step.get("stdout")), + "stderr": "\n".join(step.get("stderr", "") for step in completed_steps if step.get("stderr")), + } diff --git a/tests/test_agent_execution.py b/tests/test_agent_execution.py index 760d42e..3be0528 100644 --- a/tests/test_agent_execution.py +++ b/tests/test_agent_execution.py @@ -2,6 +2,7 @@ from __future__ import annotations +import asyncio import tempfile from typing import List from unittest.mock import AsyncMock @@ -184,6 +185,188 @@ async def file_list_handler(args): lt.close() +@pytest.mark.asyncio +async def test_tool_execution_ledger_skips_duplicate_external_tool() -> None: + with tempfile.TemporaryDirectory() as td: + settings = make_settings(td) + from leapflow.platform.mock import MockBridge + + rpc = MockBridge() + llm = StubLLM([]) + wm = WorkingMemoryProvider(max_tokens=1024) + lt = SemanticMemoryProvider(source=settings.duckdb_path) + imm = EpisodicMemoryProvider() + calls: list[dict[str, object]] = [] + + async def shell_handler(args): + calls.append(dict(args)) + return {"ok": True, "stdout": "pushed"} + + try: + reg = build_default_registry(rpc, llm, wm, lt) + classifier = _FixedClassifier("complex") + engine = AgentEngine(settings, rpc, llm, wm, lt, imm, reg, classifier) + engine._tool_bridge = None + engine._current_session_id = "session-1" + engine._session_turn_count = 1 + engine._begin_turn_context("push once") + call = {"name": "shell_run", "arguments": {"command": "git push"}} + + first = await engine._execute_tool_with_ledger(call, {"shell_run": shell_handler}, tool_call_id="a") + second = await engine._execute_tool_with_ledger(call, {"shell_run": shell_handler}, tool_call_id="b") + + assert len(calls) == 1 + assert first["ok"] is True + assert first["execution_policy"] == "external_side_effect" + assert second["already_executed"] is True + assert second["original_result"]["stdout"] == "pushed" + finally: + lt.close() + + +@pytest.mark.asyncio +async def test_tool_execution_ledger_waits_for_inflight_duplicate_external_tool() -> None: + with tempfile.TemporaryDirectory() as td: + settings = make_settings(td) + from leapflow.platform.mock import MockBridge + + rpc = MockBridge() + llm = StubLLM([]) + wm = WorkingMemoryProvider(max_tokens=1024) + lt = SemanticMemoryProvider(source=settings.duckdb_path) + imm = EpisodicMemoryProvider() + started = asyncio.Event() + release = asyncio.Event() + calls: list[dict[str, object]] = [] + + async def shell_handler(args): + calls.append(dict(args)) + started.set() + await release.wait() + return {"ok": True, "stdout": "pushed"} + + try: + reg = build_default_registry(rpc, llm, wm, lt) + classifier = _FixedClassifier("complex") + engine = AgentEngine(settings, rpc, llm, wm, lt, imm, reg, classifier) + engine._tool_bridge = None + engine._current_session_id = "session-1" + engine._session_turn_count = 1 + engine._begin_turn_context("push once") + call = {"name": "shell_run", "arguments": {"command": "git push"}} + + first_task = asyncio.create_task( + engine._execute_tool_with_ledger(call, {"shell_run": shell_handler}, tool_call_id="a") + ) + await started.wait() + second_task = asyncio.create_task( + engine._execute_tool_with_ledger(call, {"shell_run": shell_handler}, tool_call_id="b") + ) + await asyncio.sleep(0) + + assert len(calls) == 1 + assert not second_task.done() + + release.set() + first, second = await asyncio.gather(first_task, second_task) + + assert first["ok"] is True + assert second["already_executed"] is True + assert second["ok"] is True + assert second["original_result"]["stdout"] == "pushed" + assert len(calls) == 1 + finally: + lt.close() + + +@pytest.mark.asyncio +async def test_tool_execution_ledger_allows_repeated_read_only_tool() -> None: + with tempfile.TemporaryDirectory() as td: + settings = make_settings(td) + from leapflow.platform.mock import MockBridge + + rpc = MockBridge() + llm = StubLLM([]) + wm = WorkingMemoryProvider(max_tokens=1024) + lt = SemanticMemoryProvider(source=settings.duckdb_path) + imm = EpisodicMemoryProvider() + calls: list[dict[str, object]] = [] + + async def file_list_handler(args): + calls.append(dict(args)) + return {"ok": True, "entries": []} + + try: + reg = build_default_registry(rpc, llm, wm, lt) + classifier = _FixedClassifier("complex") + engine = AgentEngine(settings, rpc, llm, wm, lt, imm, reg, classifier) + engine._tool_bridge = None + engine._current_session_id = "session-1" + engine._session_turn_count = 1 + engine._begin_turn_context("list twice") + call = {"name": "file_list", "arguments": {"path": "."}} + + first = await engine._execute_tool_with_ledger(call, {"file_list": file_list_handler}, tool_call_id="a") + second = await engine._execute_tool_with_ledger(call, {"file_list": file_list_handler}, tool_call_id="b") + + assert len(calls) == 2 + assert first["execution_policy"] == "read_only" + assert second["execution_policy"] == "read_only" + assert "already_executed" not in second + finally: + lt.close() + + +@pytest.mark.asyncio +async def test_side_effect_failure_stops_remaining_native_tool_batch() -> None: + from leapflow.engine.execution_trace import ExecutionTrace + from leapflow.llm.base import ToolCallInfo + from leapflow.platform.mock import MockBridge + + with tempfile.TemporaryDirectory() as td: + settings = make_settings(td) + rpc = MockBridge() + llm = StubLLM([]) + wm = WorkingMemoryProvider(max_tokens=1024) + lt = SemanticMemoryProvider(source=settings.duckdb_path) + imm = EpisodicMemoryProvider() + calls: list[str] = [] + + async def execute_tool(tool_call, _handlers): + calls.append(str(tool_call.get("arguments", {}).get("command"))) + return {"ok": False, "returncode": 1, "stderr": "cd: no such file or directory"} + + try: + reg = build_default_registry(rpc, llm, wm, lt) + classifier = _FixedClassifier("complex") + engine = AgentEngine(settings, rpc, llm, wm, lt, imm, reg, classifier) + engine._tool_bridge = None + engine._execute_general_tool = AsyncMock(side_effect=execute_tool) # type: ignore[method-assign] + engine._current_session_id = "session-1" + engine._session_turn_count = 1 + engine._begin_turn_context("run git commands") + messages: list[dict[str, object]] = [] + + results = await engine._execute_tools_concurrent( + [ + ToolCallInfo(id="tc1", name="shell_run", arguments={"command": "cd missing"}), + ToolCallInfo(id="tc2", name="shell_run", arguments={"command": "git status"}), + ], + {"shell_run": execute_tool}, + trace=ExecutionTrace(), + messages=messages, + ) + + assert calls == ["cd missing"] + assert len(results) == 2 + assert results[0]["result"]["ok"] is False + assert results[1]["result"]["execution_skipped"] is True + assert results[1]["result"]["counts_as_failure"] is False + assert AgentEngine._count_consecutive_tool_failures(messages) == 1 + finally: + lt.close() + + @pytest.mark.asyncio async def test_unknown_tool_returns_structured_retry_feedback() -> None: """Unknown tools should produce structured feedback instead of a bare string.""" @@ -964,36 +1147,43 @@ def test_task_graph_retry_policy() -> None: # ═══════════════════════════════════════════════════════════════════ -def test_platform_action_fingerprint_deduplicates_identical_calls() -> None: - """_platform_action_fingerprint returns the same key for identical calls.""" - from leapflow.engine.engine import _platform_action_fingerprint +def test_platform_action_idempotency_key_deduplicates_identical_calls() -> None: + """Unified idempotency keys replace the old platform_action fingerprint.""" + from leapflow.engine.tool_execution import build_idempotency_key - fp1 = _platform_action_fingerprint( - "platform_action", - {"platform": "feishu", "action": "im.send_message", "payload": {"chat_id": "oc_1", "text": "hi"}}, + args = {"platform": "feishu", "action": "im.send_message", "payload": {"chat_id": "oc_1", "text": "hi"}} + key1 = build_idempotency_key( + session_id="session-1", + turn_id="turn-1", + tool_name="platform_action", + arguments=args, + policy="external_side_effect", ) - fp2 = _platform_action_fingerprint( - "platform_action", - {"platform": "feishu", "action": "im.send_message", "payload": {"chat_id": "oc_1", "text": "hi"}}, + key2 = build_idempotency_key( + session_id="session-1", + turn_id="turn-2", + tool_name="platform_action", + arguments=args, + policy="external_side_effect", ) - fp_different = _platform_action_fingerprint( - "platform_action", - {"platform": "feishu", "action": "im.send_message", "payload": {"chat_id": "oc_2", "text": "hi"}}, + different_payload = build_idempotency_key( + session_id="session-1", + turn_id="turn-1", + tool_name="platform_action", + arguments={"platform": "feishu", "action": "im.send_message", "payload": {"chat_id": "oc_2", "text": "hi"}}, + policy="external_side_effect", ) - fp_read = _platform_action_fingerprint( - "platform_action", - {"platform": "feishu", "action": "im.search_chats", "payload": {"query": "LeapFlow"}}, - ) - fp_other_tool = _platform_action_fingerprint( - "file_list", - {"path": "."}, + other_tool = build_idempotency_key( + session_id="session-1", + turn_id="turn-1", + tool_name="file_list", + arguments={"path": "."}, + policy="read_only", ) - assert fp1 is not None - assert fp1 == fp2, "Identical calls must produce the same fingerprint" - assert fp1 != fp_different, "Different payload must produce a different fingerprint" - assert fp_read is not None, "Read platform_actions are also fingerprinted (dedup applies)" - assert fp_other_tool is None, "Non-platform_action tools must return None" + assert key1 == key2, "External side effects deduplicate across turns in the same session" + assert key1 != different_payload, "Different payload must produce a different idempotency key" + assert key1 != other_tool, "Tool name and policy participate in the key" def test_last_tool_failures_recovery_message_from_unknown_action() -> None: @@ -1038,8 +1228,37 @@ def test_last_tool_failures_recovery_message_missing_fields() -> None: assert "text" in result -def test_last_tool_failures_recovery_message_returns_empty_when_no_failures() -> None: - """Returns empty string when there are no failed tool results in history.""" +def test_duplicate_suppression_is_not_counted_as_consecutive_tool_failure() -> None: + """Suppressed duplicate side effects are control signals, not failed executions.""" + import json + from leapflow.engine.engine import _last_tool_failures_recovery_message + + root_failure = { + "ok": False, + "error": "git push rejected", + "stderr": "non-fast-forward", + "execution_policy": "external_side_effect", + } + duplicate_suppressed = { + "ok": False, + "already_executed": True, + "duplicate_suppressed": True, + "counts_as_failure": False, + "error": "An identical side-effect attempt is already recorded. Review the original result before retrying.", + } + messages = [ + {"role": "tool", "content": json.dumps(root_failure)}, + {"role": "tool", "content": json.dumps(duplicate_suppressed)}, + ] + + assert AgentEngine._count_consecutive_tool_failures(messages) == 1 + recovery = _last_tool_failures_recovery_message(messages) + assert "git push rejected" in recovery + assert "consecutive tool failures" not in recovery + assert "duplicate execution was not replayed" not in recovery + + + import json from leapflow.engine.engine import _last_tool_failures_recovery_message diff --git a/tests/test_app_connector.py b/tests/test_app_connector.py index 6af2da5..7ecb05f 100644 --- a/tests/test_app_connector.py +++ b/tests/test_app_connector.py @@ -425,12 +425,12 @@ def test_empty_required_field_counts_as_missing(self) -> None: # ═══════════════════════════════════════════════════════════════ -# Task-scoped side-effect dedup tests +# Direct gateway handler execution tests # ═══════════════════════════════════════════════════════════════ @pytest.mark.asyncio -async def test_side_effect_dedup_blocks_duplicate_send() -> None: - """Second identical send call returns already_executed without execution.""" +async def test_platform_action_direct_handler_does_not_deduplicate_send() -> None: + """Direct gateway handler calls stay stateless; engine ledger owns idempotency.""" from leapflow.gateway.connectors.protocol import ActionPreview, ActionResult, BackendStatus from leapflow.gateway.adapters.feishu import FeishuAdapter from leapflow.gateway.server import GatewayServer @@ -481,9 +481,8 @@ async def preview(self, spec, payload): assert result1.get("ok") is True assert result2.get("ok") is True - assert result2.get("already_executed") is True - assert "Do not re-invoke" in result2.get("execution_note", "") - assert backend.call_count == 1 + assert result2.get("already_executed") is None + assert backend.call_count == 2 finally: await server.stop() set_gateway_approval_gate(None) @@ -492,8 +491,8 @@ async def preview(self, spec, payload): @pytest.mark.asyncio -async def test_side_effect_dedup_allows_read_actions_to_repeat() -> None: - """Read actions (effect=read) are not subject to dedup.""" +async def test_platform_action_direct_handler_allows_read_actions_to_repeat() -> None: + """Read actions (effect=read) naturally repeat through the stateless handler.""" from leapflow.gateway.connectors.protocol import ActionPreview, ActionResult, BackendStatus from leapflow.gateway.adapters.feishu import FeishuAdapter from leapflow.gateway.server import GatewayServer @@ -554,8 +553,8 @@ async def preview(self, spec, payload): @pytest.mark.asyncio -async def test_side_effect_dedup_resets_across_turns() -> None: - """After reset_platform_action_scope, same action can execute again.""" +async def test_reset_platform_action_scope_is_compatibility_noop() -> None: + """The reset hook no longer owns deduplication; direct calls still execute.""" from leapflow.gateway.connectors.protocol import ActionPreview, ActionResult, BackendStatus from leapflow.gateway.adapters.feishu import FeishuAdapter from leapflow.gateway.server import GatewayServer diff --git a/tests/test_daemon_rpc.py b/tests/test_daemon_rpc.py index 414301f..1528b2c 100644 --- a/tests/test_daemon_rpc.py +++ b/tests/test_daemon_rpc.py @@ -65,6 +65,23 @@ async def engine_chat(self, message: str, **kwargs: Any) -> AsyncIterator[Stream yield StreamChunk(request_id="", content="done", event_type="final") +class _RequestIdCaptureService(_FakeService): + def __init__(self) -> None: + super().__init__() + self.request_ids: list[str] = [] + + async def engine_chat(self, message: str, **kwargs: Any) -> AsyncIterator[StreamChunk]: + request_id = str(kwargs.get("request_id") or "") + self.request_ids.append(request_id) + yield StreamChunk( + request_id=request_id, + content=f"request {message}", + event_type="chunk", + metadata={"request_id": request_id}, + ) + yield StreamChunk(request_id=request_id, content="done", event_type="final") + + async def _start_server(runtime_dir: Path, service=None, *, stream_heartbeat_s: float | None = None): server = UnixRpcServer( service or _FakeService(), @@ -104,6 +121,96 @@ async def test_daemon_client_receives_stream_events() -> None: assert events[0].metadata == {"session_id": "sess-1"} +@pytest.mark.asyncio +async def test_daemon_server_injects_rpc_request_id_into_engine_chat() -> None: + with tempfile.TemporaryDirectory(prefix="lfd-", dir="/tmp") as root: + service = _RequestIdCaptureService() + server, task, runtime_dir = await _start_server(Path(root) / "runtime", service=service) + client = DaemonClient(runtime_dir / "leapd.sock") + + try: + events = [event async for event in client.engine_chat("world")] + finally: + task.cancel() + await server.stop() + try: + await task + except asyncio.CancelledError: + pass + + assert len(service.request_ids) == 1 + assert service.request_ids[0] + assert events[0].metadata == {"request_id": service.request_ids[0]} + + +@pytest.mark.asyncio +async def test_runtime_service_replays_duplicate_engine_request_without_rerun() -> None: + from types import SimpleNamespace + + from leapflow.daemon.service import RuntimeLeapService + from leapflow.engine import StreamEvent + + class FakeEngine: + def __init__(self) -> None: + self.calls = 0 + self._current_session_id = "session-1" + + async def run_stream(self, message: str, *, enable_thinking: bool = False, request_id: str = ""): + self.calls += 1 + yield StreamEvent( + type="chunk", + content=f"{request_id}:{message}", + metadata={"seen_request_id": request_id}, + ) + yield StreamEvent(type="final", content="done") + + class FakeContext: + def __init__(self) -> None: + self.settings = SimpleNamespace(llm_context_length=100) + self.engine = FakeEngine() + + def reload_runtime_config_if_changed(self) -> bool: + return False + + service = RuntimeLeapService(SimpleNamespace(llm_context_length=100)) + ctx = FakeContext() + service._ctx = ctx + + first = [chunk async for chunk in service.engine_chat("hello", request_id="req-1")] + second = [chunk async for chunk in service.engine_chat("hello", request_id="req-1")] + + assert ctx.engine.calls == 1 + assert [chunk.content for chunk in first] == ["req-1:hello", "done"] + assert [chunk.content for chunk in second] == ["req-1:hello", "done"] + assert second[0].metadata["replayed_request"] is True + + +@pytest.mark.asyncio +async def test_runtime_service_prunes_completed_engine_request_replay_records() -> None: + from types import SimpleNamespace + + from leapflow.daemon.service import RuntimeLeapService + + settings = SimpleNamespace( + llm_context_length=100, + daemon_request_ledger_ttl_s=1_000_000_000_000.0, + daemon_request_ledger_max_entries=2, + ) + service = RuntimeLeapService(settings) + service._engine_request_ledger = { + "old-1": {"status": "completed", "chunks": [], "created_at": 1.0, "completed_at": 1.0}, + "old-2": {"status": "failed", "chunks": [], "created_at": 2.0, "completed_at": 2.0}, + "new-1": {"status": "completed", "chunks": [], "created_at": 3.0, "completed_at": 3.0}, + "running": {"status": "running", "chunks": [], "created_at": 0.0}, + } + + service._prune_engine_request_ledger() + + assert "running" in service._engine_request_ledger + assert "new-1" in service._engine_request_ledger + assert len(service._engine_request_ledger) <= 2 + + @pytest.mark.asyncio async def test_daemon_client_can_cancel_engine_turn() -> None: with tempfile.TemporaryDirectory(prefix="lfd-", dir="/tmp") as root: @@ -234,6 +341,8 @@ def reload_runtime_config_if_changed(self) -> bool: assert approval_event.event_type == "approval_request" assert isinstance(payload, dict) assert payload["pending_id"] + assert (approval_event.metadata or {}).get("request_id") + assert payload.get("request_id") == (approval_event.metadata or {}).get("request_id") status = await service.approval_status() assert len(status["pending"]) == 1 @@ -242,7 +351,9 @@ def reload_runtime_config_if_changed(self) -> bool: finally: await stream.aclose() - assert resolved == {"ok": True, "pending_id": payload["pending_id"], "decision": "deny"} + assert resolved["ok"] is True + assert resolved["pending_id"] == payload["pending_id"] + assert resolved["decision"] == "deny" assert final.event_type == "final" assert final.content == "decision=deny" assert final.metadata["session_id"] == "sess-approval" @@ -669,7 +780,10 @@ async def test_runtime_service_hot_reloads_config_before_daemon_chat( assert first.event_type == "status" assert "Configuration reloaded" in first.content - assert first.metadata == { + metadata = dict(first.metadata or {}) + request_id = metadata.pop("request_id", "") + assert request_id + assert metadata == { "llm_model": service.context.settings.llm_model, "llm_context_length": 700_000, "context_used": 0, diff --git a/tests/test_memory_and_storage.py b/tests/test_memory_and_storage.py index 906120b..e05a0c9 100644 --- a/tests/test_memory_and_storage.py +++ b/tests/test_memory_and_storage.py @@ -374,6 +374,52 @@ def test_skill_library_crud(skill_library) -> None: assert active[0].skill_id == "s1" +def test_tool_execution_store_roundtrip_and_unique_key(tmp_path: Path) -> None: + from leapflow.engine.tool_execution import ToolExecutionRecord + from leapflow.storage.conversation_store import DuckDBConversationStore + + store = DuckDBConversationStore(tmp_path / "conversation.duckdb") + try: + record = ToolExecutionRecord.create( + session_id="session-1", + turn_id="turn-1", + command_id="command-1", + tool_call_id="tool-a", + tool_name="shell_run", + idempotency_key="idem-1", + arguments={"command": "git push"}, + policy="external_side_effect", + ) + duplicate = ToolExecutionRecord.create( + session_id="session-1", + turn_id="turn-1", + command_id="command-1", + tool_call_id="tool-b", + tool_name="shell_run", + idempotency_key="idem-1", + arguments={"command": "git push"}, + policy="external_side_effect", + ) + + store.reserve_tool_execution(record) + store.reserve_tool_execution(duplicate) + completed = record.mark_completed({"ok": True, "stdout": "pushed"}) + store.complete_tool_execution(completed) + + loaded = store.get_tool_execution_by_key("session-1", "idem-1") + rows = store._conn.execute("SELECT COUNT(*) FROM tool_executions").fetchone()[0] + + assert rows == 1 + assert loaded is not None + assert loaded.execution_id == record.execution_id + assert loaded.tool_call_id == "tool-a" + assert loaded.arguments == {"command": "git push"} + assert loaded.result == {"ok": True, "stdout": "pushed"} + assert loaded.status == "completed" + finally: + store.close() + + # ── Cross-tier integration ───────────────────────────────────────── diff --git a/tests/test_recovery_audit.py b/tests/test_recovery_audit.py new file mode 100644 index 0000000..dc4933e --- /dev/null +++ b/tests/test_recovery_audit.py @@ -0,0 +1,365 @@ +"""Tests for recovery_audit module — structured audit logging for recovery decisions.""" +from __future__ import annotations + +import json +import time +from pathlib import Path + +import pytest + +from leapflow.engine.failure_envelope import ( + FailureContext, + FailureEnvelope, + FailureSource, + Recoverability, + SideEffectState, +) +from leapflow.engine.recovery_audit import ( + JsonlAuditSink, + RecoveryAuditEntry, + create_audit_entry, +) +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_decision import ( + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +# ─── Fixtures ──────────────────────────────────────────────────────── + +@pytest.fixture +def sample_envelope() -> FailureEnvelope: + """Create a sample FailureEnvelope for testing.""" + return FailureEnvelope.create( + source=FailureSource.LLM, + category="rate_limited", + failure_class="rate_limited", + failure_code="llm_rate_limited", + message="Rate limit exceeded", + recoverability=Recoverability.AUTO_RETRY, + side_effect_state=SideEffectState.NONE, + context=FailureContext.from_dict_args(provider="openai", model="gpt-4"), + ) + + +@pytest.fixture +def sample_decision(sample_envelope: FailureEnvelope) -> RecoveryDecision: + """Create a sample RecoveryDecision for testing.""" + return RecoveryDecision.create( + envelope=sample_envelope, + action=RecoveryAction.RETRY_WITH_BACKOFF, + reason="Rate limited, retrying with backoff", + strategy_key="jittered_retry", + budget_cost=1, + ) + + +@pytest.fixture +def sample_budget() -> RecoveryBudget: + """Create a sample RecoveryBudget for testing.""" + budget = RecoveryBudget() + budget.start_deadline() + budget.consume(2, "rate_limited") + return budget + + +# ─── RecoveryAuditEntry Tests ──────────────────────────────────────── + +class TestRecoveryAuditEntry: + """Tests for RecoveryAuditEntry dataclass.""" + + def test_creation(self) -> None: + """RecoveryAuditEntry can be created with required fields.""" + entry = RecoveryAuditEntry( + timestamp=time.time(), + session_id="sess-1", + turn_id=3, + envelope_id="env-abc", + failure_source="llm", + failure_category="rate_limited", + failure_code="llm_rate_limited", + recoverability="auto_retry", + decision_id="dec-xyz", + strategy_key="jittered_retry", + action="retry_with_backoff", + reason="Rate limited", + ) + assert entry.session_id == "sess-1" + assert entry.turn_id == 3 + assert entry.strategy_key == "jittered_retry" + + def test_to_json_dict_excludes_empty_optional_fields(self) -> None: + """to_json_dict() excludes empty outcome fields and zero elapsed_ms.""" + entry = RecoveryAuditEntry( + timestamp=1234567890.0, + session_id="sess-1", + turn_id=3, + envelope_id="env-abc", + failure_source="llm", + failure_category="rate_limited", + failure_code="llm_rate_limited", + recoverability="auto_retry", + decision_id="dec-xyz", + strategy_key="jittered_retry", + action="retry_with_backoff", + reason="Rate limited", + budget_cost=0, + budget_consumed=0, + budget_remaining=0, + outcome="", + outcome_reason="", + elapsed_ms=0.0, + ) + d = entry.to_json_dict() + # budget_cost=0 and turn_id kept (they are regular fields with valid zero values) + assert d["budget_cost"] == 0 + assert d["budget_consumed"] == 0 + assert d["budget_remaining"] == 0 + assert d["turn_id"] == 3 + # Optional outcome fields removed when at default + assert "outcome" not in d + assert "outcome_reason" not in d + assert "elapsed_ms" not in d + assert d["session_id"] == "sess-1" + assert d["timestamp"] == 1234567890.0 + + def test_to_json_dict_includes_nonzero(self) -> None: + """to_json_dict() includes non-zero / non-empty values.""" + entry = RecoveryAuditEntry( + timestamp=1234567890.0, + session_id="sess-1", + turn_id=0, + envelope_id="env-abc", + failure_source="llm", + failure_category="rate_limited", + failure_code="llm_rate_limited", + recoverability="auto_retry", + decision_id="dec-xyz", + strategy_key="jittered_retry", + action="retry_with_backoff", + reason="Rate limited", + budget_cost=2, + budget_consumed=5, + budget_remaining=7, + outcome="success", + ) + d = entry.to_json_dict() + assert d["budget_cost"] == 2 + assert d["budget_consumed"] == 5 + assert d["budget_remaining"] == 7 + assert d["outcome"] == "success" + + def test_frozen_immutability(self) -> None: + """RecoveryAuditEntry is immutable (frozen dataclass).""" + entry = RecoveryAuditEntry( + timestamp=time.time(), + session_id="s", + turn_id=1, + envelope_id="e", + failure_source="llm", + failure_category="c", + failure_code="f", + recoverability="auto_retry", + decision_id="d", + strategy_key="k", + action="a", + reason="r", + ) + with pytest.raises(Exception): + entry.session_id = "modified" # type: ignore[misc] + + +# ─── JsonlAuditSink Tests ──────────────────────────────────────────── + +class TestJsonlAuditSink: + """Tests for JsonlAuditSink.""" + + def _make_entry(self, strategy: str = "jittered_retry", + action: str = "retry_with_backoff") -> RecoveryAuditEntry: + """Helper to create test entries.""" + return RecoveryAuditEntry( + timestamp=time.time(), + session_id="sess-1", + turn_id=1, + envelope_id="env-1", + failure_source="llm", + failure_category="rate_limited", + failure_code="llm_rate_limited", + recoverability="auto_retry", + decision_id="dec-1", + strategy_key=strategy, + action=action, + reason="test reason", + budget_cost=1, + ) + + def test_record_to_memory(self) -> None: + """record() stores entries in in-memory buffer.""" + sink = JsonlAuditSink() + entry = self._make_entry() + sink.record(entry) + assert len(sink.entries) == 1 + assert sink.entries[0] is entry + + def test_record_to_file(self, tmp_path: Path) -> None: + """record() writes JSONL entries to disk.""" + audit_file = tmp_path / "audit.jsonl" + sink = JsonlAuditSink(path=audit_file) + entry = self._make_entry() + sink.record(entry) + + assert audit_file.exists() + lines = audit_file.read_text().strip().split("\n") + assert len(lines) == 1 + data = json.loads(lines[0]) + assert data["strategy_key"] == "jittered_retry" + assert data["session_id"] == "sess-1" + + def test_record_multiple_entries(self, tmp_path: Path) -> None: + """Multiple records produce multiple JSONL lines.""" + audit_file = tmp_path / "audit.jsonl" + sink = JsonlAuditSink(path=audit_file) + sink.record(self._make_entry(strategy="s1")) + sink.record(self._make_entry(strategy="s2")) + sink.record(self._make_entry(strategy="s3")) + + lines = audit_file.read_text().strip().split("\n") + assert len(lines) == 3 + assert len(sink.entries) == 3 + + def test_update_outcome(self, tmp_path: Path) -> None: + """update_outcome() writes an outcome update record.""" + audit_file = tmp_path / "audit.jsonl" + sink = JsonlAuditSink(path=audit_file) + sink.record(self._make_entry()) + sink.update_outcome("dec-1", "success", reason="Retry succeeded", elapsed_ms=150.5) + + lines = audit_file.read_text().strip().split("\n") + assert len(lines) == 2 + update = json.loads(lines[1]) + assert update["type"] == "outcome_update" + assert update["decision_id"] == "dec-1" + assert update["outcome"] == "success" + assert update["elapsed_ms"] == 150.5 + + def test_summary_empty(self) -> None: + """summary() returns total=0 for empty sink.""" + sink = JsonlAuditSink() + assert sink.summary() == {"total": 0} + + def test_summary_statistics(self) -> None: + """summary() generates correct statistics.""" + sink = JsonlAuditSink() + sink.record(self._make_entry(strategy="jittered_retry", action="retry_with_backoff")) + sink.record(self._make_entry(strategy="jittered_retry", action="retry_with_backoff")) + sink.record(self._make_entry(strategy="context_compress", action="transform_and_retry")) + sink.record(self._make_entry(strategy="provider_failover", action="failover")) + + s = sink.summary() + assert s["total"] == 4 + assert s["by_strategy"]["jittered_retry"] == 2 + assert s["by_strategy"]["context_compress"] == 1 + assert s["by_strategy"]["provider_failover"] == 1 + assert s["by_action"]["retry_with_backoff"] == 2 + assert s["by_action"]["transform_and_retry"] == 1 + assert s["by_action"]["failover"] == 1 + + def test_file_write_failure_graceful(self, tmp_path: Path) -> None: + """File write failure doesn't crash; entry still stored in memory.""" + # Point to a path that cannot be written (directory as file) + bad_path = tmp_path / "readonly_dir" / "nested" / "audit.jsonl" + # Create the parent as a file to block directory creation + bad_path_parent = tmp_path / "readonly_dir" + bad_path_parent.mkdir() + (bad_path_parent / "nested").write_text("blocker") # file blocks mkdir + + sink = JsonlAuditSink(path=bad_path) + entry = self._make_entry() + # Should not raise + sink.record(entry) + # Entry still in memory buffer + assert len(sink.entries) == 1 + + def test_no_path_memory_only(self) -> None: + """Without a path, sink operates in memory-only mode.""" + sink = JsonlAuditSink(path=None) + sink.record(self._make_entry()) + sink.update_outcome("dec-1", "failure") + assert len(sink.entries) == 1 + + +# ─── create_audit_entry Factory Tests ───────────────────────────────── + +class TestCreateAuditEntry: + """Tests for the create_audit_entry factory function.""" + + def test_basic_creation( + self, sample_envelope: FailureEnvelope, sample_decision: RecoveryDecision, + sample_budget: RecoveryBudget, + ) -> None: + """Factory creates an audit entry from real coordinator types.""" + entry = create_audit_entry( + sample_envelope, sample_decision, sample_budget, + session_id="test-session", turn_id=5, + ) + assert entry.session_id == "test-session" + assert entry.turn_id == 5 + assert entry.envelope_id == sample_envelope.envelope_id + assert entry.failure_source == "llm" + assert entry.failure_category == "rate_limited" + assert entry.failure_code == "llm_rate_limited" + assert entry.recoverability == "auto_retry" + assert entry.decision_id == sample_decision.decision_id + assert entry.strategy_key == "jittered_retry" + assert entry.action == "retry_with_backoff" + assert entry.reason == "Rate limited, retrying with backoff" + assert entry.budget_cost == 1 + assert entry.budget_consumed == 2 # We consumed 2 in the fixture + assert entry.budget_remaining == 10 # 12 total - 2 consumed + + def test_serialization_round_trip( + self, sample_envelope: FailureEnvelope, sample_decision: RecoveryDecision, + sample_budget: RecoveryBudget, + ) -> None: + """Audit entry serializes to valid JSON.""" + entry = create_audit_entry( + sample_envelope, sample_decision, sample_budget, + session_id="s1", turn_id=2, + ) + json_str = json.dumps(entry.to_json_dict(), ensure_ascii=False) + data = json.loads(json_str) + assert data["failure_source"] == "llm" + assert data["strategy_key"] == "jittered_retry" + + def test_defaults_for_session_and_turn( + self, sample_envelope: FailureEnvelope, sample_decision: RecoveryDecision, + sample_budget: RecoveryBudget, + ) -> None: + """Factory uses default empty session_id and turn_id=0.""" + entry = create_audit_entry(sample_envelope, sample_decision, sample_budget) + assert entry.session_id == "" + assert entry.turn_id == 0 + + def test_tool_failure_envelope(self) -> None: + """Factory works with tool-source envelopes.""" + envelope = FailureEnvelope.create( + source=FailureSource.TOOL, + category="tool_permission", + failure_class="authorization", + failure_code="access_denied", + message="Permission denied", + recoverability=Recoverability.NON_RECOVERABLE, + ) + decision = RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.HALT_CLEAN, + reason="Non-recoverable permission failure", + strategy_key="", + ) + budget = RecoveryBudget() + entry = create_audit_entry(envelope, decision, budget) + assert entry.failure_source == "tool" + assert entry.action == "halt_clean" + assert entry.recoverability == "non_recoverable" diff --git a/tests/test_recovery_checkpoint.py b/tests/test_recovery_checkpoint.py new file mode 100644 index 0000000..df3bbd5 --- /dev/null +++ b/tests/test_recovery_checkpoint.py @@ -0,0 +1,435 @@ +"""Tests for the recovery checkpoint system (cross-turn state persistence).""" +from __future__ import annotations + +import time +from unittest.mock import patch + +import pytest + +from leapflow.engine.recovery_checkpoint import ( + CheckpointResumer, + CheckpointState, + CheckpointStore, + DefaultDriftVerifier, + InMemoryCheckpointStore, + Precondition, + RecoveryCheckpoint, + ResumeResult, + StepRecord, +) + + +# --------------------------------------------------------------------------- +# RecoveryCheckpoint dataclass tests +# --------------------------------------------------------------------------- + + +class TestRecoveryCheckpoint: + """Tests for RecoveryCheckpoint creation and computed properties.""" + + def test_creation_defaults(self) -> None: + cp = RecoveryCheckpoint() + assert cp.checkpoint_id # UUID auto-generated + assert cp.state == CheckpointState.PENDING + assert cp.version == 1 + assert cp.ttl_seconds == 3600.0 + assert cp.resume_attempts == 0 + assert cp.max_resume_attempts == 3 + assert cp.consumed_at is None + + def test_is_expired_false_when_fresh(self) -> None: + cp = RecoveryCheckpoint(created_at=time.time(), ttl_seconds=3600.0) + assert cp.is_expired is False + + def test_is_expired_true_when_past_ttl(self) -> None: + cp = RecoveryCheckpoint(created_at=time.time() - 7200, ttl_seconds=3600.0) + assert cp.is_expired is True + + def test_is_consumable_when_pending_and_fresh(self) -> None: + cp = RecoveryCheckpoint(created_at=time.time(), ttl_seconds=3600.0) + assert cp.is_consumable is True + + def test_is_consumable_false_when_expired(self) -> None: + cp = RecoveryCheckpoint(created_at=time.time() - 7200, ttl_seconds=3600.0) + assert cp.is_consumable is False + + def test_is_consumable_false_when_consumed(self) -> None: + cp = RecoveryCheckpoint(state=CheckpointState.CONSUMED) + assert cp.is_consumable is False + + def test_is_consumable_false_when_max_attempts_reached(self) -> None: + cp = RecoveryCheckpoint(resume_attempts=3, max_resume_attempts=3) + assert cp.is_consumable is False + + def test_expires_at(self) -> None: + now = time.time() + cp = RecoveryCheckpoint(created_at=now, ttl_seconds=1800.0) + assert cp.expires_at == pytest.approx(now + 1800.0, abs=0.01) + + def test_step_record_frozen(self) -> None: + step = StepRecord(step_id="s1", tool_name="bash", action="run") + assert step.step_id == "s1" + with pytest.raises(Exception): + step.step_id = "s2" # type: ignore[misc] + + def test_precondition_frozen(self) -> None: + pre = Precondition(key="session_id", expected_value="abc", critical=True) + assert pre.key == "session_id" + with pytest.raises(Exception): + pre.key = "other" # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# InMemoryCheckpointStore tests +# --------------------------------------------------------------------------- + + +class TestInMemoryCheckpointStore: + """Tests for the in-memory checkpoint store.""" + + def setup_method(self) -> None: + self.store = InMemoryCheckpointStore() + + def test_save_and_load_roundtrip(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-1", session_id="sess-1") + self.store.save(cp) + loaded = self.store.load("cp-1") + assert loaded is not None + assert loaded.checkpoint_id == "cp-1" + assert loaded.session_id == "sess-1" + + def test_load_returns_none_for_nonexistent(self) -> None: + assert self.store.load("nonexistent") is None + + def test_load_returns_none_for_expired(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-expired", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + self.store.save(cp) + assert self.store.load("cp-expired") is None + + def test_load_marks_expired_state(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-exp", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + self.store.save(cp) + result = self.store.load("cp-exp") + assert result is None + # Expired checkpoint is removed from store + assert self.store.load("cp-exp") is None + + def test_load_and_consume_cas_success(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-cas") + self.store.save(cp) + result = self.store.load_and_consume("cp-cas") + assert result is not None + assert result.state == CheckpointState.CONSUMED + assert result.consumed_at is not None + assert result.version == 2 # Bumped from 1 + + def test_load_and_consume_second_call_returns_none(self) -> None: + """CAS semantics: second consume attempt must fail.""" + cp = RecoveryCheckpoint(checkpoint_id="cp-cas2") + self.store.save(cp) + first = self.store.load_and_consume("cp-cas2") + assert first is not None + second = self.store.load_and_consume("cp-cas2") + assert second is None + + def test_load_and_consume_returns_none_for_expired(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-cas-exp", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + self.store.save(cp) + assert self.store.load_and_consume("cp-cas-exp") is None + assert cp.state == CheckpointState.EXPIRED + + def test_load_and_consume_respects_max_resume_attempts(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-attempts", + resume_attempts=3, + max_resume_attempts=3, + ) + self.store.save(cp) + assert self.store.load_and_consume("cp-attempts") is None + + def test_load_and_consume_nonexistent(self) -> None: + assert self.store.load_and_consume("nope") is None + + def test_list_pending_returns_pending_only(self) -> None: + cp1 = RecoveryCheckpoint(checkpoint_id="cp-p1", session_id="s1") + cp2 = RecoveryCheckpoint( + checkpoint_id="cp-p2", session_id="s1", + state=CheckpointState.CONSUMED, + ) + cp3 = RecoveryCheckpoint(checkpoint_id="cp-p3", session_id="s1") + self.store.save(cp1) + self.store.save(cp2) + self.store.save(cp3) + pending = self.store.list_pending() + ids = [c.checkpoint_id for c in pending] + assert "cp-p1" in ids + assert "cp-p3" in ids + assert "cp-p2" not in ids + + def test_list_pending_filters_by_session(self) -> None: + cp1 = RecoveryCheckpoint(checkpoint_id="cp-s1", session_id="alpha") + cp2 = RecoveryCheckpoint(checkpoint_id="cp-s2", session_id="beta") + self.store.save(cp1) + self.store.save(cp2) + result = self.store.list_pending(session_id="alpha") + assert len(result) == 1 + assert result[0].session_id == "alpha" + + def test_list_pending_excludes_expired(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-lp-exp", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + self.store.save(cp) + assert self.store.list_pending() == [] + assert cp.state == CheckpointState.EXPIRED + + def test_cancel_success(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-cancel") + self.store.save(cp) + assert self.store.cancel("cp-cancel") is True + assert cp.state == CheckpointState.CANCELLED + assert cp.version == 2 + + def test_cancel_fails_if_not_pending(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-cancel2", state=CheckpointState.CONSUMED, + ) + self.store.save(cp) + assert self.store.cancel("cp-cancel2") is False + + def test_cancel_nonexistent(self) -> None: + assert self.store.cancel("nope") is False + + def test_cleanup_expired_removes_old_entries(self) -> None: + cp1 = RecoveryCheckpoint( + checkpoint_id="cp-old", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + cp2 = RecoveryCheckpoint(checkpoint_id="cp-fresh") + self.store.save(cp1) + self.store.save(cp2) + removed = self.store.cleanup_expired() + assert removed == 1 + assert self.store.load("cp-old") is None + assert self.store.load("cp-fresh") is not None + + def test_cleanup_expired_returns_zero_when_none_expired(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-active") + self.store.save(cp) + assert self.store.cleanup_expired() == 0 + + def test_save_overwrites_existing(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-ow", session_id="v1") + self.store.save(cp) + cp2 = RecoveryCheckpoint(checkpoint_id="cp-ow", session_id="v2") + self.store.save(cp2) + loaded = self.store.load("cp-ow") + assert loaded is not None + assert loaded.session_id == "v2" + + +# --------------------------------------------------------------------------- +# CheckpointResumer tests +# --------------------------------------------------------------------------- + + +class TestCheckpointResumer: + """Tests for the checkpoint resume orchestration.""" + + def setup_method(self) -> None: + self.store = InMemoryCheckpointStore() + self.resumer = CheckpointResumer(store=self.store) + + def test_resume_success_path(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-resume", + session_id="s1", + messages_snapshot=[{"role": "user", "content": "hello"}], + failure_envelope_data={"message": "rate limit exceeded"}, + context_data={"plan_step": 3}, + ) + self.store.save(cp) + result = self.resumer.resume("cp-resume") + assert result.success is True + assert result.context_data == {"plan_step": 3} + # Messages include original + recovery injection + assert len(result.messages) == 2 + assert result.messages[0]["content"] == "hello" + assert "[Recovery]" in result.messages[1]["content"] + assert "rate limit exceeded" in result.messages[1]["content"] + + def test_resume_fails_on_expired_checkpoint(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-exp-resume", + created_at=time.time() - 7200, + ttl_seconds=3600.0, + ) + self.store.save(cp) + result = self.resumer.resume("cp-exp-resume") + assert result.success is False + assert "not available" in result.reason + + def test_resume_fails_on_consumed_checkpoint(self) -> None: + cp = RecoveryCheckpoint(checkpoint_id="cp-consumed-r") + self.store.save(cp) + # First consume succeeds + first = self.resumer.resume("cp-consumed-r") + assert first.success is True + # Second fails (CAS semantics) + second = self.resumer.resume("cp-consumed-r") + assert second.success is False + + def test_resume_fails_on_critical_precondition_drift(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-drift", + preconditions=( + Precondition( + key="session_id", + expected_value="sess-original", + critical=True, + ), + ), + ) + self.store.save(cp) + result = self.resumer.resume( + "cp-drift", + current_context={"session_id": "sess-different"}, + ) + assert result.success is False + assert "Critical precondition drift" in result.reason + assert "session_id" in result.reason + + def test_resume_succeeds_with_noncritical_drift(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-warn", + messages_snapshot=[{"role": "assistant", "content": "working..."}], + failure_envelope_data={"message": "timeout"}, + preconditions=( + Precondition( + key="workspace_root", + expected_value="/old/path", + critical=False, + description="Workspace may have changed", + ), + ), + ) + self.store.save(cp) + result = self.resumer.resume( + "cp-warn", + current_context={"workspace_root": "/new/path"}, + ) + assert result.success is True + assert len(result.drift_warnings) == 1 + assert "workspace_root" in result.drift_warnings[0] + + def test_resume_injects_recovery_message(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-msg", + messages_snapshot=[ + {"role": "user", "content": "deploy the service"}, + {"role": "assistant", "content": "I will deploy now"}, + ], + failure_envelope_data={"message": "connection refused"}, + ) + self.store.save(cp) + result = self.resumer.resume("cp-msg") + assert result.success is True + assert len(result.messages) == 3 + recovery_msg = result.messages[2] + assert recovery_msg["role"] == "system" + assert "connection refused" in recovery_msg["content"] + assert "Continue from where you left off" in recovery_msg["content"] + + def test_resume_nonexistent_checkpoint(self) -> None: + result = self.resumer.resume("no-such-id") + assert result.success is False + assert "not available" in result.reason + + def test_resume_with_multiple_preconditions(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-multi-pre", + messages_snapshot=[], + failure_envelope_data={"message": "error"}, + preconditions=( + Precondition(key="session_id", expected_value="s1", critical=True), + Precondition(key="config_hash", expected_value="abc", critical=True), + Precondition(key="env", expected_value="prod", critical=False), + ), + ) + self.store.save(cp) + result = self.resumer.resume( + "cp-multi-pre", + current_context={"session_id": "s1", "config_hash": "abc", "env": "dev"}, + ) + assert result.success is True + assert len(result.drift_warnings) == 1 + assert "env" in result.drift_warnings[0] + + +# --------------------------------------------------------------------------- +# Protocol conformance tests +# --------------------------------------------------------------------------- + + +class TestProtocolConformance: + """Verify InMemoryCheckpointStore satisfies CheckpointStore protocol.""" + + def test_inmemory_store_is_checkpoint_store(self) -> None: + store = InMemoryCheckpointStore() + assert isinstance(store, CheckpointStore) + + def test_default_drift_verifier(self) -> None: + verifier = DefaultDriftVerifier() + assert verifier.get_current_value("anything") == "" + + +# --------------------------------------------------------------------------- +# Edge case tests +# --------------------------------------------------------------------------- + + +class TestEdgeCases: + """Edge cases and boundary conditions.""" + + def test_checkpoint_with_zero_ttl_is_immediately_expired(self) -> None: + cp = RecoveryCheckpoint( + checkpoint_id="cp-zero-ttl", + created_at=time.time() - 0.001, + ttl_seconds=0.0, + ) + assert cp.is_expired is True + assert cp.is_consumable is False + + def test_checkpoint_with_completed_steps(self) -> None: + steps = ( + StepRecord(step_id="1", tool_name="bash", has_side_effect=True), + StepRecord(step_id="2", tool_name="read_file", has_side_effect=False), + ) + cp = RecoveryCheckpoint( + checkpoint_id="cp-steps", + completed_steps=steps, + pending_steps=("step 3: validate",), + ) + assert len(cp.completed_steps) == 2 + assert cp.completed_steps[0].has_side_effect is True + assert cp.pending_steps == ("step 3: validate",) + + def test_resume_result_frozen(self) -> None: + result = ResumeResult(success=True) + with pytest.raises(Exception): + result.success = False # type: ignore[misc] diff --git a/tests/test_recovery_coordinator.py b/tests/test_recovery_coordinator.py new file mode 100644 index 0000000..c943fb9 --- /dev/null +++ b/tests/test_recovery_coordinator.py @@ -0,0 +1,690 @@ +"""Comprehensive tests for the P0 recovery coordinator subsystem. + +Covers: +- FailureEnvelope construction and immutability +- RecoveryDecision construction and properties +- RecoveryBudget: can_afford, consume, deadline, per-category limits +- OneShotGuard: is_available, mark_used, idempotency +- RecoveryCoordinator: evaluate with mock strategies, priority ordering, + one-shot enforcement, terminal decision, budget exhaustion +""" +from __future__ import annotations + +import time +from dataclasses import dataclass + +import pytest + +from leapflow.engine.failure_envelope import ( + FailureContext, + FailureEnvelope, + FailureSource, + Recoverability, + RecoveryHint, + SideEffectState, +) +from leapflow.engine.oneshot_guard import OneShotGuard +from leapflow.engine.recovery_budget import RecoveryBudget +from leapflow.engine.recovery_coordinator import ( + RecoveryCoordinator, + RecoveryState, + RecoveryStrategy, +) +from leapflow.engine.recovery_decision import ( + BackoffConfig, + RecoveryAction, + RecoveryDecision, + RetrySemantics, +) + + +# --------------------------------------------------------------------------- +# Test helpers: mock strategies implementing the Protocol +# --------------------------------------------------------------------------- + + +@dataclass +class MockRetryStrategy: + """A mock strategy that always suggests retry with backoff.""" + + key: str = "mock_retry" + priority: int = 10 + repeatable: bool = True + applicable_sources: frozenset[str] = frozenset({"llm"}) + applicable_categories: frozenset[str] = frozenset({"rate_limited", "transient"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + return state.consecutive_failures < 5 + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.RETRY_WITH_BACKOFF, + reason="Transient error, retrying", + strategy_key=self.key, + retry_semantics=RetrySemantics( + consumes_retry_budget=True, + backoff_config=BackoffConfig(base_delay=1.0), + ), + budget_cost=1, + ) + + +@dataclass +class MockCompressStrategy: + """A mock strategy that suggests transform-and-retry for context overflow.""" + + key: str = "mock_compress" + priority: int = 5 + repeatable: bool = True + applicable_sources: frozenset[str] = frozenset() + applicable_categories: frozenset[str] = frozenset({"context_overflow"}) + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + return True + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Context overflow, compressing", + strategy_key=self.key, + budget_cost=1, + ) + + +@dataclass +class MockFailoverStrategy: + """A mock strategy that suggests failover.""" + + key: str = "mock_failover" + priority: int = 20 + repeatable: bool = False + applicable_sources: frozenset[str] = frozenset({"llm"}) + applicable_categories: frozenset[str] = frozenset() + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + return state.consecutive_failures >= 3 + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.FAILOVER, + reason="Too many consecutive failures, switching model", + strategy_key=self.key, + budget_cost=2, + ) + + +@dataclass +class MockCatchAllStrategy: + """Low priority strategy that matches anything.""" + + key: str = "mock_catchall" + priority: int = 100 + repeatable: bool = True + applicable_sources: frozenset[str] = frozenset() + applicable_categories: frozenset[str] = frozenset() + + def can_apply(self, envelope: FailureEnvelope, state: RecoveryState, + budget: RecoveryBudget | None = None) -> bool: + return True + + def decide(self, envelope: FailureEnvelope, state: RecoveryState) -> RecoveryDecision: + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.SKIP_AND_CONTINUE, + reason="Catch-all: skip and continue", + strategy_key=self.key, + budget_cost=0, + ) + + +# --------------------------------------------------------------------------- +# Fixture helpers +# --------------------------------------------------------------------------- + + +def _make_envelope( + *, + source: FailureSource = FailureSource.LLM, + category: str = "rate_limited", + recoverability: Recoverability = Recoverability.AUTO_RETRY, + side_effect_state: SideEffectState = SideEffectState.NONE, +) -> FailureEnvelope: + return FailureEnvelope.create( + source=source, + category=category, + failure_class="transient", + failure_code="rate_limit_exceeded", + message="Rate limit exceeded", + recoverability=recoverability, + side_effect_state=side_effect_state, + ) + + +# =========================================================================== +# FailureEnvelope Tests +# =========================================================================== + + +class TestFailureEnvelope: + def test_create_generates_id_and_timestamp(self) -> None: + env = _make_envelope() + assert len(env.envelope_id) == 32 + assert env.timestamp > 0 + assert env.source == FailureSource.LLM + assert env.category == "rate_limited" + + def test_frozen_immutability(self) -> None: + env = _make_envelope() + with pytest.raises(AttributeError): + env.message = "changed" # type: ignore[misc] + + def test_failure_context_from_dict_args(self) -> None: + ctx = FailureContext.from_dict_args( + tool_name="shell_run", + arguments={"cmd": "ls", "timeout": 30}, + execution_id="exec-1", + turn_id=5, + ) + assert ctx.tool_name == "shell_run" + assert ctx.arguments_dict == {"cmd": "ls", "timeout": 30} + assert ctx.turn_id == 5 + + def test_failure_context_empty(self) -> None: + ctx = FailureContext() + assert ctx.arguments == () + assert ctx.arguments_dict == {} + + def test_recovery_hint(self) -> None: + hint = RecoveryHint( + hint_text="Retry after 60s", + suggested_command="leap config llm key", + ) + assert hint.hint_text == "Retry after 60s" + assert hint.documentation_url == "" + + def test_envelope_with_context_and_hint(self) -> None: + ctx = FailureContext.from_dict_args(tool_name="gp_web_search") + hint = RecoveryHint(hint_text="Check API key") + env = FailureEnvelope.create( + source=FailureSource.TOOL, + category="auth_error", + failure_class="authorization", + failure_code="missing_scope", + message="Missing read scope", + recoverability=Recoverability.USER_FIXABLE, + context=ctx, + provider_hint=hint, + ) + assert env.context.tool_name == "gp_web_search" + assert env.provider_hint is not None + assert env.provider_hint.hint_text == "Check API key" + + +# =========================================================================== +# RecoveryDecision Tests +# =========================================================================== + + +class TestRecoveryDecision: + def test_create_generates_id(self) -> None: + env = _make_envelope() + dec = RecoveryDecision.create( + envelope=env, + action=RecoveryAction.RETRY_WITH_BACKOFF, + reason="Transient error", + strategy_key="test_strategy", + budget_cost=1, + ) + assert len(dec.decision_id) == 32 + assert dec.action == RecoveryAction.RETRY_WITH_BACKOFF + assert dec.strategy_key == "test_strategy" + + def test_is_terminal(self) -> None: + env = _make_envelope() + dec = RecoveryDecision.create( + envelope=env, + action=RecoveryAction.HALT_CLEAN, + reason="Budget exhausted", + strategy_key="test", + ) + assert dec.is_terminal is True + assert dec.is_retry is False + + def test_is_retry(self) -> None: + env = _make_envelope() + dec = RecoveryDecision.create( + envelope=env, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Compressing", + strategy_key="compress", + ) + assert dec.is_retry is True + assert dec.is_terminal is False + + def test_audit_metadata_roundtrip(self) -> None: + env = _make_envelope() + dec = RecoveryDecision.create( + envelope=env, + action=RecoveryAction.SKIP_AND_CONTINUE, + reason="Skip", + strategy_key="skip", + audit_metadata={"attempt": 3, "source": "test"}, + ) + assert dec.audit_metadata_dict == {"attempt": 3, "source": "test"} + + def test_backoff_config_frozen(self) -> None: + cfg = BackoffConfig(base_delay=2.0, max_delay=120.0) + with pytest.raises(AttributeError): + cfg.base_delay = 5.0 # type: ignore[misc] + + def test_retry_semantics_defaults(self) -> None: + sem = RetrySemantics() + assert sem.consumes_retry_budget is True + assert sem.resets_retry_count is False + assert sem.backoff_config is None + + +# =========================================================================== +# RecoveryBudget Tests +# =========================================================================== + + +class TestRecoveryBudget: + def test_initial_state(self) -> None: + budget = RecoveryBudget() + assert budget.remaining() == 12 + assert budget.can_afford(1) is True + assert budget.is_deadline_exceeded() is False + + def test_consume_reduces_remaining(self) -> None: + budget = RecoveryBudget(total_recovery_actions=5) + budget.start_deadline() + budget.consume(2) + assert budget.remaining() == 3 + budget.consume(3) + assert budget.remaining() == 0 + assert budget.can_afford(1) is False + + def test_consume_raises_on_overbudget(self) -> None: + budget = RecoveryBudget(total_recovery_actions=3) + budget.start_deadline() + budget.consume(3) + with pytest.raises(ValueError, match="Recovery budget exceeded"): + budget.consume(1) + + def test_per_category_limits(self) -> None: + budget = RecoveryBudget(max_retry_per_category=2, total_recovery_actions=10) + budget.start_deadline() + assert budget.category_remaining("rate_limited") == 2 + budget.consume(1, "rate_limited") + assert budget.category_remaining("rate_limited") == 1 + budget.consume(1, "rate_limited") + assert budget.category_remaining("rate_limited") == 0 + assert budget.can_afford(1, "rate_limited") is False + # Other category is still available + assert budget.can_afford(1, "transient") is True + + def test_deadline_exceeded(self) -> None: + budget = RecoveryBudget(turn_deadline_s=0.01) + budget.start_deadline() + time.sleep(0.02) + assert budget.is_deadline_exceeded() is True + assert budget.can_afford(1) is False + + def test_deadline_not_started(self) -> None: + budget = RecoveryBudget(turn_deadline_s=0.001) + # Deadline not started — should never be exceeded + assert budget.is_deadline_exceeded() is False + + def test_transform_failover_rotation_tracking(self) -> None: + budget = RecoveryBudget( + max_transform_attempts=1, + max_failovers=1, + max_credential_rotations=1, + ) + assert budget.can_transform() is True + budget.consume_transform() + assert budget.can_transform() is False + + assert budget.can_failover() is True + budget.consume_failover() + assert budget.can_failover() is False + + assert budget.can_rotate() is True + budget.consume_rotation() + assert budget.can_rotate() is False + + def test_summary(self) -> None: + budget = RecoveryBudget(total_recovery_actions=10) + budget.start_deadline() + budget.consume(2, "rate_limited") + budget.consume_transform() + summary = budget.summary() + assert summary["consumed"] == 2 + assert summary["remaining"] == 8 + assert summary["category_consumed"] == {"rate_limited": 2} + assert summary["transforms_used"] == 1 + + +# =========================================================================== +# OneShotGuard Tests +# =========================================================================== + + +class TestOneShotGuard: + def test_fresh_guard_all_available(self) -> None: + guard = OneShotGuard() + assert guard.is_available("compress") is True + assert guard.is_available("failover") is True + assert len(guard) == 0 + + def test_mark_used_makes_unavailable(self) -> None: + guard = OneShotGuard() + guard.mark_used("compress") + assert guard.is_available("compress") is False + assert guard.is_available("failover") is True + assert len(guard) == 1 + + def test_mark_used_is_idempotent(self) -> None: + guard = OneShotGuard() + guard.mark_used("compress") + ts1 = guard.usage_history()["compress"] + time.sleep(0.01) + guard.mark_used("compress") + ts2 = guard.usage_history()["compress"] + # Timestamp should not change on re-marking + assert ts1 == ts2 + + def test_used_strategies_ordered_by_time(self) -> None: + guard = OneShotGuard() + guard.mark_used("a") + time.sleep(0.01) + guard.mark_used("b") + strategies = guard.used_strategies() + assert strategies == ["a", "b"] + + def test_contains_operator(self) -> None: + guard = OneShotGuard() + assert "x" not in guard + guard.mark_used("x") + assert "x" in guard + + def test_reset_clears_all(self) -> None: + guard = OneShotGuard() + guard.mark_used("a") + guard.mark_used("b") + guard.reset() + assert guard.is_available("a") is True + assert len(guard) == 0 + + +# =========================================================================== +# RecoveryCoordinator Tests +# =========================================================================== + + +class TestRecoveryCoordinator: + def test_evaluate_matches_strategy_by_source_and_category(self) -> None: + strategies: list[RecoveryStrategy] = [MockRetryStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope(source=FailureSource.LLM, category="rate_limited") + decision = coord.evaluate(env) + + assert decision.action == RecoveryAction.RETRY_WITH_BACKOFF + assert decision.strategy_key == "mock_retry" + + def test_evaluate_priority_ordering(self) -> None: + """Higher priority (lower number) strategies are evaluated first.""" + # MockCompressStrategy has priority 5, MockRetryStrategy has 10 + strategies: list[RecoveryStrategy] = [ + MockRetryStrategy(), + MockCompressStrategy(), + ] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + # context_overflow matches compress (priority 5) before retry + env = _make_envelope(category="context_overflow") + decision = coord.evaluate(env) + assert decision.action == RecoveryAction.TRANSFORM_AND_RETRY + assert decision.strategy_key == "mock_compress" + + def test_evaluate_skips_non_matching_source(self) -> None: + """Strategy with source filter skips non-matching envelopes.""" + strategies: list[RecoveryStrategy] = [ + MockRetryStrategy(), # only matches LLM source + MockCatchAllStrategy(), + ] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope(source=FailureSource.TOOL, category="rate_limited") + decision = coord.evaluate(env) + # Falls through to catch-all + assert decision.strategy_key == "mock_catchall" + + def test_evaluate_one_shot_enforcement(self) -> None: + """Failover marks one-shot, subsequent calls skip it.""" + strategies: list[RecoveryStrategy] = [ + MockFailoverStrategy(), + MockCatchAllStrategy(), + ] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + # Set consecutive failures high enough for failover + coord.state.consecutive_failures = 5 + + env = _make_envelope(source=FailureSource.LLM, category="transient") + decision1 = coord.evaluate(env) + assert decision1.action == RecoveryAction.FAILOVER + + # Second call should skip failover (one-shot used) and fall to catch-all + coord.state.consecutive_failures = 5 + decision2 = coord.evaluate(env) + assert decision2.strategy_key == "mock_catchall" + + def test_evaluate_terminal_on_no_match(self) -> None: + """When no strategy matches, a terminal HALT_CLEAN decision is returned.""" + coord = RecoveryCoordinator(strategies=[]) + coord.budget.start_deadline() + + env = _make_envelope() + decision = coord.evaluate(env) + assert decision.action == RecoveryAction.HALT_CLEAN + assert "No applicable" in decision.reason + + def test_evaluate_terminal_on_budget_exhaustion(self) -> None: + """When budget is exhausted, terminal decision regardless of strategies.""" + budget = RecoveryBudget(total_recovery_actions=0) + budget.start_deadline() + strategies: list[RecoveryStrategy] = [MockRetryStrategy()] + coord = RecoveryCoordinator(strategies=strategies, budget=budget) + + env = _make_envelope() + decision = coord.evaluate(env) + assert decision.action == RecoveryAction.HALT_CLEAN + assert "exhausted" in decision.reason.lower() + + def test_evaluate_terminal_on_deadline_exceeded(self) -> None: + """When deadline is exceeded, terminal decision immediately.""" + budget = RecoveryBudget(turn_deadline_s=0.01) + budget.start_deadline() + time.sleep(0.02) + strategies: list[RecoveryStrategy] = [MockRetryStrategy()] + coord = RecoveryCoordinator(strategies=strategies, budget=budget) + + env = _make_envelope() + decision = coord.evaluate(env) + assert decision.action == RecoveryAction.HALT_CLEAN + assert "deadline" in decision.reason.lower() + + def test_evaluate_non_recoverable_short_circuits(self) -> None: + """Non-recoverable envelopes skip all strategies.""" + strategies: list[RecoveryStrategy] = [MockCatchAllStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope(recoverability=Recoverability.NON_RECOVERABLE) + decision = coord.evaluate(env) + assert decision.action == RecoveryAction.HALT_CLEAN + assert "non-recoverable" in decision.reason.lower() + + def test_evaluate_budget_cost_check(self) -> None: + """Strategy with cost > remaining budget is skipped.""" + budget = RecoveryBudget(total_recovery_actions=1) + budget.start_deadline() + # MockFailoverStrategy costs 2 + strategies: list[RecoveryStrategy] = [ + MockFailoverStrategy(), + MockCatchAllStrategy(), # costs 0 + ] + coord = RecoveryCoordinator(strategies=strategies, budget=budget) + coord.state.consecutive_failures = 5 + + env = _make_envelope(source=FailureSource.LLM) + decision = coord.evaluate(env) + # Failover costs 2 but budget only has 1, should skip to catch-all + assert decision.strategy_key == "mock_catchall" + + def test_on_strategy_outcome_success_resets_counters(self) -> None: + coord = RecoveryCoordinator() + coord.state.consecutive_failures = 3 + coord.state.consecutive_api_errors = 2 + coord.on_strategy_outcome("dec-1", success=True) + assert coord.state.consecutive_failures == 0 + assert coord.state.consecutive_api_errors == 0 + + def test_on_strategy_outcome_failure_increments(self) -> None: + coord = RecoveryCoordinator() + coord.state.consecutive_failures = 2 + coord.on_strategy_outcome("dec-1", success=False) + assert coord.state.consecutive_failures == 3 + + def test_record_success(self) -> None: + coord = RecoveryCoordinator() + coord.state.consecutive_failures = 5 + coord.state.consecutive_api_errors = 3 + coord.state.last_error_category = "rate_limited" + coord.record_success() + assert coord.state.consecutive_failures == 0 + assert coord.state.consecutive_api_errors == 0 + assert coord.state.last_error_category == "" + + def test_audit_log_populated(self) -> None: + strategies: list[RecoveryStrategy] = [MockCatchAllStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope() + coord.evaluate(env) + assert len(coord.audit_log) == 1 + entry = coord.audit_log[0] + assert entry["event"] == "recovery_decision" + assert entry["strategy_key"] == "mock_catchall" + assert "budget_remaining" in entry + + def test_protocol_compliance(self) -> None: + """Mock strategies satisfy the RecoveryStrategy protocol.""" + assert isinstance(MockRetryStrategy(), RecoveryStrategy) + assert isinstance(MockCompressStrategy(), RecoveryStrategy) + assert isinstance(MockFailoverStrategy(), RecoveryStrategy) + assert isinstance(MockCatchAllStrategy(), RecoveryStrategy) + + def test_multiple_evaluations_consume_budget(self) -> None: + """Multiple evaluations consume budget incrementally.""" + budget = RecoveryBudget(total_recovery_actions=3) + budget.start_deadline() + strategies: list[RecoveryStrategy] = [MockRetryStrategy()] + coord = RecoveryCoordinator(strategies=strategies, budget=budget) + + env = _make_envelope() + d1 = coord.evaluate(env) + assert d1.action == RecoveryAction.RETRY_WITH_BACKOFF + d2 = coord.evaluate(env) + assert d2.action == RecoveryAction.RETRY_WITH_BACKOFF + d3 = coord.evaluate(env) + assert d3.action == RecoveryAction.RETRY_WITH_BACKOFF + # Budget now exhausted + d4 = coord.evaluate(env) + assert d4.action == RecoveryAction.HALT_CLEAN + + def test_new_turn_resets_guard_and_state(self) -> None: + """new_turn() resets one-shot guard and per-turn state.""" + strategies: list[RecoveryStrategy] = [MockFailoverStrategy(), MockCatchAllStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + coord.state.consecutive_failures = 5 + + env = _make_envelope(source=FailureSource.LLM, category="transient") + d1 = coord.evaluate(env) + assert d1.action == RecoveryAction.FAILOVER + + # Failover is one-shot, should fall to catch-all + coord.state.consecutive_failures = 5 + d2 = coord.evaluate(env) + assert d2.strategy_key == "mock_catchall" + + # After new_turn, failover should be available again + coord.new_turn(turn_id=2) + coord.state.consecutive_failures = 5 + d3 = coord.evaluate(env) + assert d3.action == RecoveryAction.FAILOVER + assert coord.state.current_turn_id == 2 + + def test_non_repeatable_transform_is_one_shot(self) -> None: + """Non-repeatable TRANSFORM_AND_RETRY strategies fire only once.""" + + @dataclass + class MockTransformStrategy: + key: str = "mock_transform" + priority: int = 5 + repeatable: bool = False + applicable_sources: frozenset[str] = frozenset() + applicable_categories: frozenset[str] = frozenset({"format_error"}) + + def can_apply(self, envelope, state, budget=None): + return True + + def decide(self, envelope, state): + return RecoveryDecision.create( + envelope=envelope, + action=RecoveryAction.TRANSFORM_AND_RETRY, + reason="Transform once", + strategy_key=self.key, + budget_cost=0, + ) + + strategies: list[RecoveryStrategy] = [MockTransformStrategy(), MockCatchAllStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope(category="format_error") + d1 = coord.evaluate(env) + assert d1.action == RecoveryAction.TRANSFORM_AND_RETRY + assert d1.strategy_key == "mock_transform" + + # Second call should skip transform (one-shot marked) and fall to catch-all + d2 = coord.evaluate(env) + assert d2.strategy_key == "mock_catchall" + + def test_repeatable_strategy_fires_multiple_times(self) -> None: + """Repeatable strategies can fire multiple times per turn.""" + strategies: list[RecoveryStrategy] = [MockRetryStrategy()] + coord = RecoveryCoordinator(strategies=strategies) + coord.budget.start_deadline() + + env = _make_envelope(source=FailureSource.LLM, category="rate_limited") + d1 = coord.evaluate(env) + assert d1.strategy_key == "mock_retry" + d2 = coord.evaluate(env) + assert d2.strategy_key == "mock_retry" + d3 = coord.evaluate(env) + assert d3.strategy_key == "mock_retry" diff --git a/tests/test_recovery_strategies.py b/tests/test_recovery_strategies.py new file mode 100644 index 0000000..0348df8 --- /dev/null +++ b/tests/test_recovery_strategies.py @@ -0,0 +1,473 @@ +"""Tests for built-in recovery strategies. + +Covers each strategy: +- can_apply returns True for matching envelope and False for non-matching +- decide returns correct RecoveryAction and RetrySemantics +- Priority ordering is correct +- Strategies respect budget limits +- JitteredRetry uses different BackoffConfig based on category +""" +from __future__ import annotations + +import pytest + +from leapflow.engine.failure_envelope import ( + FailureContext, + FailureEnvelope, + FailureSource, + Recoverability, + SideEffectState, +) +from leapflow.engine.recovery_coordinator import RecoveryState, RecoveryStrategy +from leapflow.engine.recovery_decision import ( + BackoffConfig, + RecoveryAction, + RetrySemantics, +) +from leapflow.engine.recovery_strategies import ( + ContextCompressStrategy, + CredentialRotateStrategy, + JitteredRetryStrategy, + MultimodalStripStrategy, + NativeToTextFallbackStrategy, + ProviderFailoverStrategy, + ThinkingDisableStrategy, + ToolSchemaExpandStrategy, + default_strategies, +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_envelope( + *, + source: FailureSource = FailureSource.LLM, + category: str = "transient", + message: str = "test error", + recoverability: Recoverability = Recoverability.AUTO_RETRY, + tool_name: str = "", +) -> FailureEnvelope: + return FailureEnvelope.create( + source=source, + category=category, + failure_class="test", + failure_code="test_code", + message=message, + recoverability=recoverability, + context=FailureContext.from_dict_args(tool_name=tool_name), + ) + + +def _fresh_state() -> RecoveryState: + return RecoveryState() + + +# =========================================================================== +# Protocol Compliance Tests +# =========================================================================== + + +class TestProtocolCompliance: + def test_all_strategies_implement_protocol(self) -> None: + strategies = default_strategies() + for s in strategies: + assert isinstance(s, RecoveryStrategy), f"{s.__class__.__name__} does not implement Protocol" + + def test_default_strategies_priority_ordering(self) -> None: + strategies = default_strategies() + priorities = [s.priority for s in strategies] + assert priorities == sorted(priorities), "Strategies should be in priority order" + + def test_all_strategies_have_unique_keys(self) -> None: + strategies = default_strategies() + keys = [s.key for s in strategies] + assert len(keys) == len(set(keys)), "Strategy keys must be unique" + + def test_strategy_count(self) -> None: + strategies = default_strategies() + assert len(strategies) == 8 + + +# =========================================================================== +# ContextCompressStrategy Tests +# =========================================================================== + + +class TestContextCompressStrategy: + def test_priority(self) -> None: + s = ContextCompressStrategy() + assert s.priority == 10 + + def test_applicable_sources(self) -> None: + s = ContextCompressStrategy() + assert s.applicable_sources == frozenset({"llm"}) + + def test_applicable_categories(self) -> None: + s = ContextCompressStrategy() + assert s.applicable_categories == frozenset({"context_overflow", "payload_too_large"}) + + def test_can_apply_fresh_state(self) -> None: + s = ContextCompressStrategy() + env = _make_envelope(category="context_overflow") + state = _fresh_state() + assert s.can_apply(env, state) is True + + def test_can_apply_exhausted_phases(self) -> None: + s = ContextCompressStrategy() + env = _make_envelope(category="context_overflow") + state = _fresh_state() + state.compress_phase_index = 3 + assert s.can_apply(env, state) is False + + def test_decide_phase_progression(self) -> None: + s = ContextCompressStrategy() + env = _make_envelope(category="context_overflow") + state = _fresh_state() + + d1 = s.decide(env, state) + assert d1.action == RecoveryAction.TRANSFORM_AND_RETRY + assert "history_summarize" in d1.reason + assert d1.audit_metadata_dict["phase_index"] == 0 + # decide() no longer mutates state; coordinator handles phase advancement + assert state.compress_phase_index == 0 + + # Simulate coordinator committing phase advancement + state.compress_phase_index = 1 + d2 = s.decide(env, state) + assert "multimodal_to_text" in d2.reason + assert d2.audit_metadata_dict["phase_index"] == 1 + + state.compress_phase_index = 2 + d3 = s.decide(env, state) + assert "disclosure_shrink" in d3.reason + assert d3.audit_metadata_dict["phase_index"] == 2 + + def test_decide_does_not_consume_budget(self) -> None: + s = ContextCompressStrategy() + env = _make_envelope(category="context_overflow") + state = _fresh_state() + decision = s.decide(env, state) + assert decision.retry_semantics.consumes_retry_budget is False + assert decision.budget_cost == 0 + + +# =========================================================================== +# MultimodalStripStrategy Tests +# =========================================================================== + + +class TestMultimodalStripStrategy: + def test_priority(self) -> None: + assert MultimodalStripStrategy().priority == 15 + + def test_can_apply_with_image_message(self) -> None: + s = MultimodalStripStrategy() + env = _make_envelope(category="image_too_large", message="Image file too large to encode") + assert s.can_apply(env, _fresh_state()) is True + + def test_can_apply_no_image_keyword(self) -> None: + s = MultimodalStripStrategy() + env = _make_envelope(category="image_too_large", message="random error text") + # Still applies because category itself is image_too_large + assert s.can_apply(env, _fresh_state()) is True + + def test_decide_action(self) -> None: + s = MultimodalStripStrategy() + env = _make_envelope(category="image_too_large", message="Image too large") + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.TRANSFORM_AND_RETRY + assert decision.retry_semantics.consumes_retry_budget is False + + +# =========================================================================== +# ProviderFailoverStrategy Tests +# =========================================================================== + + +class TestProviderFailoverStrategy: + def test_priority(self) -> None: + assert ProviderFailoverStrategy().priority == 20 + + def test_applicable_categories(self) -> None: + s = ProviderFailoverStrategy() + expected = frozenset({"billing", "auth_permanent", "overloaded", "model_not_found", "content_blocked"}) + assert s.applicable_categories == expected + + def test_can_apply(self) -> None: + s = ProviderFailoverStrategy() + env = _make_envelope(category="billing") + assert s.can_apply(env, _fresh_state()) is True + + def test_decide_action(self) -> None: + s = ProviderFailoverStrategy() + env = _make_envelope(category="billing") + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.FAILOVER + assert decision.retry_semantics.consumes_retry_budget is True + assert decision.retry_semantics.resets_retry_count is True + assert decision.budget_cost == 1 + + +# =========================================================================== +# CredentialRotateStrategy Tests +# =========================================================================== + + +class TestCredentialRotateStrategy: + def test_priority(self) -> None: + assert CredentialRotateStrategy().priority == 25 + + def test_applicable_categories(self) -> None: + s = CredentialRotateStrategy() + assert s.applicable_categories == frozenset({"auth_error", "rate_limited", "billing"}) + + def test_decide_action(self) -> None: + s = CredentialRotateStrategy() + env = _make_envelope(category="auth_error") + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.FAILOVER + assert decision.retry_semantics.consumes_retry_budget is True + assert decision.retry_semantics.resets_retry_count is True + + +# =========================================================================== +# ThinkingDisableStrategy Tests +# =========================================================================== + + +class TestThinkingDisableStrategy: + def test_priority(self) -> None: + assert ThinkingDisableStrategy().priority == 30 + + def test_applicable_categories(self) -> None: + assert ThinkingDisableStrategy().applicable_categories == frozenset({"format_error"}) + + def test_can_apply_always_true(self) -> None: + s = ThinkingDisableStrategy() + env = _make_envelope(category="format_error") + assert s.can_apply(env, _fresh_state()) is True + + def test_decide_action(self) -> None: + s = ThinkingDisableStrategy() + env = _make_envelope(category="format_error") + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.TRANSFORM_AND_RETRY + assert "thinking mode" in decision.reason.lower() + assert decision.retry_semantics.consumes_retry_budget is False + + +# =========================================================================== +# NativeToTextFallbackStrategy Tests +# =========================================================================== + + +class TestNativeToTextFallbackStrategy: + def test_priority(self) -> None: + assert NativeToTextFallbackStrategy().priority == 35 + + def test_can_apply_with_tool_call_message(self) -> None: + s = NativeToTextFallbackStrategy() + env = _make_envelope(category="format_error", message="Failed to parse tool_call response") + assert s.can_apply(env, _fresh_state()) is True + + def test_can_apply_with_native_message(self) -> None: + s = NativeToTextFallbackStrategy() + env = _make_envelope(category="format_error", message="native function calling error") + assert s.can_apply(env, _fresh_state()) is True + + def test_can_apply_without_keywords(self) -> None: + s = NativeToTextFallbackStrategy() + env = _make_envelope(category="format_error", message="generic format error") + assert s.can_apply(env, _fresh_state()) is False + + def test_decide_action(self) -> None: + s = NativeToTextFallbackStrategy() + env = _make_envelope(category="format_error", message="tool_call parse failed") + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.TRANSFORM_AND_RETRY + assert "text mode" in decision.reason.lower() + assert decision.retry_semantics.consumes_retry_budget is False + + +# =========================================================================== +# ToolSchemaExpandStrategy Tests +# =========================================================================== + + +class TestToolSchemaExpandStrategy: + def test_priority(self) -> None: + assert ToolSchemaExpandStrategy().priority == 40 + + def test_applicable_sources(self) -> None: + assert ToolSchemaExpandStrategy().applicable_sources == frozenset({"tool"}) + + def test_applicable_categories(self) -> None: + assert ToolSchemaExpandStrategy().applicable_categories == frozenset({"tool_unknown"}) + + def test_can_apply_auto_recover(self) -> None: + s = ToolSchemaExpandStrategy() + env = _make_envelope( + source=FailureSource.TOOL, + category="tool_unknown", + recoverability=Recoverability.AUTO_RECOVER, + tool_name="web_search", + ) + assert s.can_apply(env, _fresh_state()) is True + + def test_can_apply_not_auto_recover(self) -> None: + s = ToolSchemaExpandStrategy() + env = _make_envelope( + source=FailureSource.TOOL, + category="tool_unknown", + recoverability=Recoverability.NON_RECOVERABLE, + ) + assert s.can_apply(env, _fresh_state()) is False + + def test_decide_action(self) -> None: + s = ToolSchemaExpandStrategy() + env = _make_envelope( + source=FailureSource.TOOL, + category="tool_unknown", + recoverability=Recoverability.AUTO_RECOVER, + tool_name="advanced_search", + ) + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.TRANSFORM_AND_RETRY + assert "schema" in decision.reason.lower() + assert decision.budget_cost == 0 + + +# =========================================================================== +# JitteredRetryStrategy Tests +# =========================================================================== + + +class TestJitteredRetryStrategy: + def test_priority(self) -> None: + assert JitteredRetryStrategy().priority == 100 + + def test_applicable_sources(self) -> None: + assert JitteredRetryStrategy().applicable_sources == frozenset({"llm", "tool", "system"}) + + def test_applicable_categories(self) -> None: + # Empty frozenset = wildcard, matches all categories + assert JitteredRetryStrategy().applicable_categories == frozenset() + + def test_can_apply_auto_retry(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="transient", recoverability=Recoverability.AUTO_RETRY) + assert s.can_apply(env, _fresh_state()) is True + + def test_can_apply_not_auto_retry(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="transient", recoverability=Recoverability.USER_FIXABLE) + assert s.can_apply(env, _fresh_state()) is False + + def test_decide_rate_limited_backoff(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="rate_limited", recoverability=Recoverability.AUTO_RETRY) + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.RETRY_WITH_BACKOFF + assert decision.retry_semantics.backoff_config is not None + assert decision.retry_semantics.backoff_config.base_delay == 5.0 + assert decision.retry_semantics.backoff_config.max_delay == 120.0 + assert decision.retry_semantics.consumes_retry_budget is True + assert decision.budget_cost == 1 + + def test_decide_transient_backoff(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="transient", recoverability=Recoverability.AUTO_RETRY) + decision = s.decide(env, _fresh_state()) + assert decision.retry_semantics.backoff_config is not None + assert decision.retry_semantics.backoff_config.base_delay == 1.0 + assert decision.retry_semantics.backoff_config.max_delay == 60.0 + + def test_decide_overloaded_backoff(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="overloaded", recoverability=Recoverability.AUTO_RETRY) + decision = s.decide(env, _fresh_state()) + assert decision.retry_semantics.backoff_config is not None + assert decision.retry_semantics.backoff_config.base_delay == 1.0 + assert decision.retry_semantics.backoff_config.max_delay == 60.0 + + def test_decide_tool_timeout_backoff(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope( + source=FailureSource.TOOL, + category="tool_timeout", + recoverability=Recoverability.AUTO_RETRY, + ) + decision = s.decide(env, _fresh_state()) + assert decision.retry_semantics.backoff_config is not None + assert decision.retry_semantics.backoff_config.base_delay == 2.0 + assert decision.retry_semantics.backoff_config.max_delay == 30.0 + + def test_decide_does_not_reset_retry_count(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope(category="transient", recoverability=Recoverability.AUTO_RETRY) + decision = s.decide(env, _fresh_state()) + assert decision.retry_semantics.resets_retry_count is False + + def test_decide_system_timeout(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope( + source=FailureSource.SYSTEM, + category="system_timeout", + recoverability=Recoverability.AUTO_RETRY, + ) + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.RETRY_WITH_BACKOFF + assert decision.retry_semantics.backoff_config is not None + + def test_decide_system_network(self) -> None: + s = JitteredRetryStrategy() + env = _make_envelope( + source=FailureSource.SYSTEM, + category="system_network", + recoverability=Recoverability.AUTO_RETRY, + ) + decision = s.decide(env, _fresh_state()) + assert decision.action == RecoveryAction.RETRY_WITH_BACKOFF + + +# =========================================================================== +# Integration: Strategy Priority Ordering +# =========================================================================== + + +class TestStrategyPriorityOrdering: + def test_priorities_are_in_expected_order(self) -> None: + strategies = default_strategies() + expected_keys = [ + "context_compress", + "multimodal_strip", + "provider_failover", + "credential_rotate", + "thinking_disable", + "native_to_text", + "tool_schema_expand", + "jittered_retry", + ] + actual_keys = [s.key for s in strategies] + assert actual_keys == expected_keys + + def test_priorities_are_strictly_increasing(self) -> None: + strategies = default_strategies() + for i in range(len(strategies) - 1): + assert strategies[i].priority < strategies[i + 1].priority, ( + f"{strategies[i].key} (priority={strategies[i].priority}) should be " + f"lower than {strategies[i+1].key} (priority={strategies[i+1].priority})" + ) + + def test_repeatable_property_correctness(self) -> None: + strategies = default_strategies() + repeatable_keys = {s.key for s in strategies if s.repeatable} + non_repeatable_keys = {s.key for s in strategies if not s.repeatable} + assert repeatable_keys == {"context_compress", "jittered_retry"} + assert non_repeatable_keys == { + "multimodal_strip", "provider_failover", "credential_rotate", + "thinking_disable", "native_to_text", "tool_schema_expand", + } diff --git a/tests/test_scm_tools.py b/tests/test_scm_tools.py new file mode 100644 index 0000000..8f716a3 --- /dev/null +++ b/tests/test_scm_tools.py @@ -0,0 +1,76 @@ +from __future__ import annotations + +import pytest + +from leapflow.tools.scm_tools import GitCommandResult, scm_sync + + +@pytest.mark.asyncio +async def test_scm_sync_pull_then_push_defaults_to_current_branch(tmp_path) -> None: + commands: list[tuple[str, ...]] = [] + + async def fake_runner(args, cwd, timeout_s): + commands.append(tuple(args)) + if tuple(args) == ("branch", "--show-current"): + return GitCommandResult(returncode=0, stdout="feature/refactor\n", stderr="") + return GitCommandResult(returncode=0, stdout="ok", stderr="") + + result = await scm_sync( + { + "action": "pull_then_push", + "cwd": str(tmp_path), + "remote": "origin", + "pull_ref": "main", + }, + runner=fake_runner, + ) + + assert result["ok"] is True + assert result["push_ref"] == "feature/refactor" + assert commands == [ + ("branch", "--show-current"), + ("pull", "origin", "main"), + ("push", "origin", "feature/refactor"), + ] + + +@pytest.mark.asyncio +async def test_scm_sync_pull_failure_does_not_push(tmp_path) -> None: + commands: list[tuple[str, ...]] = [] + + async def fake_runner(args, cwd, timeout_s): + commands.append(tuple(args)) + if tuple(args) == ("branch", "--show-current"): + return GitCommandResult(returncode=0, stdout="feature/refactor\n", stderr="") + if tuple(args) == ("pull", "origin", "main"): + return GitCommandResult(returncode=1, stdout="", stderr="fatal: refusing to merge") + raise AssertionError(f"unexpected git command: {args}") + + result = await scm_sync( + { + "action": "pull_then_push", + "cwd": str(tmp_path), + "remote": "origin", + "pull_ref": "main", + }, + runner=fake_runner, + ) + + assert result["ok"] is False + assert result["failed_step"] == "pull" + assert result["error"] == "fatal: refusing to merge" + assert commands == [ + ("branch", "--show-current"), + ("pull", "origin", "main"), + ] + + +@pytest.mark.asyncio +async def test_scm_sync_reports_missing_working_directory(tmp_path) -> None: + missing = tmp_path / "missing" + + result = await scm_sync({"action": "status", "cwd": str(missing)}) + + assert result["ok"] is False + assert result["failure_code"] == "path_not_found" + assert str(missing) in result["error"] diff --git a/tests/test_tui_command_queue.py b/tests/test_tui_command_queue.py index 6c9db5e..86aabde 100644 --- a/tests/test_tui_command_queue.py +++ b/tests/test_tui_command_queue.py @@ -221,6 +221,20 @@ def test_submit_text_assigns_ids_and_keeps_input_editable() -> None: assert app._input_area.buffer.read_only() is False +def test_submit_text_skips_duplicate_pending_command() -> None: + app, console, status = _make_app() + + first = app.submit_text("repeat me") + duplicate = app.submit_text(" repeat me ") + + assert first.status is TuiCommandStatus.QUEUED + assert duplicate.status is TuiCommandStatus.SKIPPED + assert duplicate.error == "duplicate queued/running command skipped" + assert app._pending_input.qsize() == 1 + assert [card.status for card in console.cards] == [TuiCommandStatus.SKIPPED] + assert status.counts[-1] == (0, 1) + + def test_submit_text_rejects_empty_commands() -> None: app, console, status = _make_app() @@ -606,6 +620,47 @@ def newline(self) -> None: assert " | " in line +def test_stream_renderer_hides_duplicate_suppression_metadata() -> None: + class CaptureConsole: + def __init__(self) -> None: + self.lines: list[str] = [] + + def print(self, renderable) -> None: + self.lines.append(getattr(renderable, "plain", str(renderable))) + + def thinking(self, text: str) -> None: + pass + + def markdown(self, text: str, *, indent: int = 0, margin_top: int = 0) -> None: + pass + + def response_label(self, elapsed_s: float, *, tool_count: int = 0, command=None) -> None: + pass + + def newline(self) -> None: + pass + + console = CaptureConsole() + renderer = StreamRenderer(console) # type: ignore[arg-type] + renderer.start() + + renderer.tool_started("shell_run", metadata={"command": "git push"}) + renderer.tool_finished( + "shell_run", + metadata={ + "ok": False, + "already_executed": True, + "duplicate_suppressed": True, + "counts_as_failure": False, + "ui_hidden": True, + "error_preview": "duplicate execution was not replayed", + }, + ) + + assert console.lines == [] + assert renderer.tool_count == 0 + + def test_stream_renderer_prints_context_evidence_metadata() -> None: class CaptureConsole: def __init__(self) -> None: diff --git a/tests/test_unified_classifier.py b/tests/test_unified_classifier.py new file mode 100644 index 0000000..9cf4d52 --- /dev/null +++ b/tests/test_unified_classifier.py @@ -0,0 +1,335 @@ +"""Tests for the UnifiedErrorClassifier. + +Covers: +- classify_llm_error produces correct FailureEnvelope for each ErrorCategory +- classify_tool_result returns None for non-failures +- classify_tool_result classifies permission failures, timeouts, unknown tools +- classify_tool_result maps execution_policy to SideEffectState +- classify_system_error handles common exceptions +""" +from __future__ import annotations + +import pytest + +from leapflow.engine.failure_envelope import ( + FailureEnvelope, + FailureSource, + Recoverability, + SideEffectState, +) +from leapflow.engine.unified_classifier import UnifiedErrorClassifier + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture +def classifier() -> UnifiedErrorClassifier: + return UnifiedErrorClassifier() + + +# =========================================================================== +# classify_llm_error Tests +# =========================================================================== + + +class TestClassifyLlmError: + def test_rate_limited(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("429 Too Many Requests: rate limit exceeded") + env = classifier.classify_llm_error(exc, provider="openai", model="gpt-4") + assert env.source == FailureSource.LLM + assert env.category == "rate_limited" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_transient(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("502 Bad Gateway: temporary server error") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "transient" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_context_overflow(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("422 maximum context length exceeded") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "context_overflow" + assert env.recoverability == Recoverability.AUTO_RECOVER + + def test_auth_error(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("unauthorized: invalid api_key") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "auth_error" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_auth_permanent(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("401 Unauthorized: invalid credentials") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "auth_permanent" + assert env.recoverability == Recoverability.NON_RECOVERABLE + + def test_billing(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("insufficient_quota: payment required") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "billing" + assert env.recoverability == Recoverability.USER_FIXABLE + + def test_content_blocked(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("400 content_policy violation: blocked") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "content_blocked" + assert env.recoverability == Recoverability.NON_RECOVERABLE + + def test_model_not_found(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("404 model 'gpt-5' does not exist") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "model_not_found" + assert env.recoverability == Recoverability.USER_FIXABLE + + def test_overloaded(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("503 server overloaded") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "overloaded" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_payload_too_large(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("413 Request Entity Too Large") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "payload_too_large" + assert env.recoverability == Recoverability.AUTO_RECOVER + + def test_format_error(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("422 invalid format json parse error") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category in ("format_error", "context_overflow") + # Both are AUTO_RECOVER + + def test_image_too_large(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("400 image size too large") + env = classifier.classify_llm_error(exc) + assert env.source == FailureSource.LLM + assert env.category == "image_too_large" + assert env.recoverability == Recoverability.AUTO_RECOVER + + def test_envelope_has_provider_hint(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("429 rate limit") + env = classifier.classify_llm_error(exc) + assert env.provider_hint is not None + assert env.provider_hint.hint_text != "" + + def test_envelope_has_context_with_provider(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("timeout") + env = classifier.classify_llm_error(exc, provider="dashscope", model="qwen-max") + assert env.context.arguments_dict.get("provider") == "dashscope" + assert env.context.arguments_dict.get("model") == "qwen-max" + + +# =========================================================================== +# classify_tool_result Tests — Non-failures +# =========================================================================== + + +class TestClassifyToolResultNonFailures: + def test_ok_true_returns_none(self, classifier: UnifiedErrorClassifier) -> None: + result = {"ok": True, "output": "success"} + assert classifier.classify_tool_result(result) is None + + def test_ok_default_returns_none(self, classifier: UnifiedErrorClassifier) -> None: + result = {"output": "success"} + assert classifier.classify_tool_result(result) is None + + def test_duplicate_suppressed_returns_none(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "duplicate_suppressed": True, + "error": "duplicate execution", + } + assert classifier.classify_tool_result(result) is None + + def test_counts_as_failure_false_returns_none(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "counts_as_failure": False, + "error": "non-failure signal", + } + assert classifier.classify_tool_result(result) is None + + +# =========================================================================== +# classify_tool_result Tests — Failures +# =========================================================================== + + +class TestClassifyToolResultFailures: + def test_permission_failure_by_class(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "failure_class": "authorization", + "failure_code": "missing_scope", + "error": "Missing read scope for resource", + } + env = classifier.classify_tool_result(result, tool_name="feishu_send") + assert env is not None + assert env.source == FailureSource.TOOL + assert env.category == "tool_permission" + assert env.recoverability == Recoverability.NON_RECOVERABLE + + def test_permission_failure_by_code(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "failure_class": "other", + "failure_code": "access_denied", + "error": "Access denied to endpoint", + } + env = classifier.classify_tool_result(result, tool_name="platform_action") + assert env is not None + assert env.category == "tool_permission" + assert env.recoverability == Recoverability.NON_RECOVERABLE + + def test_scope_denied_failure(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "failure_class": "scope_denied", + "error": "Scope not granted", + } + env = classifier.classify_tool_result(result) + assert env is not None + assert env.category == "tool_permission" + + def test_unknown_tool_retryable(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "error_type": "unknown_tool", + "error": "Tool 'web_search' not found", + "retryable": True, + } + env = classifier.classify_tool_result(result, tool_name="web_search") + assert env is not None + assert env.source == FailureSource.TOOL + assert env.category == "tool_unknown" + assert env.recoverability == Recoverability.AUTO_RECOVER + + def test_timeout_read_only(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "error": "Execution timed out after 30s", + } + env = classifier.classify_tool_result( + result, tool_name="web_search", execution_policy="read_only" + ) + assert env is not None + assert env.category == "tool_timeout" + assert env.recoverability == Recoverability.AUTO_RETRY + assert env.side_effect_state == SideEffectState.NONE + + def test_timeout_external_side_effect(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "error": "Request timeout waiting for response", + } + env = classifier.classify_tool_result( + result, tool_name="gateway_send", execution_policy="external_side_effect" + ) + assert env is not None + assert env.category == "tool_timeout" + assert env.recoverability == Recoverability.USER_FIXABLE + assert env.side_effect_state == SideEffectState.UNKNOWN + + def test_generic_failure_retryable(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "error": "Connection refused to service", + "retryable": True, + } + env = classifier.classify_tool_result(result, tool_name="shell_run") + assert env is not None + assert env.category == "tool_failure" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_generic_failure_not_retryable(self, classifier: UnifiedErrorClassifier) -> None: + result = { + "ok": False, + "error": "Invalid arguments", + "retryable": False, + } + env = classifier.classify_tool_result(result, tool_name="file_write") + assert env is not None + assert env.category == "tool_failure" + assert env.recoverability == Recoverability.USER_FIXABLE + + def test_execution_policy_read_only_side_effect(self, classifier: UnifiedErrorClassifier) -> None: + result = {"ok": False, "error": "failed"} + env = classifier.classify_tool_result(result, execution_policy="read_only") + assert env is not None + assert env.side_effect_state == SideEffectState.NONE + + def test_execution_policy_mutating_idempotent(self, classifier: UnifiedErrorClassifier) -> None: + result = {"ok": False, "error": "failed"} + env = classifier.classify_tool_result(result, execution_policy="mutating_idempotent") + assert env is not None + assert env.side_effect_state == SideEffectState.PARTIAL + + def test_execution_policy_external(self, classifier: UnifiedErrorClassifier) -> None: + result = {"ok": False, "error": "failed"} + env = classifier.classify_tool_result(result, execution_policy="external_side_effect") + assert env is not None + assert env.side_effect_state == SideEffectState.UNKNOWN + + +# =========================================================================== +# classify_system_error Tests +# =========================================================================== + + +class TestClassifySystemError: + def test_timeout_error(self, classifier: UnifiedErrorClassifier) -> None: + exc = TimeoutError("Connection timed out") + env = classifier.classify_system_error(exc) + assert env.source == FailureSource.SYSTEM + assert env.category == "system_timeout" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_memory_error(self, classifier: UnifiedErrorClassifier) -> None: + exc = MemoryError("Cannot allocate memory") + env = classifier.classify_system_error(exc) + assert env.source == FailureSource.SYSTEM + assert env.category == "system_resource" + assert env.recoverability == Recoverability.NON_RECOVERABLE + + def test_os_error(self, classifier: UnifiedErrorClassifier) -> None: + exc = OSError(28, "No space left on device") + env = classifier.classify_system_error(exc) + assert env.source == FailureSource.SYSTEM + assert env.category == "system_io" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_connection_error_from_message(self, classifier: UnifiedErrorClassifier) -> None: + exc = Exception("Connection refused to localhost:8080") + env = classifier.classify_system_error(exc) + assert env.source == FailureSource.SYSTEM + assert env.category == "system_network" + assert env.recoverability == Recoverability.AUTO_RETRY + + def test_generic_exception(self, classifier: UnifiedErrorClassifier) -> None: + exc = RuntimeError("Something unexpected happened") + env = classifier.classify_system_error(exc) + assert env.source == FailureSource.SYSTEM + assert env.category == "system_unknown" + assert env.recoverability == Recoverability.USER_FIXABLE + + def test_envelope_structure(self, classifier: UnifiedErrorClassifier) -> None: + exc = TimeoutError("test timeout") + env = classifier.classify_system_error(exc) + assert isinstance(env, FailureEnvelope) + assert len(env.envelope_id) == 32 + assert env.timestamp > 0 + assert env.message == "test timeout"