feat(agentic): nest search_web verifier in Langfuse and price it

README.md

@@ -23,7 +23,7 @@ Also in this README: [Setup](#setup) · [Core concepts](#core-concepts) · [Repo
 - **Core library** — `aieng-forecasting` (`aieng.forecasting`): data services, cutoff enforcement, forecasting tasks, prediction payloads, backtesting, evaluation, and artifacts.
 - **Reusable methods** — `aieng.forecasting.methods`: `Predictor` implementations including naive baselines (continuous, binary, and categorical), Darts numerical predictors, LLM-process predictors (continuous, binary-probability, and categorical-probability), and ADK-based agentic infrastructure (`build_adk_agent`, `AdkTextRunner`, `AgentPredictor`).
 - **Reference implementations** — `implementations/<use-case>/`: notebooks, helper modules, task-specific configuration, and co-located YAML specs.
-- **Tracing** — Langfuse / OpenTelemetry bootstrap (`aieng.forecasting.langfuse_tracing`) for LiteLLM and Google ADK.
+- **Tracing** — Langfuse / OpenTelemetry bootstrap (`aieng.forecasting.langfuse_tracing`) for LiteLLM and Google ADK. Agent `search_web` calls nest inner search and leakage-verifier generations in the same trace.
 - **Data scripts** — `scripts/`: one fetch script per data source, plus `build_e2b_template.py` for the agentic code-execution sandbox.
 
 ## Two ways to use a forecaster

aieng-forecasting/README.md

@@ -8,7 +8,7 @@ This package provides the stable infrastructure used across all reference implem
 - Forecasting task and prediction payload models.
 - Backtesting, evaluation, scoring, and artifact helpers.
 - Reusable reference predictors under `aieng.forecasting.methods`.
-- Langfuse / OpenTelemetry tracing bootstrap in `aieng.forecasting.langfuse_tracing`.
+- Langfuse / OpenTelemetry tracing bootstrap in `aieng.forecasting.langfuse_tracing` (including nested `search_web` search/verifier generations).
 
 Current data adapters cover StatCan tables, FRED series, and daily yfinance
 market series.

aieng-forecasting/aieng/forecasting/langfuse_tracing.py

@@ -4,6 +4,10 @@
 ``llm`` or ``agentic`` extras and Langfuse credentials are set in the
 environment.
 
+Inner LiteLLM calls that ADK OpenInference does not see (``search_web``'s
+grounded search and leakage verifier) should wrap with
+:func:`langfuse_generation` so they nest under the active agent trace.
+
 Call :func:`print_langfuse_trace_url` after a ``predict()`` call to flush
 pending spans and print a clickable Langfuse UI link.
 """
@@ -12,11 +16,20 @@
 
 import logging
 import os
+from contextlib import contextmanager
+from typing import Any, Iterator
 
 
 logger = logging.getLogger(__name__)
 
 
+class _NoOpObservation:
+    """Stand-in when Langfuse is unavailable so callers can always ``.update()``."""
+
+    def update(self, **_kwargs: Any) -> None:
+        return None
+
+
 def _langfuse_credentials_present() -> bool:
     pub = os.environ.get("LANGFUSE_PUBLIC_KEY", "").strip()
     sec = os.environ.get("LANGFUSE_SECRET_KEY", "").strip()
@@ -88,6 +101,48 @@ def _instrument_google_adk(self) -> None:
 _bootstrap = _LangfuseTracingBootstrap()
 
 
+@contextmanager
+def langfuse_generation(
+    name: str,
+    *,
+    model: str | None = None,
+    input: Any = None,  # noqa: A002 — matches Langfuse observation field name
+    metadata: dict[str, Any] | None = None,
+) -> Iterator[Any]:
+    """Open a Langfuse generation nested under the current observation.
+
+    Used for inner LiteLLM calls that ADK OpenInference does not see (the
+    ``search_web`` googleSearch completion and the independent leakage
+    verifier). When an ADK tool span is already active, the new generation
+    becomes its child, so verifier traces show up inside the agent tree
+    rather than as a separate root.
+
+    No-op when credentials are absent or the SDK raises, so tool code can
+    wrap completions without a tracing extra.
+    """
+    if not _langfuse_credentials_present():
+        yield _NoOpObservation()
+        return
+    try:
+        from langfuse import get_client  # noqa: PLC0415
+
+        kwargs: dict[str, Any] = {"name": name, "as_type": "generation"}
+        if model is not None:
+            kwargs["model"] = model
+        if input is not None:
+            kwargs["input"] = input
+        if metadata:
+            kwargs["metadata"] = metadata
+        observation_cm = get_client().start_as_current_observation(**kwargs)
+    except Exception:
+        logger.debug("langfuse_generation(%s) failed; continuing without a span.", name, exc_info=True)
+        yield _NoOpObservation()
+        return
+
+    with observation_cm as generation:
+        yield generation
+
+
 def init_langfuse_tracing() -> None:
     """Wire LiteLLM and Google ADK to Langfuse.
 

aieng-forecasting/aieng/forecasting/methods/README.md

@@ -104,7 +104,7 @@ from aieng.forecasting.methods.agentic import (
 |---|---|---|
 | `agentic/adk_runner.py` | `AdkTextRunner` | Async text-in / text-out wrapper around ADK `InMemoryRunner`. Manages ADK sessions (fresh-per-message or sticky) and optionally traces each turn to Langfuse via `propagate_attributes`. |
 | `agentic/adk_runner.py` | `AdkTextRunnerConfig` | Pydantic configuration for `AdkTextRunner` (session mode, Langfuse fields). |
-| `agentic/agent_factory.py` | `build_adk_agent` | Generic ADK `LlmAgent` factory with optional code execution, context retrieval, skills, generation controls, and structured output schema. |
+| `agentic/agent_factory.py` | `build_adk_agent` | Generic ADK `LlmAgent` factory with optional code execution, context retrieval, skills, generation controls, and structured output schema. `search_web` runs a cutoff-aware googleSearch sub-call plus an independent leakage verifier on historical origins (skipped when `as_of` is today or later); both inner LLM calls emit nested Langfuse generations (`search_web.google_search`, `search_web.leakage_verifier`) under the agent trace. |
 | `agentic/agent_factory.py` | `AgentConfig` | Pydantic configuration for reusable ADK agents. `output_schema=None` supports interactive/free-form agents; a structured `AgentForecastOutput` schema supports Track 1 predictors. The `function_tools` field attaches conventional ADK tools (e.g. `ForecastTool`). Use-case-specific prompts and presets should live in `implementations/<use-case>/`. |
 | `agentic/forecast_tool.py` | `ForecastTool` | Conventional ADK `FunctionTool` that runs a pre-specified `Predictor` (AutoARIMA by default) on any registered series at a given cutoff/horizon, returning a structured JSON forecast. A controlled, reproducible alternative to open-ended code execution; series data never enters the LLM context. |
 | `agentic/outputs.py` | `AgentForecastOutput` | Abstract output adapter interface for converting structured agent JSON into evaluation `Prediction` objects. |