Feat/feasibility check

.gitignore

@@ -184,4 +184,12 @@ vector_db*
 /wandb
 model_output/
 my.py
-.DS_store
+.DS_store
+
+# Local advisor validation artifacts + Superpowers process docs (never shipped)
+res-adapt-ckeck/
+banking77_*_repredicted.json
+claude-issue-investigation.md
+review-of-review.md
+docs/superpowers/
+.python-version

docs/source/advisor.rst

@@ -0,0 +1,89 @@
+Compute feasibility advisor
+===========================
+
+.. note::
+
+   **Experimental.** The advisor's estimates are heuristic and calibrated against a limited hardware sample. Treat them as guidance, not guarantees, and read :ref:`advisor-accuracy` before relying on a number. The Python surface may change in a minor release.
+
+Optimizing a search space can take hours and needs more VRAM than a laptop GPU has. The advisor answers "will this fit, and how long will it take?" *before* anything is downloaded or trained.
+
+Command line
+------------
+
+Two subcommands. ``inspect`` prices a specific preset or config; ``recommend`` detects your hardware and picks the heaviest bundled preset that still fits.
+
+.. code-block:: bash
+
+   # What will transformers-light cost on this machine?
+   autointent-advisor inspect transformers-light
+
+   # ...against a real dataset rather than placeholder sizes
+   autointent-advisor inspect transformers-light --dataset banking77
+
+   # Which preset should I use?
+   autointent-advisor recommend --dataset banking77
+
+   # Machine-readable output
+   autointent-advisor inspect ./my-config.yaml --json
+
+Without ``--dataset``, the advisor uses placeholder dataset sizes (``--n-samples``, ``--n-classes``, ``--avg-tokens``, ``--task``), so it is useful before you have assembled any data. ``--budget-vram-gb`` overrides hardware detection, and ``recommend`` also accepts ``--budget-time-h``.
+
+Both subcommands exit non-zero when nothing is feasible, so they work as a CI gate.
+
+Reading a report
+----------------
+
+Each finding carries a severity:
+
+``ample``
+   Comfortably within budget.
+``tight``
+   Fits, but with little headroom — expect swapping or thermal throttling.
+``over``
+   Exceeds the budget. Any ``over`` finding makes the whole report infeasible.
+
+The drivers table lists the modules that dominate the cost, so it shows *what* to change. ``low confidence`` on a report means Hub metadata was unavailable or incomplete for at least one model, and conservative large-model defaults were substituted — the numbers are much rougher when you see it.
+
+Findings are not only about hardware. The advisor also prices your ``DataConfig``: it reports ``over`` when a class has too few samples for the stratified split to succeed, using the same minimum as :py:func:`~autointent.context.data_handler.check_split_readiness`, and when ``LogisticRegressionCV`` would not have ``cv`` samples per class *after* the train/validation split. Per-class counts are measured on the train split you supply, so the advisor discounts them by whatever ``validation_size``, ``n_folds``, and ``separation_ratio`` will take away.
+
+From Python
+-----------
+
+.. code-block:: python
+
+   from autointent import Dataset
+   from autointent.advisor import dataset_stats, detect_hardware, estimate, recommend
+
+   report = estimate("transformers-light")
+   print(report.is_feasible, report.resource.vram_gb)
+
+   result = recommend(stats=dataset_stats(Dataset.from_json(path)))
+   print(result.chosen)
+
+``reduce_to_fit`` goes further: it prunes the most expensive scoring module repeatedly until the search space fits, raising ``ReduceToFitError`` if nothing does.
+
+Inside ``Pipeline.fit``
+-----------------------
+
+``Pipeline.fit`` accepts a ``preflight`` gate. It defaults to ``"off"``, so the advisor never runs unless you ask — it makes network calls to the Hugging Face Hub for model metadata, which does not belong on every fit by default.
+
+.. code-block:: python
+
+   pipeline.fit(dataset, preflight="warn")     # log findings, always continue
+   pipeline.fit(dataset, preflight="strict")   # raise PreflightError if infeasible
+
+``"strict"`` raises :class:`autointent.advisor.PreflightError` before allocating any VRAM, which is the useful mode in CI.
+
+.. _advisor-accuracy:
+
+How accurate is it?
+-------------------
+
+Validated end to end on one machine class (RTX 3060 Laptop, 6 GB VRAM / 16 GB RAM), where all four fitted presets matched their predicted verdict: both ``over`` predictions did run out of memory, and both feasible predictions did fit. Known limits:
+
+- **Feasibility verdicts are the reliable part.** That is what the advisor was built and validated for.
+- **VRAM is close but not a guaranteed ceiling.** One preset used 1.22× its prediction. Leave headroom rather than trusting the figure exactly.
+- **Wall-time estimates are indicative only.** Measured error has run in both directions across formula revisions, once by more than an order of magnitude for cross-encoders. ``--budget-time-h`` inherits that uncertainty.
+- **CPU parallelism is modelled, not measured.** The CPU coefficients are calibrated single-threaded; core count is then applied as a capped Amdahl speedup (higher for CatBoost, which uses every core by default, than for scikit-learn's L-BFGS, which only threads inside BLAS), divided across concurrent ``hpo_config.n_jobs`` trials. It is a correction for the fact that core count used to change nothing at all, not a validated speedup curve.
+- **Preset ranking does not depend on time estimates.** ``recommend`` orders presets by a declared cost ranking, so unstable time figures cannot reorder its choice.
+- **Only one hardware class has been validated end to end.** Treat other machines as unverified.

docs/source/index.rst

@@ -68,6 +68,9 @@ Reference
 :doc:`🌐 Inference servers <server>`
    Deploy a trained pipeline behind HTTP (FastAPI) or MCP (FastMCP): installation extras, environment variables, and how to run each server.
 
+:doc:`🔍 Compute feasibility advisor <advisor>`
+   Estimate VRAM, RAM, disk, and wall-time for a search space before training. Run it from the CLI or gate ``Pipeline.fit`` on it.
+
 :doc:`🔧 API Reference <autoapi/autointent/index>`
    Complete technical documentation for all classes, methods, and functions. Essential reference for developers integrating AutoIntent into their applications.
 
@@ -84,4 +87,5 @@ Reference
    user_guides
    learn/index
    server
+   advisor
    autoapi/autointent/index

pyproject.toml

@@ -46,6 +46,7 @@ dependencies = [
     "aiofiles (>=24.1.0,<25.0.0)",
     "threadpoolctl (>=3.0.0,<4.0.0)",
     "packaging (>=23.2)",
+    "psutil (>=5.9.0,<8.0.0)",
 ]
 
 [project.optional-dependencies]
@@ -121,6 +122,7 @@ typing = [
     "joblib-stubs (>=1.4.2.5.20240918,<2.0.0)",
     "pandas-stubs (>= 2.2.3.250527, <3.0.0)",
     "types-aiofiles (>=24.1.0.20250606)",
+    "types-psutil>=7.2.2.20260518",
 ]
 docs = [
     "sphinx (>=8.1.3,<9.0.0)",
@@ -145,6 +147,7 @@ Documentation = "https://deeppavlov.github.io/AutoIntent/"
 [project.scripts]
 "basic-aug" = "autointent.generation.utterances._basic.cli:main"
 "evolution-aug" = "autointent.generation.utterances._evolution.cli:main"
+"autointent-advisor" = "autointent.advisor._cli:main"
 
 [build-system]
 requires = ["uv_build>=0.8.7,<0.9.0"]

src/autointent/_pipeline/__init__.py

@@ -1,3 +1,3 @@
-from ._pipeline import Pipeline
+from ._pipeline import Pipeline, PreflightMode
 
-__all__ = ["Pipeline"]
+__all__ = ["Pipeline", "PreflightMode"]

src/autointent/_pipeline/_pipeline.py

@@ -5,7 +5,7 @@
 import json
 import logging
 from pathlib import Path
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Any, Literal
 
 import numpy as np
 import yaml
@@ -34,10 +34,14 @@
 
 if TYPE_CHECKING:
     from autointent import Dataset
+    from autointent.advisor import Finding, PreflightReport
     from autointent.custom_types import ListOfGenericLabels, SearchSpacePreset, SearchSpaceValidationMode
     from autointent.modules.base import BaseDecision, BaseRegex, BaseScorer
 
 
+PreflightMode = Literal["off", "warn", "strict"]
+
+
 class Pipeline:
     """Pipeline optimizer class.
 
@@ -152,6 +156,45 @@ def from_optimization_config(cls, config: dict[str, Any] | Path | str | Optimiza
         pipeline.set_config(optimization_config.hpo_config)
         return pipeline
 
+    def _build_advisor_config(self) -> dict[str, Any]:
+        """Reconstruct an ``OptimizationConfig``-shaped dict for the advisor."""
+        search_space = [
+            {"node_type": opt.node_type, "search_space": opt.modules_search_spaces}
+            for opt in self.nodes.values()
+            if isinstance(opt, NodeOptimizer)
+        ]
+        return {
+            "search_space": search_space,
+            "data_config": self.data_config.model_dump(),
+            "logging_config": self.logging_config.model_dump(),
+            "embedder_config": self.embedder_config.model_dump(),
+            "cross_encoder_config": self.cross_encoder_config.model_dump(),
+            "transformer_config": self.transformer_config.model_dump(),
+            "hpo_config": self.hpo_config.model_dump(),
+        }
+
+    def _run_preflight(self, dataset: Dataset, *, refit_after: bool, mode: PreflightMode) -> PreflightReport:
+        """Run the advisor against this pipeline's effective config + dataset.
+
+        Logs each finding at INFO/WARNING/ERROR (by severity). When ``mode`` is
+        ``"strict"`` and any OVER finding is produced, raises ``PreflightError``.
+
+        Imported lazily: the advisor probes the HF Hub, so it must stay off the
+        ``import autointent`` path.
+        """
+        from autointent.advisor import PreflightError, Severity, dataset_stats, detect_hardware, run_preflight
+
+        config = self._build_advisor_config()
+        stats = dataset_stats(dataset)
+        hardware = detect_hardware()
+        report = run_preflight(config, stats, hardware, refit_after=refit_after)
+        _log_preflight_report(report, self._logger)
+        if mode == "strict":
+            over: list[Finding] = [f for f in report.findings if f.severity == Severity.OVER]
+            if over:
+                raise PreflightError(over)
+        return report
+
     def _fit(self, context: Context) -> None:
         """Optimize the pipeline.
 
@@ -193,22 +236,42 @@ def fit(
         dataset: Dataset,
         refit_after: bool = False,
         incompatible_search_space: SearchSpaceValidationMode = "filter",
+        preflight: PreflightMode = "off",
     ) -> Context:
         """Optimize the pipeline from dataset.
 
         Args:
             dataset: dataset for optimization.
             refit_after: whether to refit on whole data after optimization. Valid only for hold-out validaiton.
-            sampler: sampler type to use.
-            incompatible_search_space: wow to handle data-incompatible modules occurring in search space.
+            incompatible_search_space: how to handle data-incompatible modules occurring in search space.
+            preflight: **experimental** gate that runs
+                :func:`autointent.advisor.run_preflight` over the pipeline's
+                effective config + dataset before any heavy work.
+                ``"off"`` (default) skips it entirely. ``"warn"`` logs findings —
+                INFO for AMPLE, WARNING for TIGHT, ERROR for OVER — but never
+                raises; note it probes the HF Hub for model metadata, so it adds
+                network round-trips. ``"strict"`` additionally raises
+                :class:`autointent.advisor.PreflightError` when any finding has
+                severity OVER, so unfeasible runs abort before fit.
 
         Raises:
             RuntimeError: If pipeline is in inference mode.
+            PreflightError: If ``preflight="strict"`` and any OVER finding is produced.
         """
         if self._is_inference():
             msg = "Pipeline in inference mode cannot be fitted"
             raise RuntimeError(msg)
 
+        # Filter the search space first: ``validate_modules`` drops modules this
+        # dataset cannot use (e.g. ``mlknn`` on multiclass, ``dnnc`` and its ~6.4 GB
+        # reranker on multilabel), and preflight must price what will actually run
+        # rather than what was requested. It takes ``dataset`` only, so it does not
+        # depend on the ``Context`` built below.
+        self.validate_modules(dataset, mode=incompatible_search_space)
+
+        if preflight != "off":
+            self._run_preflight(dataset, refit_after=refit_after, mode=preflight)
+
         context = Context(self._seed)
         context.set_dataset(dataset, self.data_config)
         context.configure_logging(self.logging_config)
@@ -218,8 +281,6 @@ def fit(
         context.configure_hpo(self.hpo_config)
         context.configure_vector_index(self.vector_index_config)
 
-        self.validate_modules(dataset, mode=incompatible_search_space)
-
         test_utterances = context.data_handler.test_utterances()
         if test_utterances is None:
             self._logger.warning(
@@ -472,3 +533,25 @@ def make_report(logs: dict[str, Any], nodes: list[NodeType]) -> str:
     messages = [json.dumps(c, indent=4) for c in configs]
     msg = "\n".join(messages)
     return "resulting pipeline configuration is the following:\n" + msg
+
+
+def _log_preflight_report(report: PreflightReport, logger: logging.Logger) -> None:
+    """Log each preflight finding at the appropriate level."""
+    # Imported lazily for the same reason as in ``Pipeline._run_preflight``: the
+    # advisor probes the HF Hub, so it must stay off the ``import autointent``
+    # path. Do not hoist to module scope.
+    from autointent.advisor import Severity
+
+    level_for = {
+        Severity.AMPLE: logging.INFO,
+        Severity.TIGHT: logging.WARNING,
+        Severity.OVER: logging.ERROR,
+    }
+    header = (
+        f"Preflight ({report.preset_name or 'pipeline'}): verdict={'feasible' if report.is_feasible else 'INFEASIBLE'}"
+    )
+    logger.info(header)
+    for finding in report.findings:
+        logger.log(level_for[finding.severity], "[%s] %s", finding.phase, finding.message)
+    if report.low_confidence:
+        logger.info("Preflight: low-confidence (heuristic fallback in use)")

src/autointent/advisor/__init__.py

@@ -0,0 +1,37 @@
+"""Pre-flight compute feasibility advisor.
+
+**Experimental.** This subpackage estimates VRAM, RAM, disk, and wall-time for a
+search space before any training starts. Estimates are heuristic and calibrated
+against a limited hardware sample, so treat them as guidance rather than
+guarantees — see the accuracy caveats in the ``advisor`` page of the docs. The
+public surface may change in a minor release.
+
+Two ways in: the ``autointent-advisor`` console script, and the functions below.
+``Pipeline.fit(preflight=...)`` wires the same machinery into a fit, opt-in.
+"""
+
+from __future__ import annotations
+
+from ._errors import PreflightError
+from ._hardware import HardwareProfile, detect_hardware
+from ._report import DatasetStats, Finding, PreflightReport, RecommendationResult, ResourceEstimate, Severity
+from ._runner import run_preflight
+from ._workflows import ReduceToFitError, dataset_stats, estimate, recommend, reduce_to_fit
+
+__all__ = [
+    "DatasetStats",
+    "Finding",
+    "HardwareProfile",
+    "PreflightError",
+    "PreflightReport",
+    "RecommendationResult",
+    "ReduceToFitError",
+    "ResourceEstimate",
+    "Severity",
+    "dataset_stats",
+    "detect_hardware",
+    "estimate",
+    "recommend",
+    "reduce_to_fit",
+    "run_preflight",
+]