diff --git a/guides/ai-agents/deep-research.mdx b/guides/ai-agents/deep-research.mdx index d0f077eb..facda8d2 100644 --- a/guides/ai-agents/deep-research.mdx +++ b/guides/ai-agents/deep-research.mdx @@ -10,12 +10,12 @@ description: "Run a durable, multi-step AI investigation and get an evidence-bac Deep research is a long-running mode for Lightdash AI agents. It is designed for questions that need several queries, competing explanations, and a reusable report rather than one immediate answer. -The agent investigates in the background using its Lightdash context and the sources you approve. It returns a structured Markdown report with confidence levels, supporting charts, citations for external sources, and explicit caveats. +The agent investigates in the background using its Lightdash context and configured sources. It returns a structured Markdown report with confidence levels, supporting charts, citations for external sources, and explicit caveats. Deep Research prompt interface showing the question, research depth options, and selected MCP sources @@ -23,7 +23,7 @@ The agent investigates in the background using its Lightdash context and the sou - Choose this mode for multi-step investigations that need cross-checking, several data cuts, or a report you can share and revisit. + Choose this mode for multi-step investigations that need cross-checking, several data cuts, or a report you can save and revisit. Stay in the default mode for a quick lookup, one chart, or an interactive conversation where you want to steer each follow-up. @@ -40,32 +40,27 @@ Questions that work well include: ```mermaid flowchart LR - A["Question, depth, and sources"] --> B["Durable background run"] - B --> C["Discover relevant context"] - C --> D["Query data and test explanations"] - D --> C - D --> E["Validate findings"] + A["Question and agent configuration"] --> B["Plan competing hypotheses"] + B --> C["Investigate hypotheses in parallel"] + C --> D["Compare supporting and falsifying evidence"] + D --> E["Independent judge synthesizes the report"] E --> F["Report and chart snapshots"] ``` -Deep research uses the selected AI agent's configuration, including its instructions, semantic-layer access, knowledge documents, project and repository context, and enabled tools. It records progress as it works and saves the final report in the originating thread. +Deep research uses the selected AI agent's configuration, including its instructions, semantic-layer access, knowledge documents, project and repository context, and enabled tools. It automatically inherits the organization's research limits and every MCP server attached to the agent; there is no per-run depth or source selection. + +For each run, a planner creates distinct, falsifiable hypotheses. Separate investigators test them in parallel, looking for both supporting and contradicting evidence. An independent judge then compares their findings and produces the final report. This structure helps the agent distinguish correlation from causation and makes inconclusive results explicit instead of forcing a single explanation. Because the run executes on the server, you can close the tab or leave the thread. Reopening the thread restores the run card and its latest state. ## Start a run - - Start a new thread or open an existing thread that you own. Deep research is unavailable in read-only threads, such as another user's thread or a thread started in Slack. - - - Click **Deep research (Beta)** in the composer. The research settings appear above the message input. + + Use the Ask AI composer on the homepage, start a new agent thread, or open an existing thread that you own. Deep research is unavailable in read-only threads, such as another user's thread or a thread started in Slack. - - Select **Low**, **Medium**, **High**, or **Extra High**. Medium is the default. - - - Choose which MCP servers attached to the agent can participate in this run. All available attached servers are selected by default. + + Select **Deep research** in a new conversation, or select the telescope icon in an existing conversation. The control changes color when the mode is active. Include the decision or question, relevant time period, important segments, and any definitions or constraints the agent should preserve. @@ -75,26 +70,18 @@ Because the run executes on the server, you can close the tab or leave the threa -## Research depth +## Organization-wide limits -Depth controls the run's resource budget, including model tokens, tool calls, warehouse queries, and result rows. It is not a fixed duration: warehouse latency does not use up the reasoning budget. +Organization admins set the safety limits inherited by every deep research run. Go to **Organization settings** → **Ask AI** → **Deep research** to configure: - - - Up to **10 warehouse queries**. Best for a focused check of the strongest available evidence. - - - Up to **25 warehouse queries**. Best for a balanced investigation with validation and alternatives. - - - Up to **50 warehouse queries**. Best for a broad investigation with more competing explanations. - - - Up to **100 warehouse queries**. Best for the widest evidence review on high-stakes questions. - - +- **Maximum tokens** — total model tokens across planning, parallel investigations, and judging +- **Maximum tool calls** — total tool calls across the run +- **Maximum warehouse queries** — total semantic-layer and SQL queries across the run +- **Maximum hypotheses** — number of competing explanations the planner creates and investigators test + +Each limit must be a positive whole number. When a run reaches a limit after producing usable findings, Lightdash preserves the best available report and marks the run as partially completed. -Start with the smallest depth likely to answer your question. A larger depth gives the agent more opportunities to explore and validate, but it does not guarantee a better answer when the semantic layer or source data is incomplete. +By default, a run can use up to 10 million model tokens, 1,000 tool calls, 100 warehouse queries, and 5 hypotheses. Organization admins can change these values to match their governance and cost requirements. ## Sources and permissions @@ -103,24 +90,24 @@ Deep research can use: - **Agent context and project data** — the semantic layer, saved Lightdash content, knowledge documents, and other context configured on the selected agent, subject to its [data access settings](/guides/ai-agents/data-access). - **Warehouse queries** — semantic queries and, when the agent and user are allowed to use it, SQL. - **Repository context** — project context and source-code tools configured on the agent. -- **MCP servers** — only the [agent-attached servers](/guides/ai-agents/mcp-servers) selected in the research settings. +- **MCP servers** — every [server attached to the agent](/guides/ai-agents/mcp-servers) and its enabled tools. - Deep research runs without pausing for approval on each warehouse query or MCP tool call. An enabled MCP server can expose write actions, and those actions can run unattended. Select only servers whose enabled tools you want the agent to use for this investigation. + Deep research runs without pausing for approval on each warehouse query or MCP tool call. An attached MCP server can expose write actions, and its enabled actions can run unattended. Review the agent's attached servers and enabled tools before starting a run. -Deep research does not grant new permissions. The run starts with the creator's access and the selected agent's configuration, and Lightdash revalidates that access while the investigation runs. Revoking access can stop an active run. +Deep research does not grant new permissions. The run starts with the creator's access and the selected agent's configuration, and Lightdash revalidates that access while the investigation runs. Revoking access or disconnecting a source can stop an active investigation or leave that source unavailable. ## Follow progress -The run card stays in the thread and shows the latest phase, elapsed time, warehouse-query count, finding count, and recent activity. +The run card stays next to the question that started it and shows the latest phase, elapsed time, warehouse-query count, finding count, and recent activity. Lightdash accepted the run and is waiting for a background worker to start it. - The agent is gathering context, testing explanations, validating findings, or writing the report. Select **View activity** to inspect recent progress. + The agent is planning hypotheses, investigating them in parallel, or synthesizing the report. Select **View activity** to inspect recent progress. The full report is ready and saved in the thread. @@ -138,6 +125,8 @@ The run card stays in the thread and shows the latest phase, elapsed time, wareh Select **Stop research** while a run is queued or running to request cancellation. Cancellation is asynchronous, so a running tool call may reach its next safe checkpoint before the status changes. +Only one deep research run can be active in a thread at a time. While it is active, the deep research control is disabled, but you can continue sending regular chat messages in the same thread. The control becomes available again when the run completes, partially completes, fails, or is cancelled. + ## Read the report Select **Open full report** from a completed or partially completed run card. A report contains: @@ -165,6 +154,12 @@ Warehouse-backed charts open on a snapshot of the data the agent used when it wr Select **Live data** on a warehouse-backed chart to rerun its stored query and compare the latest result with the snapshot. Charts computed by the agent from derived or external data have no single warehouse query and cannot be refreshed. +### Report retention + +Deep research report content and chart snapshots expire **30 days after the run completes**. The run's question, status, and completion date remain in the thread. After expiry, select **Run again** to start a new investigation from the original question using the agent's current configuration and the organization's current limits. + +The regular chat agent can use the status and report from deep research runs in the same conversation when answering follow-up questions. Ask it to clarify a finding, compare evidence, or explain a limitation without pasting the report back into the chat. + ## Get better reports - State the decision you are trying to make, not only the metric you want to inspect. @@ -172,7 +167,7 @@ Select **Live data** on a warehouse-backed chart to rerun its stored query and c - Include the time range and comparison period. - Name segments the agent must test, such as channel, region, plan, or product category. - Ask it to test alternatives or contradictions instead of assuming one cause. -- Treat a partially completed report as a starting point and rerun at a larger depth only when the missing evidence matters. +- Treat a partially completed report as a starting point. Resolve unavailable sources or tighten the question before running it again; ask an organization admin to review the limits if runs repeatedly exhaust them. ## Access requirements @@ -190,14 +185,18 @@ No. The run continues on the server and its state is saved in the thread. Yes. You can start a run from an existing thread you own as long as the thread is not read-only and the agent's model is still available. +**Can I start another run while research is still active?** + +Not in the same thread. You can keep chatting normally, or start deep research in another eligible thread. The control returns when the current run reaches a terminal state. + **Why did my run partially complete?** The agent reached a resource budget or encountered a recoverable failure after producing a valid draft. Lightdash preserves the supported findings rather than discarding the report. **Does deep research only read data?** -No. It uses the selected agent's configured tools. Warehouse queries run without individual approval, and selected MCP servers may include write actions. Review the source selection before starting. +No. It uses the selected agent's configured tools. Warehouse queries run without individual approval, and attached MCP servers may include write actions. Review the agent's configuration before starting. **Can I continue chatting after the report is ready?** -Yes. The report remains attached to the thread, and you can continue the conversation with the same agent. +Yes. The report remains available in the thread for 30 days, and the same agent can use it as context for follow-up questions. diff --git a/images/guides/ai-agents/deep-research/deep-research-settings.png b/images/guides/ai-agents/deep-research/deep-research-settings.png index 42ab5829..bec120ed 100644 Binary files a/images/guides/ai-agents/deep-research/deep-research-settings.png and b/images/guides/ai-agents/deep-research/deep-research-settings.png differ