Stop losing AI messages when a deploy interrupts them

[elias-ba]

Description

A deploy cuts off whatever is running. When that was an AI chat message, nothing said so: the message stayed :processing forever, the panel stayed locked for everyone in that session, and no error was raised. You only found it by looking at the row.

Oban announces a job being stopped and we were not listening. The handler for it had been written and never attached. Attaching it needed three other things to be true.

It gets its own handler id rather than joining the exception one. :telemetry drops a handler from every event it registered for the first time it raises, so sharing an id would let one bad :stop take Oban exception reporting down with it until the next restart.

The two [:oban, :circuit, _] events go with it. Oban has not emitted them since 2.6, and there was no clause for them, so one arriving would have raised and detached everything.

The lookups become get rather than get!. :stop fires for every successful job in every queue, which is the busiest path we have, and a message deleted while its job ran would raise there and trigger exactly the detach above. Both paths now have a test that deletes the message first, since returning at all is what proves the clause is there.

The drain window moves from two minutes to six. It was shorter than an AI job's own ceiling, so a deploy landing on a running job killed it after Oban had already stopped the producer that would have reported it. That is no telemetry at all, and the one case attaching :stop cannot rescue. Application start now warns if that inverts again, and the ceiling is available on its own as job_timeout/0 for the callers that have no job in hand.

Two log lines drop to warning: a timeout on the exception path, and a non-success stop. Both sat next to a Sentry capture deliberately set to warning, so the error level was raising a second and louder event for something the code had already judged not to be a fault.

The new test emits the event rather than calling the handler directly, the way the others do. That is why this was covered and still broken.

Part of #4260 (the other half is #5071).

Validation steps

1. Open a workflow, open the AI assistant, and send a message. While it is still thinking, stop the server (Ctrl+C twice) and start it again. Reload the page: the message should show as failed with a retry, instead of the panel sitting on "..." forever.

2. In IEx, check the drain window is longer than a run can take, which is what makes the above work rather than relying on the sweep:

    Application.get_env(:lightning, Oban) |> Keyword.get(:shutdown_grace_period)
    Lightning.AiAssistant.MessageProcessor.job_timeout()

   The first should be comfortably larger than the second.

3. Set APOLLO_REQUEST_TIMEOUT_MS high enough to invert those two and restart. There should be a warning at boot naming both.

Additional notes for the reviewer

The two [:oban, :circuit, _] events are removed rather than given a clause. Oban has not emitted them since 2.6, so nothing should arrive on them; say so if you disagree.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our Responsible AI Policy

Pre-submission checklist

• I have performed an AI review of my code (we recommend using /review with Claude Code)
• I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
• I have updated the changelog
• I have ticked a box in "AI usage" in this PR

[github-actions]

Security Review ✅

• S0 (project scoping): N/A — telemetry handlers look up ChatMessage by a message_id from trusted Oban.Job.args set at enqueue time, not from a spoofable HTTP/LiveView param, and no new project-scoped query surface is introduced.
• S1 (authorization): N/A — no new controllers, LiveView events, or CRUD actions; changes are limited to telemetry attachment, Oban shutdown_grace_period, and internal Logger-level tweaks.
• S2 (audit trail): N/A — the only mutation is an internal ChatMessage status transition to :error for interrupted AI messages, which is not a project/instance configuration resource in the audit scope.

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.6%. Comparing base (dbe62b1) to head (0797f22).

Files with missing lines	Patch %	Lines
lib/lightning/ai_assistant/message_processor.ex	87.5%	1 Missing ⚠️

Additional details and impacted files

@@                      Coverage Diff                       @@
##           timeout-stable-message-order   #5069     +/-   ##
==============================================================
- Coverage                          90.6%   90.6%   -0.0%     
==============================================================
  Files                               421     421             
  Lines                             20012   20014      +2     
==============================================================
- Hits                              18138   18134      -4     
- Misses                             1874    1880      +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.