UN-3142 [FEAT] Trace an execution to its LLMWhisperer call - #2246
Open
Deepak-Kesavan wants to merge 1 commit into
Open
UN-3142 [FEAT] Trace an execution to its LLMWhisperer call#2246Deepak-Kesavan wants to merge 1 commit into
Deepak-Kesavan wants to merge 1 commit into
Conversation
Customers had no way to connect a workflow execution to the LLMWhisperer call behind it. Two gaps caused this: - The document is streamed to LLMWhisperer from the execution's internal copy (INFILE), and the client's `filename` param was never set, so LLMWhisperer's reports had nothing identifying the source document. - The whisper hash, which identifies the call, was only recorded when highlighting was enabled, and only ever reached service logs. Changes: - Forward the source filename to LLMWhisperer. `X2Text.process` injects it from `usage_kwargs`, which every caller already populates; callers may override with an explicit `file_name` kwarg. - Capture the whisper hash for every extraction that returns one, not just highlighted ones, and report it in the customer-facing execution log. - Preserve the rest of METADATA.json when recording the hash. The write was a blind overwrite that dropped source name, source hash and timing; now that it runs on every extraction rather than only highlighted ones, it merges instead. - Guard against a missing execution data dir, which would previously raise on `Path(None)`. The LLMWhisperer `tag` is deliberately unchanged — it carries customer workflow tags, and repurposing it for execution ids needs LLMWhisperer-side support for multiple tags.
|
Contributor
|
| Filename | Overview |
|---|---|
| unstract/sdk1/src/unstract/sdk1/x2txt.py | Injects the source filename from usage metadata while preserving an explicitly supplied filename. |
| unstract/sdk1/src/unstract/sdk1/adapters/x2text/llm_whisperer_v2/src/helper.py | Adds the filename parameter to LLMWhisperer V2 requests with an empty-string fallback. |
| workers/executor/executors/legacy_executor.py | Captures hashes across highlighted and non-highlighted legacy extraction paths and merges them into shared metadata. |
| workers/file_processing/structure_tool_task.py | Forwards the agentic source filename but discards the extraction metadata, leaving whisper-hash tracing incomplete for this path. |
| unstract/sdk1/tests/test_x2text_source_filename.py | Covers filename precedence and propagation through the V2 request stack. |
| workers/tests/test_legacy_executor_extract.py | Covers legacy executor hash persistence, logging, missing directories, and preservation of existing metadata. |
Sequence Diagram
sequenceDiagram
participant Workflow as Agentic workflow
participant Task as _run_agentic_extraction
participant X2T as X2Text / LLMWhisperer
participant Metadata as METADATA.json / execution log
Workflow->>Task: Run agentic extraction
Task->>X2T: "process(file_name=source_file_name)"
X2T-->>Task: TextExtractionResult(text, whisper_hash)
Task->>Task: Consume extracted_text only
Note over Task,Metadata: whisper_hash is not persisted or logged
Task-->>Workflow: Structured extraction result
Prompt To Fix All With AI
### Issue 1
workers/file_processing/structure_tool_task.py:725-730
**Agentic path drops whisper hash**
When an agentic studio tool extracts a document through `_run_agentic_extraction`, this direct `X2Text` call consumes only `extracted_text` and discards `extraction_metadata.whisper_hash`, causing the hash to be absent from `METADATA.json` and customer-facing execution logs.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "UN-3142 [FEAT] Trace an execution to its..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
725
to
730
| extraction_result = x2text.process( | ||
| input_file_path=input_file_path, | ||
| enable_highlight=enable_highlight, | ||
| file_name=source_file_name, | ||
| fs=fs, | ||
| ) |
Contributor
There was a problem hiding this comment.
When an agentic studio tool extracts a document through _run_agentic_extraction, this direct X2Text call consumes only extracted_text and discards extraction_metadata.whisper_hash, causing the hash to be absent from METADATA.json and customer-facing execution logs.
Knowledge Base Used: Workers (Celery) Service
Prompt To Fix With AI
This is a comment left during a code review.
Path: workers/file_processing/structure_tool_task.py
Line: 725-730
Comment:
**Agentic path drops whisper hash**
When an agentic studio tool extracts a document through `_run_agentic_extraction`, this direct `X2Text` call consumes only `extracted_text` and discards `extraction_metadata.whisper_hash`, causing the hash to be absent from `METADATA.json` and customer-facing execution logs.
**Knowledge Base Used:** [Workers (Celery) Service](https://app.greptile.com/zipstack/-/custom-context/knowledge-base/zipstack/unstract/-/docs/workers-celery.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Contributor
Unstract test resultsPer-group results
Critical paths
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes a workflow execution traceable to the LLMWhisperer call behind it.
METADATA.jsonwrite that this change would otherwise have made far more frequent.Why
Raised by a customer (via Abhishek Pathak) in UN-3142: there is no way to connect an execution to an LLMWhisperer call. Two concrete gaps caused this:
INFILE—execution_file_handler.py), and the client'sfilenameparam was never set. LLMWhisperer's reports therefore had nothing identifying the source document.enable_highlightwas true, and otherwise only reached service logs — which customers cannot see.How
Filename → LLMWhisperer
WhispererRequestParamsgains afilenamefield;get_whisperer_paramsemits it as the client'sfilenamequery param (supported byllmwhisperer-client≥ 2.6.2).X2Text.processinjects it fromusage_kwargs[file_name], which every caller already populates for usage auditing — so this works at all call sites without touching each one. Callers can override with an explicitfile_namekwarg; the agentic path instructure_tool_task.pydoes exactly that, since it does not buildusage_kwargs.Whisper hash → customer-facing log
LegacyExecutor._handle_extractnow reads the hash after both the highlight and non-highlight branches, records it whenever one is returned, and appends it to theText extraction completedlog streamed to the user.METADATA.json data loss (found during self-review)
_update_exec_metadatausedToolUtils.dump_json, a blind full overwrite, writing only{"whisper-hash": ...}.METADATA.jsonis shared —source_name,source_hash, tags and timing are written before extraction and tool results after — so that write destroyed them. Today it only fires when highlighting is on; capturing unconditionally would have made it fire on every tool-source extraction. It now merges into the existing file, matching what_write_tool_resultalready does.execution_data_dir, which would previously raise onPath(None).Deliberately not changed: the LLMWhisperer
tag. It currently carries the customer's workflow tags, and repurposing it to hold execution ids would regress anyone relying on tag-based reporting. Tagging the call with execution details (bullet 2 of the ticket) needs LLMWhisperer-side support for multiple tags or a dedicated metadata field — see "Related Issues".Can this PR break any existing features. If yes, please list possible items. If no, please explain why.
Low risk, but three behaviour changes worth naming:
filenameis now sent to LLMWhisperer. It is an optional param the client already supports and defaults to""when the name is unknown, so behaviour is unchanged when no name is available. It carries the customer's own document name to a service that is already receiving the document itself, so this exposes nothing new._update_exec_metadatamerges instead of overwriting. Strictly recovers data that was previously being dropped; nothing readsMETADATA.jsonexpecting it to contain only the whisper hash (destination_connector.pyreadswhisper-hashby key).The
tagpath is untouched, and a test pins that filename injection does not disturb it.Database Migrations
Env Config
Relevant Docs
filename,tagparams.Related Issues or PRs
This covers bullets 1 and 3 of the ticket (log the details in customer-facing logs; send the correct file name). Bullet 2 — tagging the whisper call with execution details — is deliberately deferred, as it cannot be done well without LLMWhisperer-side support for more than one tag. The whisper hash now exposed in the logs provides the correlation the customer asked for without requiring changes to both products.
Dependencies Versions
filename, already supported by the pinnedllmwhisperer-client>=2.6.2.Notes on Testing
Automated — full suites green locally:
workers: 1329 passed, 141 skippedunstract/sdk1: 527 passed (+7 new)New coverage:
unstract/sdk1/tests/test_x2text_source_filename.py(7 tests) — filename injected fromusage_kwargs; explicit kwarg wins; absent name does not raise; adapter forwards it intoWhispererRequestParams; emitted as thefilenamequery param; defaults to empty; tag behaviour unchanged.workers/tests/test_legacy_executor_extract.py(7 new) — hash captured with highlight disabled; extractors returning no hash write nothing; missingexecution_data_dirdoes not crash; hash present in the customer-facing log; completion log clean when there is no hash; pre-existing METADATA.json keys preserved; corrupt METADATA.json does not fail the extraction.Not yet verified live: that the deployed LLMWhisperer API surfaces the
filenamewe now send in its reports. The param is accepted by the pinned client and is optional, so the risk is that it is ignored, not that extraction breaks. Worth confirming against a dev-namespace run before this is treated as closing the customer's request.Screenshots
N/A — no UI surface; the change appears as an extra detail on the existing
Text extraction completedexecution log line.Checklist
I have read and understood the Contribution Guidelines.