[dev] [Marfuen] mariano/fix-evidence-export-oom-streaming#2943
Open
github-actions[bot] wants to merge 2 commits into
Open
[dev] [Marfuen] mariano/fix-evidence-export-oom-streaming#2943github-actions[bot] wants to merge 2 commits into
github-actions[bot] wants to merge 2 commits into
Conversation
…ent OOM The previous OOM fix loaded automations one at a time but still accumulated all runs for a single automation in memory. For orgs with large cloud security check histories, a single automation's runs could exceed the 6GB heap limit. Now uses async generators to stream run batches (50 at a time) through PDF and JSON generation. Peak memory is bounded by one batch of runs + the jsPDF document, regardless of total automation size. - evidence-data-loader: add streamAutomationRuns async generator - evidence-pdf-generator: extract renderRunToPDF, add generateAutomationPDFFromStream - evidence-json-builder: add buildAutomationJsonStream using Readable.from() - evidence-export.service: wire streaming into ZIP export path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…task The auditor bulk evidence export previously ran in the API process, peaking at ~20% memory per request. Multiple concurrent exports could OOM the container. Now the heavy work (DB queries, PDF generation, ZIP creation) runs in a Trigger.dev background task with its own memory. The API endpoint triggers the task and returns a runId for progress tracking. - Add export-organization-evidence Trigger.dev task (S3 upload + presigned URL) - Change POST /v1/evidence-export/all to trigger background task - Frontend uses useRealtimeRun for progress + auto-download on completion - API process memory stays flat regardless of export size Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
1 issue found across 11 files
Confidence score: 4/5
- This PR is likely safe to merge, with a minor-to-moderate UX risk rather than a functional failure.
- In
apps/app/src/app/(app)/[orgId]/auditor/(overview)/components/ExportEvidenceButton.tsx, the sheet remains non-closable during export even though the UI text says it can be closed, which can confuse users and feel like the app is unresponsive. - The reported issue is mid-severity (5/10) but narrowly scoped to export-sheet interaction behavior, so overall merge risk appears limited.
- Pay close attention to
apps/app/src/app/(app)/[orgId]/auditor/(overview)/components/ExportEvidenceButton.tsx- allowonOpenChangeduring running exports to align behavior with the UI message.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/app/src/app/(app)/[orgId]/auditor/(overview)/components/ExportEvidenceButton.tsx">
<violation number="1" location="apps/app/src/app/(app)/[orgId]/auditor/(overview)/components/ExportEvidenceButton.tsx:90">
P2: The sheet cannot be closed while export is running, but the UI says it can be closed. Allow `onOpenChange` to update state during running exports.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
|
|
||
| <Sheet open={isOpen} onOpenChange={setIsOpen}> | ||
| <Sheet open={isOpen} onOpenChange={(open) => { | ||
| if (!isRunning) setIsOpen(open); |
Contributor
There was a problem hiding this comment.
P2: The sheet cannot be closed while export is running, but the UI says it can be closed. Allow onOpenChange to update state during running exports.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/app/src/app/(app)/[orgId]/auditor/(overview)/components/ExportEvidenceButton.tsx, line 90:
<comment>The sheet cannot be closed while export is running, but the UI says it can be closed. Allow `onOpenChange` to update state during running exports.</comment>
<file context>
@@ -14,82 +14,202 @@ import {
- <Sheet open={isOpen} onOpenChange={setIsOpen}>
+ <Sheet open={isOpen} onOpenChange={(open) => {
+ if (!isRunning) setIsOpen(open);
+ }}>
<SheetContent>
</file context>
Suggested change
| if (!isRunning) setIsOpen(open); | |
| setIsOpen(open); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated pull request to merge mariano/fix-evidence-export-oom-streaming into dev.
It was created by the [Auto Pull Request] action.
Summary by cubic
Fixes OOM during org-wide evidence export by streaming runs in small batches and moving ZIP generation to a
@trigger.dev/sdkbackground task. The API now returns arunIdandpublicAccessToken; the app shows live progress and auto-downloads when ready.New Features
streamAutomationRuns; build PDFs/JSON from streams viagenerateAutomationPDFFromStreamandbuildAutomationJsonStream.export-organization-evidenceTrigger.dev task: generates ZIP, uploads to S3, and returns a presigned download URL with progress metadata./v1/evidence-export/allto trigger the background job instead of streaming ZIP from the API.downloadAllEvidenceZipwithtriggerBulkEvidenceExport; use@trigger.dev/react-hooksuseRealtimeRunfor progress and auto-download.Migration
/v1/evidence-export/alland handle{ runId, publicAccessToken }, then track progress via Trigger.dev to retrieve the presigned URL.APP_AWS_BUCKET_NAME,APP_AWS_REGION,APP_AWS_ACCESS_KEY_ID,APP_AWS_SECRET_ACCESS_KEY(optionalAPP_AWS_ENDPOINT).Written for commit dd5a2f0. Summary will update on new commits.
Review in cubic