fix(ci): cap every job at 30m and name hanging tests (refs #303)#304
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
CodeLens CI has not gone green in 60+ runs. Every run is marked "cancelled", but nobody cancels them: run 29502899082 ran from 13:36:17Z to 19:36:35Z — 6h00m18s, GitHub's hard job ceiling. The "Run test suite" step hangs on Linux while the same suite finishes in 160s locally, so each push burned 6h of Actions minutes across CI + Quality Gate (4 python matrix) + SARIF, and the merge gate has been fiction since at least July 14. No workflow had timeout-minutes at all. That is what made this durable: logs are unavailable while a job is in_progress, so a hang that only ends at the 6h kill can never be read. The problem locked away the evidence needed to diagnose it. Two changes, both diagnostics rather than a cure: - timeout-minutes: 30 on all 14 jobs across 8 workflows — a hang now dies in minutes with a readable log instead of burning 6h in silence. - pytest-timeout at --timeout=120 --timeout-method=thread on the test job — the suite is ~160s in full, so any single test past 120s is stuck, not slow. Verified locally: the timeout fires and prints the offending test's name, file and line. This does NOT fix the hang. #303 stays open until the root cause is known; the cap only makes it visible and cheap to investigate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #303 — deliberately does not close it. This makes the hang visible and cheap; it does not cure it.
Masalah
CodeLens CI: nol sukses dalam 60+ run, semuanyacancelled. Tak ada manusia yang membatalkan — run29502899082jalan13:36:17Z → 19:36:35Z= 6j 0m 18d, batas keras job GitHub. Step "Run test suite" hang (step 1-4 beres dalam 5 detik, runner nyata ter-assign), sementara suite yang sama selesai 160 detik lokal.Akibatnya: setiap push membakar 6 jam menit Actions × CI + Quality Gate (4 matriks) + SARIF, dan gate merge-nya fiktif sejak ~14 Juli — #292/#295/#296 semua mendarat tanpa CI hijau.
Tak ada satu pun
timeout-minutesdi seluruh.github/workflows/. Itu yang membuat masalah ini awet: log tak bisa dibaca saat jobin_progress(BlobNotFound), jadi hang yang baru mati di jam ke-6 tak pernah bisa dibaca. Masalahnya mengunci bukti untuk memecahkan dirinya sendiri.Yang berubah
timeout-minutes: 30di 14 job / 8 workflow — hang mati dalam menit dengan log terbaca, bukan membakar 6 jam dalam diam.pytest-timeoutdi job test:--timeout=120 --timeout-method=thread. Suite penuh ~160 detik, jadi test tunggal yang lewat 120 detik itu macet, bukan lambat.threadmencetak stack-nya.Verifikasi (dijalankan)
yaml.safe_load+ audit per-job, bukan dilihat sekilas).time.sleep(60)dengan--timeout=5→ mencetak nama test, file, dan baris. Jadi hang Linux nanti akan bernama di log.cd scripts(cwd CI beda dari lokal) diuji dan gugur: perintah CI persis itu selesai 160 detik di lokal, 19 gagal / 2541 lolos — sama dengan baseline. Hang-nya benar-benar spesifik Linux/CI dan tak bisa direpro di mesin dev.Kenapa #303 tetap terbuka
Root cause hang belum diketahui. Menutup #303 karena timeout sudah masuk = menukar gate palsu dengan gate palsu lain. Constraint itu sudah ditulis di issue-nya sebelum PR ini dibuat. Run PR inilah yang akan memberi log bernama pertama kali.
🤖 Generated with Claude Code