Add investigation report on TOAST bloat visibility gap - #6
Closed
shinyaaa wants to merge 2 commits into
Closed
Conversation
Investigate whether PostgreSQL core lacks a way to monitor TOAST table bloat from the parent table's perspective, as groundwork for a future pgsql-hackers proposal. Findings: - All premises verified against source and a running build of HEAD: pg_stat_all_tables does include TOAST rows, but pg_stat_user_tables excludes them by definition, and autovacuum processes TOAST tables independently of their parents. - The accident scenario reproduces: after a parent-only vacuum the parent shows n_dead_tup = 0 while its TOAST table holds 52,000 dead chunks and 108MB on disk, invisible in pg_stat_user_tables. - Survey of 9 monitoring tool lineages shows only Datadog links TOAST dead tuples to the parent table; the mainstream bloat query family (check_postgres / wiki) is structurally blind to TOAST. - Prior art: pg_stat_toast (2022) was withdrawn over new-counter overhead, which does not apply here since the recommended minimal patch is a SQL-only view change reusing existing per-relation stats, mirroring the pg_statio_all_tables toast_blks_* precedent. The report recommends proposing toast_relid plus a small set of toast_-prefixed columns in pg_stat_all_tables, with pointers to all evidence (file:line, commit hashes, thread URLs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfVEBxd7m1SHxv2aqREW2u
The investigation report and its verification script are tracked; the in-tree build logs, the throwaway install tree, and the test cluster are not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XfVEBxd7m1SHxv2aqREW2u
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Investigate whether PostgreSQL core lacks a way to monitor TOAST table
bloat from the parent table's perspective, as groundwork for a future
pgsql-hackers proposal.
Findings:
pg_stat_all_tables does include TOAST rows, but pg_stat_user_tables
excludes them by definition, and autovacuum processes TOAST tables
independently of their parents.
parent shows n_dead_tup = 0 while its TOAST table holds 52,000 dead
chunks and 108MB on disk, invisible in pg_stat_user_tables.
dead tuples to the parent table; the mainstream bloat query family
(check_postgres / wiki) is structurally blind to TOAST.
overhead, which does not apply here since the recommended minimal
patch is a SQL-only view change reusing existing per-relation stats,
mirroring the pg_statio_all_tables toast_blks_* precedent.
The report recommends proposing toast_relid plus a small set of
toast_-prefixed columns in pg_stat_all_tables, with pointers to all
evidence (file:line, commit hashes, thread URLs).
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01XfVEBxd7m1SHxv2aqREW2u