Reviewer 2.2: give the LLM ensemble a measured role, and disclose its forward bias

[freiburgermsu]

Answers reviewer 2's comment 2. Text only — M06 (Methods) and S04 (Supplement).

"The authors state in the Ensemble LLMs predictions section that a reaction may be driven by a pathway in a cell, which may be different from their grading of thermodynamic evidence. They then introduce LLMs and make predictions. Except the numbers in Figure 3, it is not apparent how LLMs play roles in reconstruction and other analyses."

The reviewer found a real gap

The section argued against itself. Its motivation opened on pathway-level driving force —

"every reaction is a member of a pathway within a cell, and the overarching drive of the pathway … means a reaction may be driven in a direction on a level that supersedes our evaluation"

— cited three papers on exactly that, and then introduced an ensemble that sees "its name and stoichiometry alone" and therefore has no pathway context whatsoever. It then withdrew twice: the calls "do not [enter] our grading of the thermodynamic evidence" (M06), and their confidence "does not appear to distinguish correct calls … and is not appropriate to use as a downstream filter" (S04).

So the reader is handed a method motivated by something it does not do, and disclaimed for everything else. That is what the reviewer is reacting to, and the fix is to state a role the data supports.

The role: coverage

reactions carrying an LLM call	45,644 (81.5%)
reactions no thermodynamic source directs	25,855
…of which the ensemble does direct	22,902 (88.6%)

Thermodynamic assignment is bounded by structural completeness; the ensemble is not. It roughly doubles the fraction of the database carrying a direction. That is a real, quantified contribution and it is now what the section claims.

The disclosure: it is not reading thermodynamics

Checking the ensemble against measurement rather than against another prediction changes the picture materially. Biochemistry/Thermodynamics/SourceGrading/opentecr_comparison.csv ships a measured opentecr_dG_kJ for 1,365 reactions, so this is directly testable:

source	margin	n (both commit)	agree	rate	chance	κ
LLM ensemble	sign only	361	246	68.1%	52.8%	0.33
LLM ensemble	|ΔG| > 5 kJ	264	200	75.8%	53.7%	0.48
LLM ensemble	|ΔG| > 11.7 kJ	158	115	72.8%	48.0%	0.48
eQuilibrator	|ΔG| > 11.7 kJ	82	81	98.8%	66.4%	0.96

The 94.7% agreement with eQuilibrator across the whole database (8,085 co-commitments) reads far better than this — but its chance baseline is 88.3%, κ = 0.55. The headline agreement is mostly a shared prior.

The mechanism is unusually clean. 85.3% of the ensemble's calls are the direction the equation is written in, and the errors are entirely one-sided:

	LLM says >	LLM says <
measured >	72	0
measured <	43	43

Perfect when the answer is "forward", a coin flip when it is not. The ensemble has learned that reaction equations are conventionally written in the physiological direction. That is a genuine and useful prior — it is why the coverage is worth having — but it is not thermodynamics.

This is the main reason to state it ourselves. It is recoverable from shipped data in an afternoon, so it is much better disclosed than discovered, and it independently justifies the decision the paper had already made to keep these calls out of the evidence grading. S04 also now notes the practical consequence: a reverse call is the model contradicting its own prior, and is the more informative of the two — 2,350 reactions a curator could review in a tractable pass.

The three orphaned citations

mavrovouniotis1993, xu2008 and noor2014 supported the pathway-driving-force motivation being removed, and each is cited nowhere else in the manuscript (verified). Rather than drop them, they are kept on a single sentence that makes the point honestly — as a limitation of both approaches:

Neither approach models the pathway context that can drive a reaction against its own standard-state energy as metabolite concentrations shift downstream; that remains a property of a model rather than of a reference database.

This is true, keeps three relevant references, and closes the loop with the transport limitation in reviewer 1's comment 2 (PR #297).

Split of detail

Main text carries the role and the caveat in four sentences; the numbers, the confusion matrix and the κ discussion go to S04, which has no page constraint.

Verification

• latexmk -pdf clean for both main.tex and supplementary.tex; zero undefined references or citations.
• Numbers reproduced by Papers/NAR_Update_2026/analysis/review_transport_and_llm.py (added in the figures PR of this series), on the manuscript's own population — all records, obsolete included.

Note on page count

Body text ends on page 6; one reference entry spills to page 7, where the baseline fit in six. PR #297 in this series is in the same position. The combined count needs re-checking once both merge, along with \lastpage{6} — flagged rather than tuned per-branch, since neither branch can see the other's contribution.

🤖 Generated with Claude Code

---

Review pass (2026-09-23)

Three defects found and fixed. (1) The main text and supplement disagreed on the ensemble's accuracy — M06 said 76%, S04 said 72.8% — because they used different margins, and S04's 11.7 kJ was never justified. Both now use τ = 2 kcal/mol, the tolerance S03 already defines, with a sensitivity statement: from a bare sign test up to RT·ln 1000 the ensemble ranges 65–77% (κ 0.33–0.50) and eQuilibrator stays above 96%; errors are one-sided at every cut. At τ: 165/215 (76.7%, κ 0.50) vs 54/55 (98.2%, κ 0.92); correct on 119 of 120 measured forward, 46 of 95 measured reverse. (2) 25,855 undirected was all-records → 23,729 live; the 22,902 is unchanged because the ensemble only ran on non-obsolete reactions. (3) eQuilibrator's 81/82 counted obsolete duplicate anchors → 54/55 at τ.

Re-verified unchanged: 94.7% agreement / 88.3% chance / κ 0.55, the 85.3 / 5.1 / 7.8 call split.