The header comment of ipsae.py says "version 3, April 6, 2025" up to 3480750 and "version 4,
January 3, 2026" from a6b0f8c. That helps, but the comment does not pin the code: "version 3" covers
five commits (441b2a4 to 3480750, including f892de5's n0res fix), and "version 4" is on both
a6b0f8c and 6174cf9, of which only the second changes d0. There are no git tags, and the output
files (*_10_10.txt, *_byres.txt) do not record the version. So a published score can only be
traced to the code if the author noted the commit.
I measured how much this matters on the AlphaFold 3 models of Overath et al.'s binder
meta-analysis (bioRxiv 10.1101/2025.08.14.670059, Zenodo 10.5281/zenodo.15722219). I ran
3480750 (2026-01-02) and 6174cf9 (current) at 10/10 on all 3 532 designs with a single-chain
target:
|
|
| designs whose ipSAE_min changed |
976 (27.6 %) |
| direction |
all lowered, largest change −0.0083 (mean −0.0011) |
| Spearman ρ, old vs new |
0.99996 |
| designs crossing the paper's 0.61 threshold |
0 (509 kept, 203 bound, under both) |
So the change is small and does not reorder anything that matters for ranking. That is good news.
It does mean reproductions disagree in the third decimal. The dataset matches 3480750 within its
3-decimal rounding on 99.3 % of designs, and the current version on 69.6 %. A tag
would have made that a one-line explanation.
Requests, in order of usefulness:
- Tag releases, and bump the header version whenever the numbers can change. Tagging 6174cf9
(current) and 3480750 (last before the d0 change) would cover what is in use now.
- Print the version on the first line of the output files.
A related detail, already noted in the calc_d0_array docstring of #31: at n0res = 27 the scalar
calc_d0 (used for d0chn/d0dom) returns 1.0, while calc_d0_array (used for ipSAE) returns
1.0389. Is 1.0 the intended value for both? #31 keeps the difference on purpose, for
byte-identical output, so a single answer here would help reimplementations.
Script to reproduce (clones this repo, runs both revisions per design in isolated directories):
https://github.com/OtoYuki/proteus/blob/main/validate/binders/ipsae_versions.sh (results: https://github.com/OtoYuki/proteus/blob/main/validate/binders/ipsae_versions.md)
The header comment of
ipsae.pysays "version 3, April 6, 2025" up to 3480750 and "version 4,January 3, 2026" from a6b0f8c. That helps, but the comment does not pin the code: "version 3" covers
five commits (441b2a4 to 3480750, including f892de5's n0res fix), and "version 4" is on both
a6b0f8c and 6174cf9, of which only the second changes d0. There are no git tags, and the output
files (
*_10_10.txt,*_byres.txt) do not record the version. So a published score can only betraced to the code if the author noted the commit.
I measured how much this matters on the AlphaFold 3 models of Overath et al.'s binder
meta-analysis (bioRxiv 10.1101/2025.08.14.670059, Zenodo 10.5281/zenodo.15722219). I ran
3480750 (2026-01-02) and 6174cf9 (current) at 10/10 on all 3 532 designs with a single-chain
target:
So the change is small and does not reorder anything that matters for ranking. That is good news.
It does mean reproductions disagree in the third decimal. The dataset matches 3480750 within its
3-decimal rounding on 99.3 % of designs, and the current version on 69.6 %. A tag
would have made that a one-line explanation.
Requests, in order of usefulness:
(current) and 3480750 (last before the d0 change) would cover what is in use now.
A related detail, already noted in the
calc_d0_arraydocstring of #31: at n0res = 27 the scalarcalc_d0(used for d0chn/d0dom) returns 1.0, whilecalc_d0_array(used for ipSAE) returns1.0389. Is 1.0 the intended value for both? #31 keeps the difference on purpose, for
byte-identical output, so a single answer here would help reimplementations.
Script to reproduce (clones this repo, runs both revisions per design in isolated directories):
https://github.com/OtoYuki/proteus/blob/main/validate/binders/ipsae_versions.sh (results: https://github.com/OtoYuki/proteus/blob/main/validate/binders/ipsae_versions.md)