Skip to content

Add PositionScore command line tool for the mutational screening figure - #2

Open
vmesel wants to merge 9 commits into
masterfrom
feature/positionscore-cli
Open

Add PositionScore command line tool for the mutational screening figure#2
vmesel wants to merge 9 commits into
masterfrom
feature/positionscore-cli

Conversation

@vmesel

@vmesel vmesel commented Aug 12, 2026

Copy link
Copy Markdown
Member

What

The synthetic mutational screening (PositionScore) only existed inside src/training_notebooks/model_training_microexons.ipynb (cells 59 and 65). This turns it into a command line tool that takes a microexon coordinate and draws the Fig. 7A panel:

python src/positionscore.py --model src/saved_model.hdf5 \
  --genome src/data/hg38.fa --conservation src/data/hg38_cons.bw \
  --exon chrX:31126642:31126673:+ --gene DMD --panel-label A \
  --output DMD_positionscore.png --tsv DMD_positionscore.tsv

Every base of the two 100 nt flanks is replaced by each of the four nucleotides, the microexon is re-scored with the CNN, and the delta (mutant - wild type) is drawn as a 4 x 200 heatmap with the microexon diagram below it, keeping the settings of the published figure (RdBu_r, scale from -0.015 to +0.015, TGAC row order).

Closes the "Include the mutational screening in the command line tool" item of the README roadmap.

Changes

  • src/deepmex.py: the script body moves into main(), so Microexon can be imported. The CLI, its arguments and its output are unchanged. New predict_batch() scores many sequence variants in a single model call, and prediction() now delegates to it, so both paths handle strands identically. Unknown bases (N) no longer raise a KeyError, and the tool no longer crashes when stdout is redirected to a file, which is how the README documents it (> result.out).
  • src/positionscore.py: the screening, the figure, an optional TSV export of the per position / per base scores, and --demo to inspect the layout without the model or the reference files.
  • tests/test_positionscore.py: checks against a stand in model, including the batched screening against the position by position loop of the notebook. Runs without tensorflow and without the 8 GB of reference files (make test).
  • README.md / Makefile / requirements.txt: docs, make run-positionscore-example, make positionscore-demo, make test, and matplotlib.

Notes for review

  • Minus strand exons are drawn in transcript orientation (column 0 is always -100 relative to the exon). The flank handling matches what deepmex.py already does when scoring a minus strand microexon: flanks are swapped and sequences are read backwards, without complementing the bases. Worth a second look at some point, but this PR deliberately keeps the existing behaviour so the figure agrees with the score the tool already reports.
  • Only panel A of Fig. 7 is generated. Panel B (PSI distributions of the PTBP1 knock-down) and panel C (UCSC genome browser tracks) come from external data, not from this pipeline.
  • The 800 variants are scored in batches instead of one prediction per variant, which is what made the notebook version slow.
  • make download-data saves the bigwig as src/data/hg38.phastCons100way.bw while make run-example reads src/data/hg38_cons.bw; that mismatch predates this PR and is left untouched.

vmesel added 6 commits August 12, 2026 16:00
The synthetic mutational screening only existed inside the training notebook.
This adds it as a command line tool that takes a microexon coordinate and
draws the Fig. 7A heatmap: each base of the two 100 nt flanks is replaced by
each of the four nucleotides, the microexon is re-scored with the CNN, and the
delta (mutant - wild type) is plotted as a 4 x 200 heatmap with the microexon
diagram below it.

    python src/positionscore.py --model src/saved_model.hdf5 \
      --genome src/data/hg38.fa --conservation src/data/hg38_cons.bw \
      --exon chrX:31126642:31126673:+ --gene DMD --output DMD.png

- src/deepmex.py: move the script body into main(), so Microexon can be
  imported. The CLI, its arguments and its output stay the same. Adds
  predict_batch(), which scores many sequence variants in one model call and
  which prediction() now delegates to, so both paths handle strands the same
  way. Unknown bases (N) no longer raise a KeyError, and the tool no longer
  crashes when stdout is redirected to a file, as documented in the README.
- src/positionscore.py: the screening itself, the figure, an optional TSV
  export of the per position / per base scores, and --demo to check the layout
  without the model or the reference files.
- tests/test_positionscore.py: checks the wiring against a stand in model,
  including the batched screening against the position by position loop of the
  notebook. Runs without tensorflow and without the reference files.

Minus strand exons are drawn in transcript orientation, following the same
flank handling deepmex.py already applies when scoring them.
The pinned dependencies (TensorFlow 1.15, Keras 2.3, numpy 1.16) no longer
install, so the tool could not be run at all. Move the project to Python 3.13+
(pinned to 3.15) managed by uv, and turn the loose scripts into a package.

- src/deepmex/{core,cli,positionscore,gtf,conservation}.py, exposed as the
  deepmex and positionscore commands
- pyproject.toml + uv.lock, with optional extras: model (tensorflow, keras),
  genome (pyBigWig, pybedtools) and gtf (pandas, gtfparse, tqdm). TensorFlow
  has no CPython 3.15 wheels yet, so the inference environment is built on
  3.13 (make install-model)
- saved_model.hdf5 was written by Keras 2.3, whose optimizer config no longer
  deserializes on Keras 3; load it uncompiled, which scoring does not need
- drop the Python 2 leftovers: __future__ imports, .format(), iteritems(),
  indexable map(), pd.read_table, os.system('clear'), stty size
- Microexon is now a dataclass built from explicit ReferenceFiles instead of
  module level globals, and reading a flank raises on failure rather than
  returning None and blowing up later in a reshape
- gtfparse 2.x renamed parse_gtf_and_expand_attributes to read_gtf
- extract_conservation ran from hardcoded paths of another machine; it takes a
  Workspace directory now, and the _up/_down suffix is stripped with
  removesuffix instead of strip(), which corrupted keys ending in d, o, w or n
- --vmax defaulted to 0.015, the value of the published figure on the 0-1
  scale of the notebook, while the CLI reports scores from 0 to 100. Real
  deltas reach ~1.0, so every figure came out fully saturated; the default is
  1.5 and the demo envelope is scaled to match
Scoring a batch of microexons in one process piled up figures in the global
pyplot registry, which warns from the 20th figure on and keeps every one of
them alive. Use matplotlib.figure.Figure directly, which also stops the
module from switching the backend of whatever imports it.

Also ignore the generated figures and tables.
Comparing panel A of the published Fig. 7 with what the tool produced showed
the two disagreed. Extracting the panel from the TIF and correlating it with
the screening of the same microexon identifies the transform: the figure is
the reverse *complement*, not just the reverse.

- to_transcript_orientation now complements the bases as well as reversing
  the positions, so a row labelled A is the mutation to A of the transcript.
  Correlation with the published panel goes from r=0.13 to r=0.67; every other
  candidate transform scores below 0.21
- the DMD example passed strand + , but DMD is on the minus strand (16 RefSeq
  transcripts, all -). Fixed in the README and the Makefile
- .tif output is written with lossless LZW compression: a 300 dpi panel drops
  from 22 MB to 2.2 MB
Panel A alone is not the published figure. Panel B, the inclusion level of the
microexon under a knock-down, comes from vast-tools rather than from the model,
so it is read from an INCLUSION_LEVELS_FULL table.

- knockdown.py parses the table, rebuilds the Beta posterior of every replicate
  from the corrected read counts of the quality column, and draws the two plots
  of the panel: the posterior of each group, and P(|dPSI| > x) with the 95%
  MV[dPsi] marked
- positionscore.draw_position_scores now takes a vertical band of the figure,
  so panels stack; plot_figure composes A alone or A and B
- positionscore gains --knockdown/--event/--group/--knockdown-title

Panel C is a UCSC browser screenshot and stays out of scope.
Running the tool meant knowing the strand, having the 6 GB of reference files
and passing four options. This takes a bare coordinate instead:

    deepmex-figure chrX:31126642-31126673

- ucsc.py reads the flanks, the strand and the gene name from the genome
  browser API, so nothing has to be downloaded; --genome/--conservation still
  take local files
- impact.py ranks every flank base by the largest score change its three
  substitutions produce, and writes it as a table carrying the genomic
  position, the position relative to the exon and the reference base. For the
  DMD microexon the top positions are the thymines of the polypyrimidine
  tract, 14 to 22 nt upstream
- core.flank_intervals is now the single place the flank arithmetic lives, so
  the reference files, the API and the table cannot drift apart
- parse_coordinate accepts chrX:31126642-31126673 with or without the strand,
  as well as the older colon separated form

Reading the reference base back from the one hot encoding needs care: the
encoding is not the identity (A is [0,0,0,1]), so the argmax of an encoded base
is not its index in MUTATION_BASES. A test pins the invariant that the
reference base of a position is the one scoring exactly zero.
@vmesel

vmesel commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

Pushed four more commits. Two of them change behaviour and want a second opinion.

deepmex-figure: coordinate in, figure and impact ranking out

uv run deepmex-figure chrX:31126642-31126673

Strand and gene name come from RefSeq, the flanks from the UCSC API, so neither the reference files nor bedtools are needed. It writes the figure and a table of every flank base ranked by how much the model reacts to mutating it, each row carrying the genomic position, the position relative to the exon, the reference base and the delta of each substitution. For the DMD microexon the top positions are the thymines of the polypyrimidine tract, 14 to 22 nt upstream:

  #      position    rel  wt    impact   deltas
  1      31126694    -20   T     1.566   A-0.81  C-1.05  G-1.57
  2      31126690    -16   T     1.395   A-0.98  C-1.07  G-1.40

--genome/--conservation still read local files when given.

Panels A and B in one figure

Panel B (inclusion under a knock-down) is not model output, so it is read from a vast-tools INCLUSION_LEVELS_FULL table: the densities are the Beta posteriors rebuilt from the corrected read counts of the quality column, and the right hand curve is P(|dPSI| > x) with the 95% MV[dPsi] marked. Panel C is a UCSC browser screenshot and stays out of scope.

Two fixes that change what the tool outputs

The published panel is the reverse complement, not just the reverse. Extracting panel A from the Fig. 7 TIF and correlating it against the screening of the same microexon identifies the transform: r goes from 0.13 to 0.67 once the bases are complemented, and every other candidate transform scores below 0.21. Independent check: with the fix, plus and minus strand exons show the same signature upstream (purines negative, pyrimidines positive) across 38 microexons; before it, the 19 minus strand ones were inverted.

--vmax was off by a factor of 100. The 0.015 of the paper is on the 0-1 scale of the notebook, while the CLI reports 0-100. Real deltas reach ~1.0, so every figure came out fully saturated. The default is now 1.5.

Also: the DMD example passed strand +, but DMD is on the minus strand (16 RefSeq transcripts, all -).

Open question, not touched

The flanks of the published figure start one base earlier than the code computes. Correlation with the extracted panel goes 0.67 → 0.87 with the shift, and the deciding half is the downstream flank: the code reads [end+1, end+101), skipping the base immediately after the exon, where [end, end+100) looks right. I have left it alone because it changes the flanks of every prediction, not just the drawing (76.12 → 76.50 for DMD). Same question as the strand handling raised earlier — worth a look before either is changed.

vmesel added 3 commits August 13, 2026 15:11
Following the README gave a bare ModuleNotFoundError traceback: the default
environment was Python 3.15, where TensorFlow has no wheels, and the backend
was behind an optional extra that a plain `uv sync` did not install.

- tensorflow, keras and h5py are ordinary dependencies now, since scoring is
  what the tool is for. A `python_version < '3.15'` marker keeps `uv sync`
  resolvable on 3.15, where they are simply left out
- .python-version drops to 3.13, the newest interpreter the backend supports.
  The code still runs on 3.15 (make install-3.15) and so do the demo, the
  tests and the linter
- load_cnn_model raises ModelUnavailableError naming the missing package and
  the command to fix it, and the three command lines print it and exit 1
  instead of unwinding a traceback
- the `model` extra is gone; `genome` and `gtf` stay optional
- positionscore and deepmex-figure share one load_knockdown helper
predictor.ipynb downloaded hg19.fa and hg19.100way.phastCons.bw and pointed
GENOME/CONSERVATION_FILE at them, while the model was trained on hg38: its
exon coordinates come from Homo_sapiens.GRCh38.76.gtf and its conservation
from the hg38 phastCons track. Checked against the coordinates printed in the
training notebook, 36 of 38 are annotated exons in hg38 and 2 of 38 in hg19.

The mismatch was silent. Scoring the DMD microexon of Fig. 7A, a coordinate
anyone would paste from the paper, gives 0.76 on hg38 and 0.61 when the same
hg38 coordinate is read against hg19 files: still above the 0.5 threshold,
just wrong. Reading the matching hg19 coordinate against hg19 files gives 0.77,
so the assembly itself is not the problem; mixing the two is.

- the downloads and the two paths now point at hg38, and the stale hg19 run
  logs are cleared
- the example is the DMD microexon instead of an arbitrary chr1 interval, so
  the number it prints can be checked against the paper
- a header explains what was wrong and points at the command line tools
- the kernel says python3, which the code already compiled under
Executing predictor.ipynb end to end turned up two breaks that had nothing to
do with the assembly:

- it loaded '../Merged_100_filters_conservation_all_strands_merged_model_first_test-40-0.71.hdf5',
  a file that has never been in the repository. The shipped model is
  src/saved_model.hdf5, which is '../saved_model.hdf5' from the notebook
- load_model() without compile=False fails on Keras 3, because the optimizer
  of this Keras 2.3 file no longer deserializes

With those fixed it runs to the end and reports 0.76123744 for the DMD
microexon, matching the 76.124 of deepmex-figure to every digit, which is a
useful check that the refactor kept the behaviour.

Stale outputs from the 2020 hg19 run are cleared, so nothing in the notebook
still shows the old assembly or the Keras 2 banner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant