Validated FutureQueryEval_qrels.txt against your bm25 results (reshaped to generate_trec_format) — clean, 148/148 aligned; worth a preflight before pyserini trec_eval?

[Madhvansh]

Hi — I'm the author of NECS, a dependency-light structural validator for TREC qrels/run pairs (query coverage, unjudged documents, rank ordering). It computes no metrics and certifies nothing. I'm filing this because I ran it against Rankify's shipped custom-dataset example and wanted to report the result plus one interoperability question.

In rankify/metrics/metrics.py, calculate_trec_metrics writes a temp TREC run through generate_trec_format ({id} Q0 {ctx.id} {rank+1} {score} {name}) and passes the (qrel, run) pair to python -m pyserini.eval.trec_eval. I reshaped your published examples/custom_datasets/FutureQueryEval_bm25.json into exactly that six-column run and validated it against FutureQueryEval_qrels.txt (NECS v0.3.1): a clean structural pass — 148 queries on both sides, perfectly aligned (no unknown or missing queries), 2,938 graded judgements (0/1/2), and 12,961 unjudged_document warnings where top-k retrieval runs past the pool. The qrels file on its own also parses with zero structural issues.

Since NECS lands on exactly your (qrel, trec_file) pair just before run_trec_eval, my question is simply whether a structural preflight there would be useful to you, or whether it's redundant with what pyserini already enforces.

Reproduce (or drag both files into the no-install browser page, https://madhvansh.github.io/Neural-E-Commerce-Search/validate.html):

python -m pip install "https://github.com/Madhvansh/Neural-E-Commerce-Search/releases/download/v0.3.1/neural_ecommerce_search_madhvansh-0.3.1-py3-none-any.whl"
python - <<'PY'
import json
rows = json.load(open("examples/custom_datasets/FutureQueryEval_bm25.json", encoding="utf-8"))
with open("bm25.run", "w", encoding="utf-8") as f:
    for r in rows:
        for rank, ctx in enumerate(r["ctxs"], 1):
            f.write(f"{r['query_id']} Q0 {ctx['id']} {rank} {ctx['score']} bm25\n")
PY
necs-validate --qrels examples/custom_datasets/FutureQueryEval_qrels.txt --run bm25.run --format json

"Not useful here" is a complete answer; I'm not requesting any change to Rankify. Scope/notes: https://github.com/Madhvansh/Neural-E-Commerce-Search/releases/tag/v0.3.1 — immutable build reference: tag v0.3.1 = commit 6fefdad10b60e71eedfcedee1491d6e043ebe670.