Skip to content

perf(pagerank): add fast, bounded, and memory-aware auto paths - #2007

Draft
lmeyerov wants to merge 5 commits into
feat/gfql-std-proceduresfrom
perf/std-pagerank-fast-bounded
Draft

perf(pagerank): add fast, bounded, and memory-aware auto paths#2007
lmeyerov wants to merge 5 commits into
feat/gfql-std-proceduresfrom
perf/std-pagerank-fast-bounded

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Stack

Draft follow-up stacked on #2006. Review only the four commits in this PR; merge #2006 first.

Summary

  • add explicit fast, bounded, and conservative memory-aware auto execution modes to graphistry.std.pagerank
  • keep the cuGraph-compatible signature/defaults plus Graphistry extras, direct/GFQL parity, original IDs, weighted/personalized/nstart/precomputed behavior, and shared convergence semantics
  • preserve the dataframe chunked path for bounded memory and graph500-26-scale workloads
  • use persistent compact NumPy/CuPy buffers for fit-in-memory execution; cuDF uses direct atomic scatter and fused rank update
  • never silently delegate std execution to igraph or cuGraph

Selection policy

auto is the default. With chunks=1, it selects fast only when a conservative calibrated scratch estimate is at most half detected free host/device memory. Unknown or tighter memory falls back to bounded. Explicit fast bypasses the estimate; explicit bounded preserves the chunkable dataframe path.

Matched evidence

Directed unweighted cit-Patents, prepared solver calls, 3 samples / 1 warmup:

backend median peak memory evidence
final std-pandas fast 1.386604s 372.05 MiB
igraph PRPACK 1.449537s baseline
final std-cuDF auto/fast 0.164078s 262.793 MiB RMM delta
cuGraph 26.02 0.035157s baseline

Final GPU samples: [0.166276, 0.164078, 0.162755]; std-cuDF is 4.67x cuGraph and clears the <=10x target. Rank mass is 1.0000000000000013, maximum retained delta is 8.131516293641283e-20, and top-10 IDs match. The prior std-cuDF baseline was 1.783722s / 979.153 MiB. The bounded graph500-26 memory result was preserved and cuGraph graph500-26 was not rerun.

Artifact SHA-256: 91b44ed81cb4b5b43beee6b341c2d8ce6369d4cedc0e7bde9f821adfc13ff53f.

The final DGX runtime artifact was built from implementation commit 7022fe742573e4212339bb326ad887f29b03f518; current PR head ec039cdea784036229d66d43d7752751a1db1671 adds CPU-only coverage contracts and does not change runtime code.

Validation

  • scoped lint/hygiene: pass
  • full configured mypy 2.3.1: pass, 337 source files
  • local focused CPU: 62 passed, 1 explicitly gated cuDF skip
  • exact CI-baseline changed-line coverage reproduction plus focused tests: 81.15% versus 80% gate
  • guarded DGX semantic lane with TEST_CUDF=1: 56 passed, no skips
  • guarded DGX matched performance/correctness/memory lane: pass

All Docker, GPU, dataset, profiling, and performance work ran only on dgx-spark through the perf lock and safe_run.sh, with one no-network container, RMM/cgroup caps, >=80 GiB host-memory floor, watchdog, and hard timeout. Companion benchmark PR: https://github.com/graphistry/pyg-bench/pull/194

``method='fast'`` bypasses the estimate and requires ``chunks=1``. Set
``method='bounded'`` with ``chunks>1`` for the strongest explicit peak-memory
control; setting ``chunks`` above one also makes ``auto`` select ``bounded``.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought should be clear that the main value of this implementation is lower memory consumption for handling bigger graphs on smaller GPUs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, typically better to use the GPU or CPU ones when fit in memory

import cudf

return cudf.Series(values)
return pd.Series(values)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Send most of these live in our SeriesT / DataframeT cross platform files ?

raise ConvergenceError(f"wcc did not converge in {max_iter} iterations")


def _pagerank_iterations(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pagerank code should be in its own pagerank.py, same for any other algorithms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant