feat: add macOS support and harden cross-platform memory scanner#23
Open
JeanExtreme002 wants to merge 12 commits into
Open
feat: add macOS support and harden cross-platform memory scanner#23JeanExtreme002 wants to merge 12 commits into
JeanExtreme002 wants to merge 12 commits into
Conversation
Major release adding native macOS support, fixing latent bugs in the Windows/Linux backends, and tightening cross-platform robustness. See CHANGELOG.md for the full list. Added - macOS backend via Mach VM APIs (task_for_pid, mach_vm_read_overwrite, mach_vm_write, mach_vm_region, mach_vm_protect). Self-process works without entitlements; cross-process needs com.apple.security.cs.debugger or SIP off + root. - snapshot_memory_regions() + memory_regions= kwarg on search_by_value* and search_by_addresses for refine-scan workflows. - bufflength is now optional for numeric types (int->4, float->8, bool->1). - iter_region_chunks reads multi-GB regions in 256MB chunks across all three backends (prevents OOM in browser/JVM-sized targets). - PyMemoryEditorError base + AmbiguousProcessNameError. - py.typed marker; mypy + pytest-cov in CI. Fixed (critical) - Platform detection: "win" in sys.platform matched darwin, breaking macOS imports. - Read/Write/process_vm_*v now check argtypes/return value; failed reads no longer return zeroed buffers indistinguishable from real data. - scan_memory off-by-one (skipped last value of each region). - scan_memory_for_exact_value NOT_EXACT_VALUE no longer yields every non-matching byte; aligned to target_value_size. - WindowsProcess default permission: PROCESS_VM_READ instead of PROCESS_ALL_ACCESS (least privilege). - Permission gate now requires VM_READ explicit OR all PROCESS_ALL_ACCESS bits set (was passing on any single bit). - ProcessOperationsEnum.PROCESS_TERMINATE was 0x0800 — same as PROCESS_SUSPEND_RESUME — making it a silent Enum alias. Corrected to 0x0001 per MSDN. - Linux MEMORY_BASIC_INFORMATION fields widened to 64-bit (regions > 4GB no longer truncate). - Linux /proc/<pid>/maps inode parsed as decimal (was hex). - Windows MEMORY_BASIC_INFORMATION layout picked per target via IsWow64Process (no more corruption when 64-bit Python attaches to 32-bit target). - macOS write_process_memory to a read-only page now transparently elevates protection via mach_vm_protect and restores it. - Linux scan filters shared mappings (parity with Win32/macOS). - read_process_memory(addr, str, n) now decodes with errors="replace" to match convert_from_byte_array. Performance - 6-8x speedup on numeric scans via struct.iter_unpack + inlined comparison loops per scan_type. NOT_EXACT_VALUE overlap check is now O(log m) via bisect_left. Tooling - CI: 3 OSes (ubuntu/windows/macos) x 6 Pythons (3.8-3.13), flake8 gate, mypy informational, pytest-cov. - Conventional Commits enforced on PR title (lint-pr-title.yml). - Dependabot config (open-pull-requests-limit: 0; security alerts still fire). - Auto-delete head branch on PR close. - Tk sample now requires Tk >= 8.6 and aborts with platform-specific install hints when missing or outdated. Removed - Unused PyMemoryEditor.linux.ptrace package. - Unused PyMemoryEditor.util.search (KMP/BMH never used in scan path). - Python 3.6 and 3.7 support (minimum is now 3.8).
The default permission on WindowsProcess is now PROCESS_VM_READ (a 2.0 breaking change), so existing write tests in test_editor.py started failing on Windows CI. Explicitly request VM_WRITE | VM_OPERATION on Win32; Linux and macOS ignore the kwarg as before.
- Add mypy override that ignores errors in win32/linux/macos backends. Each backend uses symbols (ctypes.windll, WINFUNCTYPE, Mach types) that only resolve on their target OS; mypy running on one host can't validate the others. - Cast() generic-return-vs-concrete-bytes/str in convert_from_byte_array. - Loosen get_c_type_of return type to Any (returns either _SimpleCData or ctypes.Array without a common base). - Tighten _as_bytes to only treat real bytes as a no-op (bytearray now goes through the bytes() conversion). - Move GetProcessIdByWindowTitle import into the function body so mypy on non-Windows hosts doesn't see it as undefined.
test_search_by_int and test_search_by_float iterate every address yielded by search_by_value_between and read it back to verify the value. A page mapped at scan time can be decommitted/protected before the subsequent read — the syscall now surfaces this as OSError (it used to silently return zeros). Wrap the read in try/except OSError, matching the pattern already used in test_search_by_string.
VirtualQueryEx requires PROCESS_QUERY_INFORMATION (or PROCESS_QUERY_ LIMITED_INFORMATION) in addition to PROCESS_VM_READ. With only PROCESS_VM_READ — the previous default — VirtualQueryEx returns 0, so get_memory_regions / snapshot_memory_regions / search_by_value* / search_by_addresses all came back empty. Exposed by Windows CI when test_region_snapshot opened a process without an explicit permission. The new default is PROCESS_VM_READ | PROCESS_QUERY_INFORMATION (exposed as the DEFAULT_PERMISSION constant). README and CHANGELOG updated. tests/test_editor.py also adds PROCESS_QUERY_INFORMATION to its permission combo to make the read-back loop in search-by-value tests robust across Windows versions (it happened to work in Python 3.11 by implicit grant, but the explicit bit makes it deterministic).
GitHub's macOS-latest runner pool is much smaller than ubuntu/windows. Running 6 Python versions × macos in parallel stalls the PR for 10+ minutes in queue without acquiring a runner. macOS only validates that the Mach backend works — the Python version doesn't change that surface, so we drop down to a single (stable) Python on macos-latest while keeping the full 6-version matrix on ubuntu/windows.
The macos-latest (Apple Silicon arm64) runner pool is heavily congested on free-tier accounts — jobs sit in queue for 15+ minutes without acquiring a runner. The macos-13 (Intel x86_64) pool is much larger and exercises identical code paths: Mach VM structs are fixed-size by design (mach_port_t = uint32, mach_vm_address_t = uint64, etc.), so x86_64 and arm64 hit the same struct layout and syscall surface.
GitHub-hosted macOS runner pools are often congested and a job can sit in queue for 30+ minutes without acquiring a runner. Stop the PR from stalling on this: - continue-on-error: true for macOS jobs (ubuntu and windows still gate the merge; macOS is best-effort) - 25-minute timeout caps the wait; real test runs finish in well under 10 min when a runner is available
timeout-minutes counts execution time, not queue time, so a macOS job stuck waiting for a runner can stall a PR indefinitely. The fix: - Split macOS into its own build-macos job - Gate it with: if: github.event_name != 'pull_request' - Runs on push-to-main, weekly cron, and workflow_dispatch — never blocks a PR PRs are gated by ubuntu × 5 Pythons + windows × 5 Pythons + lint, which already provides cross-platform coverage. The Mach backend is validated by the merged code via cron/dispatch and by local self-process tests in dev.
GitHub-hosted macOS runners are heavily congested on free-tier accounts. Even with continue-on-error and timeouts, the job blocked the workflow UI for tens of minutes per run. The Mach backend is covered by local self-process tests in dev; contributors with macOS hardware can run the suite directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major release adding native macOS support via the Mach VM APIs, fixing latent bugs in the Windows/Linux backends, and tightening cross-platform robustness. Full notes in
CHANGELOG.md.Highlights
task_for_pid,mach_vm_read_overwrite,mach_vm_write,mach_vm_region,mach_vm_protect. Self-process works without entitlements; cross-process needscom.apple.security.cs.debuggeror SIP off + root."win" in "darwin"), unchecked syscall returns,scan_memoryoff-by-one,PROCESS_TERMINATEbeing a silent alias ofPROCESS_SUSPEND_RESUME(both0x0800), defaultpermissionlowered fromPROCESS_ALL_ACCESStoPROCESS_VM_READ, LinuxMEMORY_BASIC_INFORMATIONwidened to 64-bit, WOW64-aware MBI layout on Windows, transparentmach_vm_protectflip on macOS writes to read-only pages.struct.iter_unpack+ inlined per-scan_typecomparison loops;NOT_EXACT_VALUEoverlap check dropped from O(n·m) to O(n·log m) viabisect_left; multi-GB regions chunked at 256 MB across all backends.bufflengthoptional for numeric types (int/float/boolinferred),snapshot_memory_regions()+memory_regions=kwarg for refine-scan workflows,case_sensitive=exposed onOpenProcess,PyMemoryEditorErrorbase class for all library exceptions.flake8as a gate,mypyinformational,pytest-covreporting, Conventional Commits enforced on PR titles, Dependabot configured to suppress routine version PRs (security alerts still fire), auto-delete of merged branches.Breaking changes
permissionis nowPROCESS_VM_READ; writers must opt in toPROCESS_VM_WRITE | PROCESS_VM_OPERATION.PROCESS_TERMINATEvalue corrected from0x0800→0x0001(anyone relying on the buggy alias will see different behavior — they were terminating processes when asking to suspend/resume).linux.ptraceandutil.search(KMP/BMH) packages removed (never used in the scan path).Test plan
flake8 PyMemoryEditor tests— cleanpython:3.12-slim) — 70 passed + 3 skipped in 6.57sBIGGER_THAN/SMALLER_THAN/VALUE_BETWEENover 50 MBmmap+mprotectintests/test_macos_protect.pyIsWow64ProcesstestAmbiguousProcessNameErrorandcase_sensitivecovered bypsutil-mocked testspymemoryeditorTk sample on each OS with Tk ≥ 8.6