Skip to content

Make ELF loading respect program header virtual addresses for non-PIE binaries#1530

Open
cshung wants to merge 1 commit into
hyperlight-dev:mainfrom
cshung:cshung/non-pie-elf-loading
Open

Make ELF loading respect program header virtual addresses for non-PIE binaries#1530
cshung wants to merge 1 commit into
hyperlight-dev:mainfrom
cshung:cshung/non-pie-elf-loading

Conversation

@cshung

@cshung cshung commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Summary

For non-PIE (ET_EXEC) ELF binaries, the guest page table now maps the code region at the ELF's declared virtual address rather than identity-mapping it at the GPA. This allows statically-linked binaries with a fixed load address (e.g., --image-base=0x200000) to execute correctly.

Problem

Previously, Hyperlight assumed code GVA == code GPA (identity mapping). Non-PIE binaries that declare a non-zero base virtual address (via program header p_vaddr) would triple-fault because the guest CPU jumped to the ELF's declared entrypoint VA, which wasn't mapped in the page tables.

Solution

  • Compute code_virt_base from the ELF's lowest LOAD segment p_vaddr
  • For non-PIE (base_va > 0): map code at the declared VA in the guest page tables
  • For PIE (base_va == 0): preserve existing identity mapping behavior (with assertion to guard the invariant)
  • Compute entrypoint as code_virt_base + (entrypoint_va - base_va)

The fix leverages the existing Mapping struct's support for phys_base != virt_base — no changes to the page table code itself.

Testing

  • New non_pie_guest_hello_world integration test exercises full guest lifecycle (init, COW, function call, return value) with a non-PIE simpleguest built at --image-base=0x200000
  • All existing PIE guest tests continue to pass (identity mapping preserved)
  • Tested on Windows/WHP and Linux/KVM

Build infrastructure

  • Added build-rust-guests-non-pie Justfile targets
  • Non-PIE build runs first in guests recipe to avoid clobbering normal guest binaries
  • Added simple_guest_non_pie_as_string() test helper

Contributes to: #1408

Copilot AI review requested due to automatic review settings June 14, 2026 22:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds support for running and testing non-PIE Rust guest binaries, including updating snapshot virtual-address mapping so non-PIE guests execute at their declared ELF virtual addresses.

Changes:

  • Add a helper in hyperlight_testing to locate the non-PIE simpleguest binary.
  • Update snapshot mapping/entrypoint calculation to support non-identity VA mappings for non-PIE code regions.
  • Add a new integration test and build automation (Justfile) to produce and run a non-PIE guest.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
src/hyperlight_testing/src/lib.rs Adds path helper(s) for locating non-PIE Rust guest binaries.
src/hyperlight_host/tests/integration_test.rs Adds an integration test that boots and calls into a non-PIE guest.
src/hyperlight_host/src/sandbox/snapshot/mod.rs Adjusts snapshot mappings and entrypoint VA calculation to support non-PIE guests.
Justfile Adds tasks to build and stage non-PIE Rust guest artifacts.

Comment thread Justfile Outdated
Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs
Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs Outdated
Comment thread src/hyperlight_testing/src/lib.rs
@cshung
cshung force-pushed the cshung/non-pie-elf-loading branch from 7468e06 to 40c43be Compare June 14, 2026 23:10
@cshung

cshung commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed Copilot review feedback:

  1. Justfile (RUSTFLAGS on Linux) — Fixed. \RUSTFLAGS\ now prefixes \cargo\ directly after \cd &&\ so the env var applies to the build command.

  2. snapshot/mod.rs (virt_base mapping) — No change needed. The code region is a single contiguous block where
    gn.guest_region.start == load_addr\ — there is no offset to lose. The \�ssert_eq!\ is deliberate per codebase convention.

  3. snapshot/mod.rs (entrypoint underflow) — Fixed. Now uses \checked_sub\ with a proper error for malformed ELFs.

  4. lib.rs (doc says 'elf binary') — No change needed. Hyperlight guests are always ELF regardless of host OS.

@ludfjig ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Jun 15, 2026
@cshung
cshung force-pushed the cshung/non-pie-elf-loading branch 2 times, most recently from 3fa110a to 35fef7b Compare June 15, 2026 19:43

@ludfjig ludfjig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @syntactically could you have a look too

@syntactically syntactically left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this! It looks like it is moving in a good direction. I have a couple of minor suggestions/comments, as well as a bit of an alternative design that you could take (but totally up to you on that one!)

Comment thread Justfile Outdated
# to avoid clobbering the normal PIE guest artifacts.
# RUST_TARGET_PATH tells cargo where to find the x86_64-hyperlight-none.json spec.
build-rust-guests-non-pie target=default-target:
{{ if os() == "windows" { "New-Item -ItemType Directory -Path " + root + "/src/tests/rust_guests/target-non-pie -Force | Out-Null; Copy-Item " + root + "/src/tests/rust_guests/target/sysroot/lib/rustlib/x86_64-hyperlight-none/target.json " + root + "/src/tests/rust_guests/target-non-pie/x86_64-hyperlight-none.json -Force; $env:RUST_TARGET_PATH='" + root + "/src/tests/rust_guests/target-non-pie'; $env:RUSTFLAGS='--sysroot " + root + "/src/tests/rust_guests/target/sysroot --cfg=hyperlight --check-cfg=cfg(hyperlight) -C link-args=-eentrypoint -C relocation-model=static -C link-args=--no-pie -C link-args=--image-base=0x200000'; $env:RUSTC_BOOTSTRAP='1';" } else { "mkdir -p " + root + "/src/tests/rust_guests/target-non-pie && cp " + root + "/src/tests/rust_guests/target/sysroot/lib/rustlib/x86_64-hyperlight-none/target.json " + root + "/src/tests/rust_guests/target-non-pie/x86_64-hyperlight-none.json && RUST_TARGET_PATH='" + root + "/src/tests/rust_guests/target-non-pie' RUSTFLAGS='--sysroot " + root + "/src/tests/rust_guests/target/sysroot --cfg=hyperlight --check-cfg=cfg(hyperlight) -C link-args=-eentrypoint -C relocation-model=static -C link-args=--no-pie -C link-args=--image-base=0x200000' RUSTC_BOOTSTRAP=1" } }} cd src/tests/rust_guests/simpleguest && cargo build --target x86_64-hyperlight-none --target-dir ../target-non-pie --profile={{ if target == "debug" { "dev" } else { target } }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On first glance, linking the standard library built with different codegen opts seemed a little sketchy, but I guess it should probably be OK here, at least as long as we don't have something like a GOT-generating relocation show up for some reason?

Do you need the -C link-args=--no-pie? My read of the docs / quick test on aarch64 made me think that that should be more-or-less implied by the -C relocation-model=static on the rust command that invokes the linker.

More seriously, I would prefer to avoid copying so much of the RUSTFLAGS logic from cargo-hyperlight. Is the sysroot rebuild a serious problem for CI performance? If so, maybe there is a way to address it in cargo-hyperlight by e.g. adding a flag to prevent the sysroot being rebuilt (or stripping some of these things like link-args when building the sysroot).

Also, if you do pursue the approach here, why copy the target json to a new path when the path has to be passed in via an environment variable anyway?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. I've switched to using cargo hyperlight build directly with just the non-PIE link flags and a separate --target-dir. This avoids duplicating any cargo-hyperlight internals.

Regarding --no-pie - you're probably right that relocation-model=static implies it, but I kept it explicit for clarity. Happy to remove if you prefer.

// For the code region, use code_virt_base as the GVA.
// For non-PIE this is the ELF's declared base VA (non-identity mapping).
// For PIE this should equal the GPA (identity mapping).
let virt_base = if rgn.region_type == MemoryRegionType::Code {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we should have some check that the executable mapping here does not conflict with any other mappings.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call - added a conflict check that verifies the non-PIE code VA range doesn't overlap with any other memory region. This actually caught a real bug: the original --image-base=0x200000 produced a binary that extended into the PEB region. Bumped the base to 0x1000000 (16MB) to avoid this.

// For non-PIE binaries (base_va > 0), the code should appear at the
// ELF's declared virtual address. For PIE binaries (base_va == 0),
// we use the physical load address (identity mapping).
let code_virt_base = if base_va > 0 { base_va } else { load_addr };

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this choice be made based on whether the executable has .et_type = ET_DYN vs ET_EXEC, rather than the first phdr VA?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - now using elf.header.e_type == ET_DYN via an is_pie() method on ExeInfo/ElfInfo instead of checking base_va > 0. Much cleaner semantically.

let mapping = Mapping {
phys_base: rgn.guest_region.start as u64,
virt_base: rgn.guest_region.start as u64,
virt_base,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach probably works for now, especially with executables where all the segments are contiguous. One other thing that we could do while we are already touching this code is switch to mapping the ELF a bit more "properly", by copying every section copied by a segment into the snapshot contiguously (similarly to how it is being done now, but without using phdr.p_vaddr, so that there is no risk of large gaps appearing for no reason) and then making one of these vmem::map calls for each segment (~= PT_LOAD program header). In addition to being more efficient if we encounter a file with gaps between segments (at least a few kilobytes of gaps often exist due to page-aligning the phdr VAs), this would allow us to have better default permissions in the guest page tables by using the permission information in phdr.p_flags. Probably the way to do that would be to remove this mapping of this region entirely and change the exe interface in exe.rs/implementation in elf.rs to replace load/load_at with functions that take a mapping operations and do the mapping as they walk over the phdrs. That would be my slight preference/how I would do it, but I will leave it up to you whether or not to pursue it, since it is a nontrivially-larger change.

(Unrelated, not-really-review note: the need to fix up the physical vs virtual discontinuity is continuing a trend of several things that makes me slightly question the actual utility of SandboxMemoryLayout::get_memory_regions. It might be worth thinking about whether it makes sense to get rid of that / some more bits of SandboxMemoryLayout at some point, but that's definitely not a discussion for this PR).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed this would be the more proper approach. I'll leave it for a follow-up since it's a larger change, but it's on the radar.

@cshung cshung added the ready-for-review PR is ready for (re-)review label Jun 23, 2026
Comment thread src/hyperlight_host/tests/integration_test.rs Outdated

@ludfjig ludfjig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but will once again defer to @syntactically for final review

syntactically
syntactically previously approved these changes Jul 15, 2026

@syntactically syntactically left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thank you and sorry for the delay reviewing!

I do think that there is one minor move that would make the code a little bit cleaner / better fitting with existing conventions, which I've commented on inline. Feel free to push back with a reason why this code belongs here, though!

// For the code region, use code_virt_base as the GVA.
// For non-PIE this is the ELF's declared base VA (non-identity mapping).
// For PIE this should equal the GPA (identity mapping).
let virt_base = if rgn.region_type == MemoryRegionType::Code {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other place this logic (and the above non-overlapping check) could go is in hyperlight_host/src/mem/layout.rs where the region list is being built. I think it's maybe a bit better for the existing separation of concerns to have the snapshot code dumbly process the list that that function creates, and have all of the logic around guest layout there?

Add support for running non-PIE (ET_EXEC) guest binaries by mapping
code at the ELF's declared virtual address rather than assuming
identity mapping (physical == virtual).

Changes:
- Add is_pie() and base_va() methods to ExeInfo/ElfInfo to detect
  ET_DYN vs ET_EXEC binaries and extract the base virtual address
- Add SandboxMemoryLayout::code_virt_base() to compute the correct
  virtual base for the code region and validate it doesn't conflict
  with other memory regions
- Update snapshot creation to use non-identity virtual mapping for
  non-PIE code regions
- Add non-PIE guest build step to CI (cargo hyperlight with
  -C relocation-model=static -C link-args=--no-pie)
- Add integration test verifying non-PIE guest execution
- Add test helper for locating non-PIE guest binaries

Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6f20a05d-6bee-4e2e-b320-12f8d9759bbc
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants