From 7318673261d21bd5a0b0fa719cc148923edd0bd3 Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Fri, 7 Aug 2026 12:12:59 +0200 Subject: [PATCH 1/6] !sched/arch/libc: Give fork() and vfork() their real, separate semantics. NuttX implemented fork() and vfork() as the same function. Both were libc wrappers around a single up_fork() syscall; vfork() differed only by a trailing waitpid(). Underneath, the child joined the parent's address environment -- the same addrenv_join() that pthread_create() uses -- and got a private copy of the stack. So the child shared .data, .bss and the heap with its parent and ran concurrently with it. That is not fork(). It is vfork()-with-a-private-stack under fork()'s name, and the history says so: today's fork() is NuttX's old vfork(), renamed in c33d1c9c97 (2023) without any change of behaviour. The failure was silent -- a program written against POSIX fork() compiled, ran, and had its child's writes land in the parent's variables. Separate them into two primitives, chosen by which function the caller called rather than by what the hardware happens to be: fork() child gets its own copy of the parent's memory at the same virtual addresses; runs concurrently. Only where an address environment can be duplicated -- elsewhere it is not declared at all, so calling it is a build error naming the function. vfork() child shares the parent's memory; parent suspended until the child _exit()s or exec()s. Implementable everywhere. Below libc there is still one syscall. up_fork() gains a bool saying which primitive the caller used, since the per-architecture register snapshot is the same for both, and passes it to nxtask_setup_fork(), which is the single place the memory semantics are decided. The argument arrives in the first argument register and is never touched: each architecture's snapshot takes some other call-clobbered register for its scratch, so the flag is simply still there when the C worker is called. The vfork() parent suspension moves out of libc into nxtask_start_fork(), released from nxsched_release_tcb() by nxtask_resume_vfork(). Two things follow: the parent is resumed at exec(), since exec_swap() has already handed the child's pid to the loaded program by the time the vfork stub exits, and vfork() no longer depends on CONFIG_SCHED_WAITPID. Releasing there requires one fix in nxtask_exit(). It raises rtcb->lockcount directly rather than through sched_lock() while it tears the TCB down, so the nxsem_post() that wakes the vfork() parent leaves it queued where a blocked task collects while pre-emption is off -- g_pendingtasks, or g_readytorun on SMP -- and the matching raw lockcount-- does not publish it the way sched_unlock() would, leaving the parent stranded with nothing to move it on. The fix mirrors sched_unlock() for each case: nxsched_merge_pending(), or nxsched_deliver_task() under CONFIG_SMP. Both are no-ops while pre-emption is still disabled, and up_exit() re-reads this_task() afterwards, so a change of the ready-to-run head is honoured. Without it vfork() deadlocks wherever no other task happens to call sched_unlock() afterwards -- rv-virt:nsh64 and rv-virt:pnsh64, where NSH is blocked in waitpid() holding the lock, and qemu-armv8a:citest_smp, which hangs the moment the vfork() test runs. fork() is built on a new addrenv_fork(), backed by an up_addrenv_fork() hook that duplicates an address environment into freshly allocated pages mapped at the same virtual addresses -- unlike up_addrenv_clone(), which copies only the representation and leaves both pointing at the same page tables. The child then adopts the parent's stack geometry rather than being given a relocated copy: a pointer to a stack local taken before fork() must name the same object in the child that it named in the parent, and the parent's stack is already in the duplicate, with its contents, at the parent's address. No architecture implements up_addrenv_fork() yet, so this commit leaves fork() unavailable everywhere. That is the intended state. It withdraws fork() from ARCH_ARM, flat ARCH_ARM64, ARCH_RISCV, ARCH_SIM and ARCH_X86_64, where until now it named the sharing primitive; per-architecture patches restore it, with POSIX semantics, as up_addrenv_fork() lands. In the meantime the sharing primitive is still there under the name that describes it: vfork() for a child that runs a program, pthread_create() for a second flow of control that shares memory, posix_spawn() for both at once. Kconfig: ARCH_HAVE_VFORK inherits ARCH_HAVE_FORK's select lines, conditions included, so no configuration gains machinery; ARCH_HAVE_FORK is redefined to mean "can provide POSIX fork() semantics" and now depends on ARCH_ADDRENV. There is one deliberate departure from "verbatim". ARCH_ARM selected the fork family unconditionally, BUILD_KERNEL included, and that has never worked: on a kernel build the architecture's fork entry point sees the kernel's return address and stack pointer rather than the caller's, so the child resumes at a kernel address. On qemu-armv7a:knsh master faults in ostest's fork case with "Child did not run" and then a data abort; without the condition this change faults the same way through vfork(). ARCH_ARM64 and ARCH_X86_64 already carried "if !BUILD_KERNEL" for exactly this reason -- ARM was the outlier. Conditioning it turns a runtime fault into an honest absence, which is the whole point of the change; arch/arm takes the condition off again in the patch that adds its saved-syscall-frame path. Only the MMU-capable ARM ports are affected, since Cortex-M cannot build BUILD_KERNEL at all. Also fixes two latent syntax errors found on the way: a missing comma in riscv_fork.c and mips_fork.c, both in *_FRAMEPOINTER && !SAVE_GP branches that are never compiled today. BREAKING CHANGE: fork() is withdrawn from every architecture. It is no longer declared in unistd.h, so code that calls it fails to build with an error naming the function, and the sharing behaviour it used to have is gone rather than renamed. CONFIG_ARCH_HAVE_FORK no longer means "fork() exists"; it means "this configuration can provide POSIX fork() semantics", and no architecture selects it yet. Quick fix, chosen by why the call was made: to run a program vfork() + exec*(), or better posix_spawn() a second flow of control that pthread_create() shares the caller's memory a genuinely independent copy keep fork(), and wait for the per-arch patch of the process that implements up_addrenv_fork() and selects CONFIG_ARCH_HAVE_FORK Out-of-tree code that tests CONFIG_ARCH_HAVE_FORK to decide whether a fork-then-exec path is available wants CONFIG_ARCH_HAVE_VFORK instead, which is selected in exactly the places CONFIG_ARCH_HAVE_FORK used to be. The full migration guide is Documentation/guides/fork_vfork_migration.rst. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- arch/Kconfig | 40 ++- arch/arm/src/common/arm_fork.c | 126 ++++--- arch/arm/src/common/gnu/fork.S | 58 ++-- arch/arm/src/common/iar/fork.S | 52 +-- arch/arm64/src/common/arm64_fork.c | 78 +++-- arch/arm64/src/common/arm64_fork_func.S | 57 ++-- arch/ceva/src/common/ceva_fork.c | 18 +- arch/mips/Kconfig | 2 +- arch/mips/src/mips32/Kconfig | 2 +- arch/mips/src/mips32/fork.S | 38 ++- arch/mips/src/mips32/mips_fork.c | 11 +- arch/risc-v/src/common/CMakeLists.txt | 2 +- arch/risc-v/src/common/Make.defs | 2 +- arch/risc-v/src/common/fork.S | 68 ++-- arch/risc-v/src/common/riscv_fork.c | 143 ++++---- arch/sim/src/Makefile | 2 +- arch/sim/src/sim/CMakeLists.txt | 2 +- arch/sim/src/sim/sim_fork.c | 24 +- arch/sim/src/sim/sim_fork_arm.S | 42 ++- arch/sim/src/sim/sim_fork_arm64.S | 47 +-- arch/sim/src/sim/sim_fork_x86.S | 44 ++- arch/sim/src/sim/sim_fork_x86_64.S | 45 ++- arch/x86_64/src/common/CMakeLists.txt | 2 +- arch/x86_64/src/common/Make.defs | 2 +- arch/x86_64/src/common/fork.S | 41 +-- arch/x86_64/src/common/x86_64_fork.c | 82 +++-- include/nuttx/addrenv.h | 25 ++ include/nuttx/arch.h | 47 ++- include/nuttx/sched.h | 40 ++- include/sys/syscall_lookup.h | 4 +- include/unistd.h | 9 + libs/libbuiltin/libgcc/gcov.c | 6 + libs/libc/libc.csv | 1 + libs/libc/unistd/CMakeLists.txt | 2 +- libs/libc/unistd/Make.defs | 2 +- libs/libc/unistd/lib_fork.c | 68 ++-- sched/addrenv/addrenv.c | 65 ++++ sched/sched/sched.h | 10 + sched/sched/sched_releasetcb.c | 9 + sched/task/CMakeLists.txt | 2 +- sched/task/Make.defs | 2 +- sched/task/task_exit.c | 14 + sched/task/task_fork.c | 437 +++++++++++++++++++----- syscall/syscall.csv | 2 +- 44 files changed, 1202 insertions(+), 573 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index ab0a0c066c7e5..13e0db6ea983b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -11,7 +11,7 @@ config ARCH_ARM bool "ARM" select ARCH_HAVE_BACKTRACE select ARCH_HAVE_INTERRUPTSTACK - select ARCH_HAVE_FORK + select ARCH_HAVE_VFORK if !BUILD_KERNEL select ARCH_HAVE_STACKCHECK select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_STDARG_H @@ -29,7 +29,7 @@ config ARCH_ARM64 select ARCH_64BIT select ARCH_HAVE_BACKTRACE select ARCH_HAVE_INTERRUPTSTACK - select ARCH_HAVE_FORK if !BUILD_KERNEL && !BUILD_PROTECTED + select ARCH_HAVE_VFORK if !BUILD_KERNEL && !BUILD_PROTECTED select ARCH_HAVE_STACKCHECK select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_STDARG_H @@ -87,7 +87,7 @@ config ARCH_RISCV select ARCH_HAVE_CPUINFO select ARCH_HAVE_INTERRUPTSTACK select ARCH_HAVE_STACKCHECK - select ARCH_HAVE_FORK + select ARCH_HAVE_VFORK select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_SETJMP select ARCH_HAVE_STDARG_H @@ -111,7 +111,7 @@ config ARCH_SIM select ARCH_HAVE_TICKLESS select ARCH_HAVE_POWEROFF select ARCH_HAVE_TESTSET - select ARCH_HAVE_FORK if !HOST_WINDOWS + select ARCH_HAVE_VFORK if !HOST_WINDOWS select ARCH_HAVE_SETJMP select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_TCBINFO @@ -147,7 +147,7 @@ config ARCH_X86_64 select PCI_LATE_DRIVERS_REGISTER if PCI select ARCH_TOOLCHAIN_GNU select ARCH_HAVE_BACKTRACE - select ARCH_HAVE_FORK if !BUILD_KERNEL + select ARCH_HAVE_VFORK if !BUILD_KERNEL select ARCH_HAVE_SETJMP select ARCH_HAVE_PERF_EVENTS select ARCH_HAVE_POWEROFF @@ -482,9 +482,39 @@ config ARCH_HAVE_CPUID_MAPPING default n depends on ARCH_HAVE_MULTICPU +config ARCH_HAVE_VFORK + bool + default n + ---help--- + The architecture can implement POSIX vfork(): the child shares the + parent's memory and the parent is suspended until the child calls + _exit() or one of the exec family of functions. + config ARCH_HAVE_FORK bool default n + depends on ARCH_ADDRENV + ---help--- + The architecture can implement POSIX fork(): the child receives its + own copy of the parent's memory at the same virtual addresses, may + modify anything, may return from the function that called fork(), and + runs concurrently with the parent. + + This requires an address environment to duplicate, and an + up_addrenv_fork() to duplicate it with: the copy is backed by freshly + allocated pages holding a copy of the parent's contents, mapped at the + same virtual addresses. + + No architecture selects this yet. Two things are needed. First, + up_addrenv_fork() itself. Second, the architecture must build the + child's register context from the *user's* saved system call frame: + in a kernel build fork() is reached through a system call, so the + return address and stack pointer the architecture's fork entry point + can see for itself are the kernel's, not the caller's, and a child + built from those resumes at a kernel address. + + Where this is not selected fork() is not provided at all, and code + that calls it fails to build. config ARCH_HAVE_CRC32 bool diff --git a/arch/arm/src/common/arm_fork.c b/arch/arm/src/common/arm_fork.c index f98f8ec896a3c..db822caee7570 100644 --- a/arch/arm/src/common/arm_fork.c +++ b/arch/arm/src/common/arm_fork.c @@ -49,47 +49,56 @@ * Name: arm_fork * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The common ARM worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. + * + * What differs here is only the stack. Normally the child has a stack of + * its own, and this function fills it with a relocated copy of the + * parent's, rebasing the stack and frame pointers to match. When the + * child shares the parent's stack addresses -- a fork() child, inside its + * duplicated address environment -- there is nothing to relocate and the + * pointers are carried over unchanged. * * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up arm_fork(). - * 2) arm_fork() and calls nxtask_setup_fork(). + * 1) User code calls vfork() or fork(). The libc wrapper enters + * up_fork(), which collects context information and transfers control + * to arm_fork(). + * 2) arm_fork() calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: * - Allocation of the child task's TCB. * - Initialization of file descriptors and streams * - Configuration of environment variables - * - Allocate and initialize the stack + * - Establishing the child's address environment: joined to the + * parent's for vfork(), duplicated from it for fork() + * - Allocating the stack, or inheriting the parent's for fork() * - Setup the input parameters for the task. * - Initialization of the TCB (including call to up_initial_state()) * 4) arm_fork() provides any additional operating context. arm_fork must: * - Initialize special values in any CPU registers that were not * already configured by up_initial_state() - * 5) arm_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * 5) arm_fork() then calls nxtask_start_fork(), which for vfork() + * additionally suspends the caller. + * 6) which executes the child thread. * * nxtask_abort_fork() may be called if an error occurs between steps 3 and * 6. * * Input Parameters: - * context - Caller context information saved by fork() + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by the entry point * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. * ****************************************************************************/ -pid_t arm_fork(const struct fork_s *context) +pid_t arm_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -115,7 +124,7 @@ pid_t arm_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)(context->lr & ~1)); + child = nxtask_setup_fork((start_t)(context->lr & ~1), vfork); if (!child) { serr("ERROR: nxtask_setup_fork failed\n"); @@ -137,43 +146,60 @@ pid_t arm_fork(const struct fork_s *context) sinfo("Parent: stackutil:%" PRIu32 "\n", stackutil); - /* Make some feeble effort to preserve the stack contents. This is - * feeble because the stack surely contains invalid pointers and other - * content that will not work in the child context. However, if the - * user follows all of the caveats of fork() usage, even this feeble - * effort is overkill. - */ + if (child->stack_base_ptr == parent->stack_base_ptr) + { + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate: + * every stack address the child inherits is still the address it + * names. + */ - newtop = (uint32_t)child->stack_base_ptr + - child->adj_stack_size; + newsp = oldsp; + newfp = context->fp; + } + else + { + /* Make some feeble effort to preserve the stack contents. This is + * feeble because the stack surely contains invalid pointers and other + * content that will not work in the child context. However, if the + * user follows all of the caveats of vfork() usage, even this feeble + * effort is overkill. + * + * For a POSIX fork() child the stack contents are not merely a feeble + * effort: the child is entitled to use them, and it does. + */ - newsp = newtop - stackutil; + newtop = (uint32_t)child->stack_base_ptr + + child->adj_stack_size; - /* Move the register context to newtop. */ + newsp = newtop - stackutil; - memcpy((void *)(newsp - XCPTCONTEXT_SIZE), - child->xcp.regs, XCPTCONTEXT_SIZE); + /* Move the register context to newtop. */ - child->xcp.regs = (void *)(newsp - XCPTCONTEXT_SIZE); + memcpy((void *)(newsp - XCPTCONTEXT_SIZE), + child->xcp.regs, XCPTCONTEXT_SIZE); - memcpy((void *)newsp, (const void *)oldsp, stackutil); + child->xcp.regs = (void *)(newsp - XCPTCONTEXT_SIZE); - /* Was there a frame pointer in place before? */ + memcpy((void *)newsp, (const void *)oldsp, stackutil); - if (context->fp >= oldsp && context->fp < stacktop) - { - uint32_t frameutil = stacktop - context->fp; - newfp = newtop - frameutil; - } - else - { - newfp = context->fp; - } + /* Was there a frame pointer in place before? */ - sinfo("Old stack top:%08" PRIx32 " SP:%08" PRIx32 " FP:%08" PRIx32 "\n", - stacktop, oldsp, context->fp); - sinfo("New stack top:%08" PRIx32 " SP:%08" PRIx32 " FP:%08" PRIx32 "\n", - newtop, newsp, newfp); + if (context->fp >= oldsp && context->fp < stacktop) + { + uint32_t frameutil = stacktop - context->fp; + newfp = newtop - frameutil; + } + else + { + newfp = context->fp; + } + + sinfo("Old stack top:%08" PRIx32 " SP:%08" PRIx32 + " FP:%08" PRIx32 "\n", stacktop, oldsp, context->fp); + sinfo("New stack top:%08" PRIx32 " SP:%08" PRIx32 + " FP:%08" PRIx32 "\n", newtop, newsp, newfp); + } /* Update the stack pointer, frame pointer, and volatile registers. When * the child TCB was initialized, all of the values were set to zero. @@ -245,9 +271,9 @@ pid_t arm_fork(const struct fork_s *context) } #endif - /* And, finally, start the child task. On a failure, nxtask_start_fork() - * will discard the TCB by calling nxtask_abort_fork(). + /* And, finally, start the child task. A vfork() additionally suspends us + * until the child calls _exit() or exec(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } diff --git a/arch/arm/src/common/gnu/fork.S b/arch/arm/src/common/gnu/fork.S index 83e22326c26c8..60a1a2f07be71 100644 --- a/arch/arm/src/common/gnu/fork.S +++ b/arch/arm/src/common/gnu/fork.S @@ -1,5 +1,5 @@ /**************************************************************************** - * arch/arm/src/common/fork.S + * arch/arm/src/common/gnu/fork.S * * SPDX-License-Identifier: Apache-2.0 * @@ -38,43 +38,41 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives. Both need exactly the same thing from assembly -- a + * snapshot of the caller's callee-saved registers, stack pointer and + * return address -- and differ only in what the C code then does with it, + * so there is one entry point and the caller's r0 says which primitive was + * called. It is passed straight through to arm_fork(). * - * This thin layer implements fork by simply calling up_fork() with the - * fork() context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) arm_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point. + * 2) The entry point collects the context and calls arm_fork(). + * 3) arm_fork() calls nxtask_setup_fork(), which allocates and configures + * the child task's TCB. This consists of: * - Allocation of the child task's TCB. * - Initialization of file descriptors and streams * - Configuration of environment variables - * - Allocate and initialize the stack + * - Establishing the child's address environment + * - Allocating the stack, or inheriting the parent's for fork() * - Setup the input parameters for the task. * - Initialization of the TCB (including call to up_initial_state()) - * 4) arm_fork() provides any additional operating context. arm_fork must: + * 4) arm_fork() provides any additional operating context: * - Initialize special values in any CPU registers that were not * already configured by up_initial_state() + * - Relocate the copied stack, unless the child shares the parent's * 5) arm_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * 6) which executes the child thread. * * Input Parameters: - * None + * r0 - true for vfork(), false for fork() * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. * ****************************************************************************/ @@ -88,7 +86,7 @@ up_fork: /* Create a stack frame */ - mov r0, sp /* Save the value of the stack on entry */ + mov r3, sp /* Save the value of the stack on entry */ sub sp, sp, #FORK_SIZEOF /* Allocate the structure on the stack */ /* CPU registers */ @@ -102,11 +100,13 @@ up_fork: mov r7, r11 stmia r1!, {r4-r7} /* Save r8-r11 in the structure */ mov r5, lr /* Copy lr to a low register */ - stmia r1!, {r0,r5} /* Save sp and lr in the structure */ + stmia r1!, {r3,r5} /* Save sp and lr in the structure */ - /* Then, call arm_fork(), passing it a pointer to the stack structure */ + /* Then, call arm_fork(). r0 still holds the vfork flag: nothing above + * touches it. + */ - mov r0, sp + mov r1, sp bl arm_fork /* Recover r4-r7 that were destroyed before arm_fork was called */ @@ -114,7 +114,7 @@ up_fork: mov r1, sp ldmia r1!, {r4-r7} - /* Release the stack data and return the value returned by up_fork */ + /* Release the stack data and return the value returned by arm_fork */ ldr r1, [sp, #FORK_LR_OFFSET] mov r14, r1 diff --git a/arch/arm/src/common/iar/fork.S b/arch/arm/src/common/iar/fork.S index 3f16484a36530..53e3e221e3aca 100644 --- a/arch/arm/src/common/iar/fork.S +++ b/arch/arm/src/common/iar/fork.S @@ -50,43 +50,41 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives. Both need exactly the same thing from assembly -- a + * snapshot of the caller's callee-saved registers, stack pointer and + * return address -- and differ only in what the C code then does with it, + * so there is one entry point and the caller's r0 says which primitive was + * called. It is passed straight through to arm_fork(). * - * This thin layer implements fork by simply calling up_fork() with the - * fork() context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) arm_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point. + * 2) The entry point collects the context and calls arm_fork(). + * 3) arm_fork() calls nxtask_setup_fork(), which allocates and configures + * the child task's TCB. This consists of: * - Allocation of the child task's TCB. * - Initialization of file descriptors and streams * - Configuration of environment variables - * - Allocate and initialize the stack + * - Establishing the child's address environment + * - Allocating the stack, or inheriting the parent's for fork() * - Setup the input parameters for the task. * - Initialization of the TCB (including call to up_initial_state()) * 4) arm_fork() provides any additional operating context. arm_fork must: * - Initialize special values in any CPU registers that were not * already configured by up_initial_state() + * - Relocate the copied stack, unless the child shares the parent's * 5) arm_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * 6) which executes the child thread. * * Input Parameters: - * None + * r0 - true for vfork(), false for fork() * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. * ****************************************************************************/ @@ -95,7 +93,7 @@ up_fork: /* Create a stack frame */ - mov r0, sp /* Save the value of the stack on entry */ + mov r3, sp /* Save the value of the stack on entry */ sub sp, sp, #FORK_SIZEOF /* Allocate the structure on the stack */ /* CPU registers */ @@ -112,14 +110,16 @@ up_fork: /* Save the frame pointer, stack pointer, and return address */ str r11, [sp, #FORK_FP_OFFSET] /* fp not defined. use r11 */ - str r0, [sp, #FORK_SP_OFFSET] + str r3, [sp, #FORK_SP_OFFSET] str lr, [sp, #FORK_LR_OFFSET] /* Floating point registers (not yet) */ - /* Then, call arm_fork(), passing it a pointer to the stack structure */ + /* Then, call arm_fork(). r0 still holds the vfork flag: nothing above + * touches it. + */ - mov r0, sp + mov r1, sp bl arm_fork /* Release the stack data and return the value returned by arm_fork */ diff --git a/arch/arm64/src/common/arm64_fork.c b/arch/arm64/src/common/arm64_fork.c index fbbf6a56da0e5..b7d5af1f9f7d4 100644 --- a/arch/arm64/src/common/arm64_fork.c +++ b/arch/arm64/src/common/arm64_fork.c @@ -69,20 +69,18 @@ void arm64_fork_fpureg_save(struct fork_s *context) #endif /**************************************************************************** - * Name: fork + * Name: arm64_fork * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The common ARM64 worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. * * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up arm64_fork(). + * 1) User code calls vfork() or fork(). up_fork() collects context + * information and transfers control to arm64_fork(). * 2) arm64_fork() and calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: @@ -103,7 +101,8 @@ void arm64_fork_fpureg_save(struct fork_s *context) * 6. * * Input Parameters: - * context - Caller context information saved by fork() + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -113,7 +112,7 @@ void arm64_fork_fpureg_save(struct fork_s *context) * ****************************************************************************/ -pid_t arm64_fork(const struct fork_s *context) +pid_t arm64_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -125,7 +124,7 @@ pid_t arm64_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)context->lr); + child = nxtask_setup_fork((start_t)context->lr, vfork); if (!child) { serr("ERROR: nxtask_setup_fork failed\n"); @@ -143,28 +142,45 @@ pid_t arm64_fork(const struct fork_s *context) DEBUGASSERT(stacktop > context->sp); stackutil = stacktop - context->sp; - /* Make some feeble effort to preserve the stack contents. This is - * feeble because the stack surely contains invalid pointers and other - * content that will not work in the child context. However, if the - * user follows all of the caveats of fork() usage, even this feeble - * effort is overkill. - */ - - newtop = (uint64_t)child->stack_base_ptr + - child->adj_stack_size; - newsp = newtop - stackutil; - memcpy((void *)newsp, (const void *)context->sp, stackutil); - - /* Was there a frame pointer in place before? */ - - if (context->fp >= context->sp && context->fp < stacktop) + if (child->stack_base_ptr == parent->stack_base_ptr) { - uint64_t frameutil = stacktop - context->fp; - newfp = newtop - frameutil; + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate: + * every stack address the child inherits is still the address it + * names. + */ + + newsp = context->sp; + newfp = context->fp; } else { - newfp = context->fp; + /* Make some feeble effort to preserve the stack contents. This is + * feeble because the stack surely contains invalid pointers and other + * content that will not work in the child context. However, if the + * user follows all of the caveats of vfork() usage, even this feeble + * effort is overkill. + * + * For a POSIX fork() child the stack contents are not merely a feeble + * effort: the child is entitled to use them, and it does. + */ + + newtop = (uint64_t)child->stack_base_ptr + + child->adj_stack_size; + newsp = newtop - stackutil; + memcpy((void *)newsp, (const void *)context->sp, stackutil); + + /* Was there a frame pointer in place before? */ + + if (context->fp >= context->sp && context->fp < stacktop) + { + uint64_t frameutil = stacktop - context->fp; + newfp = newtop - frameutil; + } + else + { + newfp = context->fp; + } } /* Update the stack pointer, frame pointer, and volatile registers. When @@ -235,5 +251,5 @@ pid_t arm64_fork(const struct fork_s *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } diff --git a/arch/arm64/src/common/arm64_fork_func.S b/arch/arm64/src/common/arm64_fork_func.S index 79cd9566076d5..83fb6017879e4 100644 --- a/arch/arm64/src/common/arm64_fork_func.S +++ b/arch/arm64/src/common/arm64_fork_func.S @@ -41,46 +41,46 @@ ****************************************************************************/ /**************************************************************************** - * Name: fork + * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives. Both need exactly the same thing from assembly -- a + * snapshot of the caller's registers, stack pointer and return address -- + * and differ only in what the C code then does with it, so there is one + * entry point and the caller's x0 says which primitive was called. It is + * saved into the snapshot along with the other argument registers, which + * is harmless: x0-x18 are caller-saved and arm64_fork() does not + * propagate them to the child. * - * This thin layer implements fork by simply calling up_fork() with the - * fork() context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) arm64_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point. + * 2) The entry point collects the context and calls arm64_fork(). + * 3) arm64_fork() calls nxtask_setup_fork(), which allocates and + * configures the child task's TCB. This consists of: * - Allocation of the child task's TCB. * - Initialization of file descriptors and streams * - Configuration of environment variables - * - Allocate and initialize the stack + * - Establishing the child's address environment + * - Allocating the stack, or inheriting the parent's for fork() * - Setup the input parameters for the task. * - Initialization of the TCB (including call to up_initial_state()) - * 4) arm64_fork() provides any additional operating context. arm64_fork must: + * 4) arm64_fork() provides any additional operating context: * - Initialize special values in any CPU registers that were not * already configured by up_initial_state() + * - Relocate the copied stack, unless the child shares the parent's * 5) arm64_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * 6) which executes the child thread. * * Input Parameters: - * None + * x0 - true for vfork(), false for fork() * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. * ****************************************************************************/ @@ -122,10 +122,13 @@ SECTION_FUNC(text, up_fork) ldp x0, x30, [sp], #16 #endif - /* Then, call arm64_fork(), passing it a pointer to the stack structure */ + /* Then, call arm64_fork(), passing it the vfork flag and a pointer to + * the stack structure. The flag comes back out of the snapshot: the + * sequence above clobbers x0, but it saved it first. + */ - mov x0, sp - mov x1, #0 + ldr x0, [sp, #8 * FORK_REG_X0] + mov x1, sp bl arm64_fork /* Release the stack data and return the value returned by arm64_fork */ diff --git a/arch/ceva/src/common/ceva_fork.c b/arch/ceva/src/common/ceva_fork.c index 2eca6db1b7620..7cf6bd684ff18 100644 --- a/arch/ceva/src/common/ceva_fork.c +++ b/arch/ceva/src/common/ceva_fork.c @@ -50,11 +50,16 @@ * called, or calls any other function before successfully calling _exit() * or one of the exec family of functions. * + * Those are vfork()'s semantics, and vfork() is all this architecture can + * provide: POSIX fork() needs an address environment to duplicate and + * CEVA has none. So up_fork()'s argument is always true here, and the + * entry point does not carry it through the context-saving trap. + * * The overall sequence is: * * 1) User code calls fork(). fork() collects context information and * transfers control up ceva_fork(). - * 2) ceva_fork()and calls nxtask_forksetup(). + * 2) ceva_fork() and calls nxtask_forksetup(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: * - Allocation of the child task's TCB. @@ -73,7 +78,7 @@ * nxtask_abort_fork() may be called if an error occurs between steps 3 & 6. * * Input Parameters: - * regs - Caller context information saved by fork() + * regs - Caller context information saved by up_fork() * * Return: * Upon successful completion, fork() returns 0 to the child process and @@ -97,9 +102,14 @@ pid_t ceva_fork(const uint32_t *regs) void *argv; int ret; + /* How large is the parent's stack argument area? */ + + argsize = (uintptr_t)parent->stack_base_ptr - + (uintptr_t)parent->stack_alloc_ptr; + /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork(parent->start, &argsize); + child = nxtask_setup_fork(parent->start, true); if (!child) { serr("ERROR: nxtask_setup_fork failed\n"); @@ -204,7 +214,7 @@ pid_t ceva_fork(const uint32_t *regs) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, true); #else /* CONFIG_SCHED_WAITPID */ return (pid_t)ERROR; #endif diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index 1d1f2a2965d87..6efaa22ba23a4 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -56,7 +56,7 @@ endchoice config ARCH_MIPS32 bool default n - select ARCH_HAVE_FORK + select ARCH_HAVE_VFORK config ARCH_MIPS_M4K bool diff --git a/arch/mips/src/mips32/Kconfig b/arch/mips/src/mips32/Kconfig index 16fa348165463..c30962ed7b2e3 100644 --- a/arch/mips/src/mips32/Kconfig +++ b/arch/mips/src/mips32/Kconfig @@ -93,7 +93,7 @@ config MIPS32_TOOLCHAIN_MICROCHIP_XC32_LICENSED config MIPS32_FRAMEPOINTER bool "ABI Uses Frame Pointer" default n - depends on ARCH_HAVE_FORK + depends on ARCH_HAVE_VFORK ---help--- Register r30 may be a frame pointer in some ABIs. Or may just be saved register s8. It makes a difference for fork handling. diff --git a/arch/mips/src/mips32/fork.S b/arch/mips/src/mips32/fork.S index 790aa8cd43f42..be2b263a3351a 100644 --- a/arch/mips/src/mips32/fork.S +++ b/arch/mips/src/mips32/fork.S @@ -47,20 +47,17 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the behavior is - * undefined if the process created by fork() either modifies any data other than - * a variable of type pid_t used to store the return value from fork(), or returns - * from the function in which fork() was called, or calls any other function before - * successfully calling _exit() or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). Both need exactly the same snapshot from + * assembly and differ only in what the C code then does with it, so there + * is one entry point and the caller's $a0 says which primitive was called. + * It is passed straight through to mips_fork(). * - * This thin layer implements fork by simply calling up_fork() with the fork() - * context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) mips_fork() and calls nxtask_setup_fork(). + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls mips_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. This * consists of: * - Allocation of the child task's TCB. @@ -76,7 +73,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * $a0 - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and returns @@ -88,16 +85,19 @@ .text .align 2 - .globl up_fork - .type up_fork, function .set nomips16 #ifdef CONFIG_MIPS_MICROMIPS .set micromips #endif + + .globl up_fork + .type up_fork, function .ent up_fork up_fork: - /* Create a stack frame */ + /* Create a stack frame. $a0 holds the vfork flag and is not part of the + * snapshot, so it is still there when mips_fork() is called below. + */ move $t0, $sp /* Save the value of the stack on entry */ addiu $sp, $sp, -FORK_SIZEOF /* Allocate the structure on the stack */ @@ -130,9 +130,11 @@ up_fork: /* Floating point registers (not yet) */ - /* Then, call mips_fork(), passing it a pointer to the stack structure */ + /* Then, call mips_fork(), passing it a pointer to the stack structure. + * $a0 already holds the vfork flag. + */ - move $a0, $sp + move $a1, $sp jal mips_fork nop diff --git a/arch/mips/src/mips32/mips_fork.c b/arch/mips/src/mips32/mips_fork.c index 3d931d1da1d56..6ae724a52f466 100644 --- a/arch/mips/src/mips32/mips_fork.c +++ b/arch/mips/src/mips32/mips_fork.c @@ -79,7 +79,8 @@ * and 6 * * Input Parameters: - * context - Caller context information saved by fork() + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -89,7 +90,7 @@ * ****************************************************************************/ -pid_t mips_fork(const struct fork_s *context) +pid_t mips_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -113,7 +114,7 @@ pid_t mips_fork(const struct fork_s *context) context->fp, context->sp, context->ra, context->gp); #else sinfo("fp:%08" PRIx32 " sp:%08" PRIx32 " ra:%08" PRIx32 "\n", - context->fp context->sp, context->ra); + context->fp, context->sp, context->ra); #endif #else sinfo("s5:%08" PRIx32 " s6:%08" PRIx32 " s7:%08" PRIx32 @@ -130,7 +131,7 @@ pid_t mips_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)context->ra); + child = nxtask_setup_fork((start_t)context->ra, vfork); if (!child) { sinfo("nxtask_setup_fork failed\n"); @@ -217,5 +218,5 @@ pid_t mips_fork(const struct fork_s *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } diff --git a/arch/risc-v/src/common/CMakeLists.txt b/arch/risc-v/src/common/CMakeLists.txt index 84a1d43fb47c8..639cf94afef0d 100644 --- a/arch/risc-v/src/common/CMakeLists.txt +++ b/arch/risc-v/src/common/CMakeLists.txt @@ -86,7 +86,7 @@ if(CONFIG_STACK_COLORATION) list(APPEND SRCS riscv_checkstack.c) endif() -if(CONFIG_ARCH_HAVE_FORK) +if(CONFIG_ARCH_HAVE_VFORK OR CONFIG_ARCH_HAVE_FORK) list(APPEND SRCS fork.S riscv_fork.c) endif() diff --git a/arch/risc-v/src/common/Make.defs b/arch/risc-v/src/common/Make.defs index d98a828c508e0..4b46fc9ccad11 100644 --- a/arch/risc-v/src/common/Make.defs +++ b/arch/risc-v/src/common/Make.defs @@ -86,7 +86,7 @@ ifeq ($(CONFIG_STACK_COLORATION),y) CMN_CSRCS += riscv_checkstack.c endif -ifeq ($(CONFIG_ARCH_HAVE_FORK),y) +ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),) CMN_ASRCS += fork.S CMN_CSRCS += riscv_fork.c endif diff --git a/arch/risc-v/src/common/fork.S b/arch/risc-v/src/common/fork.S index 108ff00f80db1..a516e57fd96c7 100644 --- a/arch/risc-v/src/common/fork.S +++ b/arch/risc-v/src/common/fork.S @@ -46,46 +46,36 @@ ****************************************************************************/ /**************************************************************************** - * Name: fork + * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives. Both need exactly the same thing from assembly -- a + * snapshot of the caller's callee-saved registers, stack pointer and + * return address -- and differ only in what the C code then does with it, + * so there is one entry point and the caller's a0 says which primitive was + * called. It is passed straight through to riscv_fork(). * - * This thin layer implements fork by simply calling up_fork() with the - * fork() context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) riscv_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) riscv_fork() provides any additional operating context. riscv_fork must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) riscv_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point. + * 2) The entry point collects the context and calls riscv_fork(). + * 3) riscv_fork() calls nxtask_setup_fork(), which allocates and + * configures the child task's TCB. + * 4) riscv_fork() provides any additional operating context and relocates + * the copied stack. + * 5) riscv_fork() then calls nxtask_start_fork(), which for vfork() + * additionally suspends the caller. + * 6) which executes the child thread. * * Input Parameters: - * None + * a0 - true for vfork(), false for fork() * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. * ****************************************************************************/ @@ -94,7 +84,9 @@ up_fork: #ifdef CONFIG_LIB_SYSCALL - /* When coming via system call, everything is in place already */ + /* When coming via system call, everything is in place already: a0 already + * holds the vfork flag, and riscv_fork() takes its snapshot from the TCB. + */ tail riscv_fork #else @@ -129,8 +121,8 @@ up_fork: REGSTORE gp, FORK_GP_OFFSET(sp) #endif - addi a0, sp, FORK_SIZEOF - REGSTORE a0, FORK_SP_OFFSET(sp) /* original SP */ + addi a2, sp, FORK_SIZEOF + REGSTORE a2, FORK_SP_OFFSET(sp) /* original SP */ REGSTORE x1, FORK_RA_OFFSET(sp) /* return address */ /* Floating point registers */ @@ -150,9 +142,11 @@ up_fork: FSTORE fs11, FORK_FS11_OFFSET(sp) #endif - /* Then, call riscv_fork(), passing it a pointer to the stack frame */ + /* Then, call riscv_fork(). a0 still holds the vfork flag: nothing above + * touches it. + */ - mv a0, sp + mv a1, sp call riscv_fork /* Release the stack frame and return the value returned by riscv_fork */ diff --git a/arch/risc-v/src/common/riscv_fork.c b/arch/risc-v/src/common/riscv_fork.c index f25e5cf4e4062..107576297857c 100644 --- a/arch/risc-v/src/common/riscv_fork.c +++ b/arch/risc-v/src/common/riscv_fork.c @@ -41,8 +41,6 @@ #include "sched/sched.h" -#ifdef CONFIG_ARCH_HAVE_FORK - /**************************************************************************** * Pre-processor Definitions ****************************************************************************/ @@ -59,17 +57,15 @@ * Name: riscv_fork * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The common RISC-V worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. * * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up riscv_fork(). + * 1) User code calls vfork() or fork(). up_fork() collects context + * information and transfers control to riscv_fork(). * 2) riscv_fork() and calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: @@ -90,7 +86,8 @@ * and 6. * * Input Parameters: - * context - Caller context information saved by fork() + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -102,7 +99,7 @@ #ifdef CONFIG_LIB_SYSCALL -pid_t riscv_fork(const struct fork_s *context) +pid_t riscv_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -117,7 +114,7 @@ pid_t riscv_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)parent->xcp.sregs[REG_RA]); + child = nxtask_setup_fork((start_t)parent->xcp.sregs[REG_RA], vfork); if (!child) { sinfo("nxtask_setup_fork failed\n"); @@ -130,12 +127,26 @@ pid_t riscv_fork(const struct fork_s *context) DEBUGASSERT(stacktop > parent->xcp.sregs[REG_SP]); stackutil = stacktop - parent->xcp.sregs[REG_SP]; - /* Copy goes to child's user stack top */ + if (child->stack_base_ptr == parent->stack_base_ptr) + { + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate: + * every stack address the child inherits is still the address it + * names. + */ + + newsp = parent->xcp.sregs[REG_SP]; + } + else + { + /* Copy goes to child's user stack top */ - newtop = (uintptr_t)child->stack_base_ptr + child->adj_stack_size; - newsp = newtop - stackutil; + newtop = (uintptr_t)child->stack_base_ptr + child->adj_stack_size; + newsp = newtop - stackutil; - memcpy((void *)newsp, (const void *)parent->xcp.sregs[REG_SP], stackutil); + memcpy((void *)newsp, (const void *)parent->xcp.sregs[REG_SP], + stackutil); + } #ifdef CONFIG_SCHED_THREAD_LOCAL /* Save child's thread pointer */ @@ -184,12 +195,12 @@ pid_t riscv_fork(const struct fork_s *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } #else -pid_t riscv_fork(const struct fork_s *context) +pid_t riscv_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -215,7 +226,7 @@ pid_t riscv_fork(const struct fork_s *context) context->fp, context->sp, context->ra, context->gp); #else sinfo("fp:%" PRIxREG " sp:%" PRIxREG " ra:%" PRIxREG "\n", - context->fp context->sp, context->ra); + context->fp, context->sp, context->ra); #endif #else sinfo("s5:%" PRIxREG " s6:%" PRIxREG " s7:%" PRIxREG " s8:%" PRIxREG "\n", @@ -231,7 +242,7 @@ pid_t riscv_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)(uintptr_t)context->ra); + child = nxtask_setup_fork((start_t)(uintptr_t)context->ra, vfork); if (!child) { sinfo("nxtask_setup_fork failed\n"); @@ -252,52 +263,71 @@ pid_t riscv_fork(const struct fork_s *context) sinfo("Parent: stackutil:%" PRIxPTR "\n", stackutil); - /* Make some feeble effort to preserve the stack contents. This is - * feeble because the stack surely contains invalid pointers and other - * content that will not work in the child context. However, if the - * user follows all of the caveats of fork() usage, even this feeble - * effort is overkill. - */ + if (child->stack_base_ptr == parent->stack_base_ptr) + { + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate: + * every stack address the child inherits is still the address it + * names. + */ - newtop = (uintptr_t)child->stack_base_ptr + child->adj_stack_size; - newsp = newtop - stackutil; + newsp = (uintptr_t)context->sp; +#ifdef CONFIG_RISCV_FRAMEPOINTER + newfp = (uintptr_t)context->fp; +#endif + } + else + { + /* Make some feeble effort to preserve the stack contents. This is + * feeble because the stack surely contains invalid pointers and other + * content that will not work in the child context. However, if the + * user follows all of the caveats of vfork() usage, even this feeble + * effort is overkill. + * + * For a POSIX fork() child the stack contents are not merely a feeble + * effort: the child is entitled to use them, and it does. + */ - /* Set up frame for context and copy the initial context there */ + newtop = (uintptr_t)child->stack_base_ptr + child->adj_stack_size; + newsp = newtop - stackutil; - memcpy((void *)(newsp - XCPTCONTEXT_SIZE), - child->xcp.regs, XCPTCONTEXT_SIZE); + /* Set up frame for context and copy the initial context there */ - /* Copy the parent stack contents (overwrites child's SP and TP) */ + memcpy((void *)(newsp - XCPTCONTEXT_SIZE), + child->xcp.regs, XCPTCONTEXT_SIZE); - memcpy((void *)newsp, (const void *)(uintptr_t)context->sp, stackutil); + /* Copy the parent stack contents (overwrites child's SP and TP) */ - /* Set the new register restore area to the new stack top */ + memcpy((void *)newsp, (const void *)(uintptr_t)context->sp, stackutil); - child->xcp.regs = (void *)(newsp - XCPTCONTEXT_SIZE); + /* Set the new register restore area to the new stack top */ - /* Was there a frame pointer in place before? */ + child->xcp.regs = (void *)(newsp - XCPTCONTEXT_SIZE); -#ifdef CONFIG_RISCV_FRAMEPOINTER - if (context->fp >= context->sp && context->fp < stacktop) - { - uintptr_t frameutil = stacktop - context->fp; - newfp = newtop - frameutil; - } - else - { - newfp = context->fp; - } + /* Was there a frame pointer in place before? */ - sinfo("Old stack top:%" PRIxPTR " SP:%" PRIxREG " FP:%" PRIxREG "\n", - stacktop, context->sp, context->fp); - sinfo("New stack top:%" PRIxPTR " SP:%" PRIxPTR " FP:%" PRIxPTR "\n", - newtop, newsp, newfp); +#ifdef CONFIG_RISCV_FRAMEPOINTER + if (context->fp >= context->sp && context->fp < stacktop) + { + uintptr_t frameutil = stacktop - context->fp; + newfp = newtop - frameutil; + } + else + { + newfp = context->fp; + } + + sinfo("Old stack top:%" PRIxPTR " SP:%" PRIxREG " FP:%" PRIxREG "\n", + stacktop, context->sp, context->fp); + sinfo("New stack top:%" PRIxPTR " SP:%" PRIxPTR " FP:%" PRIxPTR "\n", + newtop, newsp, newfp); #else - sinfo("Old stack top:%" PRIxPTR " SP:%" PRIxREG "\n", - stacktop, context->sp); - sinfo("New stack top:%" PRIxPTR " SP:%" PRIxPTR "\n", - newtop, newsp); + sinfo("Old stack top:%" PRIxPTR " SP:%" PRIxREG "\n", + stacktop, context->sp); + sinfo("New stack top:%" PRIxPTR " SP:%" PRIxPTR "\n", + newtop, newsp); #endif + } /* Update the stack pointer, frame pointer, global pointer and saved * registers. When the child TCB was initialized, all of the values @@ -346,8 +376,7 @@ pid_t riscv_fork(const struct fork_s *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } #endif /* CONFIG_LIB_SYSCALL */ -#endif /* CONFIG_ARCH_HAVE_FORK */ diff --git a/arch/sim/src/Makefile b/arch/sim/src/Makefile index 280c3a8b05041..cc6ad23c1843d 100644 --- a/arch/sim/src/Makefile +++ b/arch/sim/src/Makefile @@ -95,7 +95,7 @@ ifeq ($(CONFIG_SCHED_BACKTRACE),y) CSRCS += sim_backtrace.c endif -ifeq ($(CONFIG_ARCH_HAVE_FORK),y) +ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),) CSRCS += sim_fork.c endif diff --git a/arch/sim/src/sim/CMakeLists.txt b/arch/sim/src/sim/CMakeLists.txt index a378448d1a324..046564e127432 100644 --- a/arch/sim/src/sim/CMakeLists.txt +++ b/arch/sim/src/sim/CMakeLists.txt @@ -82,7 +82,7 @@ if(CONFIG_SCHED_BACKTRACE) list(APPEND SRCS sim_backtrace.c) endif() -if(CONFIG_ARCH_HAVE_FORK) +if(CONFIG_ARCH_HAVE_VFORK OR CONFIG_ARCH_HAVE_FORK) list(APPEND SRCS sim_fork.c) endif() diff --git a/arch/sim/src/sim/sim_fork.c b/arch/sim/src/sim/sim_fork.c index 47ea655cbb65f..72ff6c98a7ef9 100644 --- a/arch/sim/src/sim/sim_fork.c +++ b/arch/sim/src/sim/sim_fork.c @@ -48,17 +48,15 @@ * Name: sim_fork * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The common simulator worker behind up_fork(). vfork() and fork() + * snapshot the caller's registers identically; `vfork' says which + * primitive was called, and is passed straight through to + * nxtask_setup_fork(), which is where the memory semantics are decided. * * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up sim_fork(). + * 1) User code calls vfork() or fork(). up_fork() collects context + * information and transfers control to sim_fork(). * 2) sim_fork() and calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: @@ -77,6 +75,10 @@ * nxtask_abort_fork() may be called if an error occurs between steps 3 and * 6. * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() + * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and * returns the process ID of the child process to the parent process. @@ -88,7 +90,7 @@ #ifdef CONFIG_SIM_ASAN nosanitize_address #endif -pid_t sim_fork(const xcpt_reg_t *context) +pid_t sim_fork(bool vfork, const xcpt_reg_t *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -106,7 +108,7 @@ pid_t sim_fork(const xcpt_reg_t *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)context[JB_PC]); + child = nxtask_setup_fork((start_t)context[JB_PC], vfork); if (!child) { serr("ERROR: nxtask_setup_fork failed\n"); @@ -175,5 +177,5 @@ pid_t sim_fork(const xcpt_reg_t *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } diff --git a/arch/sim/src/sim/sim_fork_arm.S b/arch/sim/src/sim/sim_fork_arm.S index ee39f54e003b5..8449c691bea3f 100644 --- a/arch/sim/src/sim/sim_fork_arm.S +++ b/arch/sim/src/sim/sim_fork_arm.S @@ -46,20 +46,20 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the behavior is - * undefined if the process created by fork() either modifies any data other than - * a variable of type pid_t used to store the return value from fork(), or returns - * from the function in which fork() was called, or calls any other function before - * successfully calling _exit() or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). The caller says which one it is, and + * the flag is passed straight through to sim_fork(). * - * This thin layer implements fork by simply calling up_fork() with the fork() - * context as an argument. The overall sequence is: + * On the simulator the caller's context is captured with setjmp() rather + * than by hand, and the child re-enters through longjmp() -- which is why + * the entry point tests setjmp()'s return value to tell which of the two + * returns it is on. * - * 1) User code calls fork(). fork() collects context information and - * transfers control up sim_fork(). - * 2) sim_fork() and calls nxtask_setup_fork(). + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls sim_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. This * consists of: * - Allocation of the child task's TCB. @@ -75,7 +75,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * r0 - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and returns @@ -86,18 +86,28 @@ ************************************************************************************/ .text + .globl up_fork .type up_fork, @function up_fork: + /* r4 is callee-saved, so it carries the vfork flag across setjmp() */ + + push {r4, lr} + mov r4, r0 + sub sp, sp, #XCPTCONTEXT_SIZE mov r0, sp bl setjmp subs r0, #1 - jz child + beq 1f + + mov r0, r4 + mov r1, sp bl sim_fork -child: +1: add sp, sp, #XCPTCONTEXT_SIZE - ret + pop {r4, lr} + bx lr .size up_fork, . - up_fork .end diff --git a/arch/sim/src/sim/sim_fork_arm64.S b/arch/sim/src/sim/sim_fork_arm64.S index 5d813822fa620..3a88939f6bf29 100644 --- a/arch/sim/src/sim/sim_fork_arm64.S +++ b/arch/sim/src/sim/sim_fork_arm64.S @@ -55,21 +55,20 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). The caller says which one it is, and + * the flag is passed straight through to sim_fork(). * - * This thin layer implements fork by simply calling sim_fork() with the - * fork() context as an argument. The overall sequence is: + * On the simulator the caller's context is captured with setjmp() rather + * than by hand, and the child re-enters through longjmp() -- which is why + * the entry point tests setjmp()'s return value to tell which of the two + * returns it is on. * - * 1) User code calls fork(). fork() collects context information and - * transfers control up sim_fork(). - * 2) sim_fork() and calls nxtask_setup_fork(). + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls sim_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: * - Allocation of the child task's TCB. @@ -85,7 +84,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * x0 - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -96,27 +95,35 @@ ***************************************************************************/ .text - .globl SYMBOL(up_fork) .align 4 + .globl SYMBOL(up_fork) + SYMBOL(up_fork): stp x29, x30, [sp] /* save FP/LR register */ - sub sp, sp, #XCPTCONTEXT_SIZE /* area from stack for setjmp() */ - mov x0, sp /* pass stack area to setjmp() */ + /* Area from stack for setjmp(), plus a slot below it holding the vfork + * flag: setjmp() is allowed to clobber every argument register, so the + * flag cannot simply stay in one. + */ + + sub sp, sp, #XCPTCONTEXT_SIZE+16 + str x0, [sp] + + add x0, sp, #16 /* pass stack area to setjmp() */ bl SYMBOL(setjmp) /* save register for longjmp() */ subs x0, x0, #1 /* 0: parent / 1: child */ cbz x0, 1f /* child --> return */ - mov x0, sp /* pass stack area to sim_fork() */ + ldr x0, [sp] /* the vfork flag */ + add x1, sp, #16 /* pass stack area to sim_fork() */ bl SYMBOL(sim_fork) /* further process task creation */ 1: - add sp, sp, #XCPTCONTEXT_SIZE /* release area from stack */ + add sp, sp, #XCPTCONTEXT_SIZE+16 /* release area from stack */ ldp x29, x30, [sp] /* restore FP/LR register */ ret - .end diff --git a/arch/sim/src/sim/sim_fork_x86.S b/arch/sim/src/sim/sim_fork_x86.S index ec7486664c568..127dd6b92d7de 100644 --- a/arch/sim/src/sim/sim_fork_x86.S +++ b/arch/sim/src/sim/sim_fork_x86.S @@ -54,20 +54,20 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the behavior is - * undefined if the process created by fork() either modifies any data other than - * a variable of type pid_t used to store the return value from fork(), or returns - * from the function in which fork() was called, or calls any other function before - * successfully calling _exit() or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). The caller says which one it is, and + * the flag is passed straight through to sim_fork(). * - * This thin layer implements fork by simply calling up_fork() with the fork() - * context as an argument. The overall sequence is: + * On the simulator the caller's context is captured with setjmp() rather + * than by hand, and the child re-enters through longjmp() -- which is why + * the entry point tests setjmp()'s return value to tell which of the two + * returns it is on. * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) sim_fork() and calls nxtask_setup_fork(). + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls sim_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. This * consists of: * - Allocation of the child task's TCB. @@ -83,7 +83,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * arg0 - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and returns @@ -94,21 +94,35 @@ ************************************************************************************/ .text + .globl SYMBOL(up_fork) #ifdef __ELF__ .type SYMBOL(up_fork), @function #endif SYMBOL(up_fork): + /* %ebx is callee-saved, so it carries the vfork flag across setjmp() */ + + push %ebx + mov 8(%esp), %ebx + sub $XCPTCONTEXT_SIZE, %esp push %esp call SYMBOL(setjmp) sub $1, %eax - jz child + jz 1f + + /* sim_fork(vfork, context). The context pointer pushed for setjmp() + * is still in place, and is the second argument. + */ + + push %ebx call SYMBOL(sim_fork) -child: + add $4, %esp +1: add $XCPTCONTEXT_SIZE+4, %esp + pop %ebx ret #ifdef __ELF__ .size SYMBOL(up_fork), . - SYMBOL(up_fork) diff --git a/arch/sim/src/sim/sim_fork_x86_64.S b/arch/sim/src/sim/sim_fork_x86_64.S index 85b072acf866e..4d1e6412ab5cf 100644 --- a/arch/sim/src/sim/sim_fork_x86_64.S +++ b/arch/sim/src/sim/sim_fork_x86_64.S @@ -54,20 +54,20 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the behavior is - * undefined if the process created by fork() either modifies any data other than - * a variable of type pid_t used to store the return value from fork(), or returns - * from the function in which fork() was called, or calls any other function before - * successfully calling _exit() or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). The caller says which one it is, and + * the flag is passed straight through to sim_fork(). * - * This thin layer implements fork by simply calling up_fork() with the fork() - * context as an argument. The overall sequence is: + * On the simulator the caller's context is captured with setjmp() rather + * than by hand, and the child re-enters through longjmp() -- which is why + * the entry point tests setjmp()'s return value to tell which of the two + * returns it is on. * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) sim_fork() and calls nxtask_setup_fork(). + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls sim_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. This * consists of: * - Allocation of the child task's TCB. @@ -83,7 +83,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * arg0 - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and returns @@ -94,26 +94,41 @@ ************************************************************************************/ .text + .globl SYMBOL(up_fork) #ifdef __ELF__ .type SYMBOL(up_fork), @function #endif SYMBOL(up_fork): + push %rbx sub $XCPTCONTEXT_SIZE, %rsp + + /* %rbx is callee-saved, so it carries the vfork flag across setjmp() */ + #ifdef CONFIG_SIM_X8664_MICROSOFT + mov %rcx, %rbx mov %rsp, %rcx #else /* if defined(CONFIG_SIM_X8664_SYSTEMV) */ + mov %rdi, %rbx mov %rsp, %rdi #endif call SYMBOL(setjmp) sub $1, %eax - jz child + jz 1f +#ifdef CONFIG_SIM_X8664_MICROSOFT + mov %rbx, %rcx + mov %rsp, %rdx +#else /* if defined(CONFIG_SIM_X8664_SYSTEMV) */ + mov %rbx, %rdi + mov %rsp, %rsi +#endif call SYMBOL(sim_fork) -child: +1: add $XCPTCONTEXT_SIZE, %rsp + pop %rbx ret #ifdef __ELF__ .size SYMBOL(up_fork), . - SYMBOL(up_fork) diff --git a/arch/x86_64/src/common/CMakeLists.txt b/arch/x86_64/src/common/CMakeLists.txt index 841fa13679935..de170a4d73945 100644 --- a/arch/x86_64/src/common/CMakeLists.txt +++ b/arch/x86_64/src/common/CMakeLists.txt @@ -34,7 +34,7 @@ set(SRCS x86_64_tcbinfo.c x86_64_tlb.c) -if(CONFIG_ARCH_HAVE_FORK) +if(CONFIG_ARCH_HAVE_VFORK OR CONFIG_ARCH_HAVE_FORK) list(APPEND SRCS x86_64_fork.c fork.S) endif() diff --git a/arch/x86_64/src/common/Make.defs b/arch/x86_64/src/common/Make.defs index a5a21aff0a63a..e72813edcdabc 100644 --- a/arch/x86_64/src/common/Make.defs +++ b/arch/x86_64/src/common/Make.defs @@ -29,7 +29,7 @@ CMN_CSRCS += x86_64_getintstack.c x86_64_initialize.c x86_64_nputs.c CMN_CSRCS += x86_64_modifyreg8.c x86_64_modifyreg16.c x86_64_modifyreg32.c CMN_CSRCS += x86_64_switchcontext.c x86_64_tlb.c -ifeq ($(CONFIG_ARCH_HAVE_FORK),y) +ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),) CMN_CSRCS += x86_64_fork.c CMN_ASRCS += fork.S endif diff --git a/arch/x86_64/src/common/fork.S b/arch/x86_64/src/common/fork.S index 1621b560474a1..a1e5396d04090 100644 --- a/arch/x86_64/src/common/fork.S +++ b/arch/x86_64/src/common/fork.S @@ -39,21 +39,18 @@ * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The architecture-specific entry point of both of NuttX's cloning + * primitives, vfork() and fork(). Both need exactly the same thing from + * assembly -- a snapshot of the caller's registers, stack pointer and + * return address -- and differ only in what the C code then does with it, + * so there is one entry point and the caller's %rdi says which primitive + * was called. It is passed straight through to x86_64_fork(). * - * This thin layer implements fork by simply calling up_fork() with the - * fork() context as an argument. The overall sequence is: + * The overall sequence is: * - * 1) User code calls fork(). fork() collects context information and - * transfers control up up_fork(). - * 2) x86_64_fork() and calls nxtask_setup_fork(). + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * this entry point, which collects context information and + * 2) calls x86_64_fork(), which calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: * - Allocation of the child task's TCB. @@ -69,7 +66,7 @@ * 6) nxtask_start_fork() then executes the child thread. * * Input Parameters: - * None + * %rdi - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -91,7 +88,7 @@ * | [r15] | i * | [r14] | n * | [r13] | g - * | [r12] | <- rsp before calling x86_64_fork, rdi = rsp + * | [r12] | <- rsp before calling x86_64_fork, rsi = rsp * | ......... | */ @@ -99,13 +96,17 @@ .type up_fork, @function up_fork: + /* %rdi still holds the vfork flag on entry and is left alone; %rdx is + * the scratch used to push %ss and %cs. + */ + movq %rsp, %rax addq $8, %rax - movq %ss, %rdi - pushq %rdi + movq %ss, %rdx + pushq %rdx pushfq - movq %cs, %rdi - pushq %rdi + movq %cs, %rdx + pushq %rdx /* push %rsp */ @@ -116,7 +117,7 @@ up_fork: pushq %r14 pushq %r13 pushq %r12 - movq %rsp, %rdi + movq %rsp, %rsi subq $8, %rsp diff --git a/arch/x86_64/src/common/x86_64_fork.c b/arch/x86_64/src/common/x86_64_fork.c index ea4d90ac044b4..05cd26016e8f0 100644 --- a/arch/x86_64/src/common/x86_64_fork.c +++ b/arch/x86_64/src/common/x86_64_fork.c @@ -79,7 +79,8 @@ * 6. * * Input Parameters: - * context - Caller context information saved by fork() + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() * * Returned Value: * Upon successful completion, fork() returns 0 to the child process and @@ -89,7 +90,7 @@ * ****************************************************************************/ -pid_t x86_64_fork(const struct fork_s *context) +pid_t x86_64_fork(bool vfork, const struct fork_s *context) { struct tcb_s *parent = this_task(); struct tcb_s *child; @@ -110,7 +111,7 @@ pid_t x86_64_fork(const struct fork_s *context) /* Allocate and initialize a TCB for the child task. */ - child = nxtask_setup_fork((start_t)context->rip); + child = nxtask_setup_fork((start_t)context->rip, vfork); if (!child) { serr("ERROR: nxtask_setup_fork failed\n"); @@ -133,44 +134,59 @@ pid_t x86_64_fork(const struct fork_s *context) sinfo("Parent: stackutil:%" PRIu64 "\n", stackutil); - /* Make some feeble effort to preserve the stack contents. This is - * feeble because the stack surely contains invalid pointers and other - * content that will not work in the child context. However, if the - * user follows all of the caveats of fork() usage, even this feeble - * effort is overkill. - */ - - newtop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)child->stack_base_ptr + - child->adj_stack_size - - XCPTCONTEXT_SIZE); - - newsp = newtop - stackutil; - - /* Move the register context (from parent) to newtop. */ + /* Move the register context (from parent) to the child. */ memcpy(child->xcp.regs, parent->xcp.regs, XCPTCONTEXT_SIZE); - memcpy((void *)newsp, (const void *)context->rsp, stackutil); - - /* Was there a frame pointer in place before? */ - - if (context->rbp >= context->rsp && context->rbp < stacktop) + if (child->stack_base_ptr == parent->stack_base_ptr) { - uint32_t frameutil = stacktop - context->rbp; - newfp = newtop - frameutil; + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate: + * every stack address the child inherits is still the address it + * names. + */ + + newsp = context->rsp; + newfp = context->rbp; } else { - newfp = context->rbp; + /* Make some feeble effort to preserve the stack contents. This is + * feeble because the stack surely contains invalid pointers and other + * content that will not work in the child context. However, if the + * user follows all of the caveats of vfork() usage, even this feeble + * effort is overkill. + * + * For a POSIX fork() child the stack contents are not merely a feeble + * effort: the child is entitled to use them, and it does. + */ + + newtop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)child->stack_base_ptr + + child->adj_stack_size - + XCPTCONTEXT_SIZE); + + newsp = newtop - stackutil; + + memcpy((void *)newsp, (const void *)context->rsp, stackutil); + + /* Was there a frame pointer in place before? */ + + if (context->rbp >= context->rsp && context->rbp < stacktop) + { + uint32_t frameutil = stacktop - context->rbp; + newfp = newtop - frameutil; + } + else + { + newfp = context->rbp; + } + + sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 + " RBP:%08" PRIx64 "\n", stacktop, context->rsp, context->rbp); + sinfo("New stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", + newtop, newsp); } - /* We do not need to update the frame-pointer */ - - sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 " RBP:%08" PRIx64 "\n", - stacktop, context->rsp, context->rbp); - sinfo("New stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", - newtop, newsp); - /* Update the stack pointer, frame pointer, and volatile registers. When * the child TCB was initialized, all of the values were set to zero. * up_initial_state() altered a few values, but the return value in RAX @@ -195,5 +211,5 @@ pid_t x86_64_fork(const struct fork_s *context) * will discard the TCB by calling nxtask_abort_fork(). */ - return nxtask_start_fork(child); + return nxtask_start_fork(child, vfork); } diff --git a/include/nuttx/addrenv.h b/include/nuttx/addrenv.h index 8e253c88eee1b..91c2eaa28d4d2 100644 --- a/include/nuttx/addrenv.h +++ b/include/nuttx/addrenv.h @@ -394,6 +394,31 @@ int addrenv_attach(FAR struct tcb_s *tcb, FAR struct addrenv_s *addrenv); int addrenv_join(FAR struct tcb_s *ptcb, FAR struct tcb_s *tcb); +/**************************************************************************** + * Name: addrenv_fork + * + * Description: + * Duplicate the parent's address environment for a POSIX fork() child and + * attach it: the child gets its own pages holding a copy of the parent's + * contents, mapped at the same virtual addresses. Contrast + * addrenv_join(), which gives the child the parent's memory. + * + * Input Parameters: + * ptcb - The tcb of the parent process. + * tcb - The tcb of the child process. + * + * Returned Value: + * This is a NuttX internal function so it follows the convention that + * 0 (OK) is returned on success and a negated errno is returned on + * failure. -ENOMEM is returned if there is not enough free memory to + * hold a copy of the parent. + * + ****************************************************************************/ + +#ifdef CONFIG_ARCH_HAVE_FORK +int addrenv_fork(FAR struct tcb_s *ptcb, FAR struct tcb_s *tcb); +#endif + /**************************************************************************** * Name: addrenv_leave * diff --git a/include/nuttx/arch.h b/include/nuttx/arch.h index e1a598c222b2d..91492fefe9c54 100644 --- a/include/nuttx/arch.h +++ b/include/nuttx/arch.h @@ -253,8 +253,18 @@ extern initializer_t _einit[]; * Name: up_fork * * Description: - * The up_fork() function is the base of fork() function that provided in - * libc, and fork() is implemented as a wrapper of up_fork() function. + * Architecture-specific base of both cloning primitives. It snapshots the + * caller's registers and hands them to the common code, which builds the + * child from them; `vfork' says which primitive was called and so which + * memory semantics the child gets. + * + * Input Parameters: + * vfork - true for vfork(): the child shares the parent's memory and the + * parent is suspended until the child _exit()s or exec()s. + * false for POSIX fork(): the child receives its own copy of the + * parent's memory at the same virtual addresses and runs + * concurrently. Only available where CONFIG_ARCH_HAVE_FORK is + * selected. * * Returned Value: * Upon successful completion, up_fork() returns 0 to the child process @@ -264,7 +274,9 @@ extern initializer_t _einit[]; * ****************************************************************************/ -pid_t up_fork(void); +#if defined(CONFIG_ARCH_HAVE_VFORK) || defined(CONFIG_ARCH_HAVE_FORK) +pid_t up_fork(bool vfork); +#endif /**************************************************************************** * Name: up_initialize @@ -1327,6 +1339,35 @@ int up_addrenv_clone(FAR const arch_addrenv_t *src, FAR arch_addrenv_t *dest); #endif +/**************************************************************************** + * Name: up_addrenv_fork + * + * Description: + * Duplicate an address environment for POSIX fork(): allocate fresh + * pages for the destination, copy the source's contents into them, and map + * them at the same virtual addresses. Unlike up_addrenv_clone(), which + * copies only the representation and leaves both pointing at the same page + * tables, the result is independent of the source. + * + * Implemented only where CONFIG_ARCH_HAVE_FORK is selected. + * + * Input Parameters: + * src - The address environment to be duplicated. + * dest - The location to receive the duplicate. It is wiped by this + * function before anything is allocated into it. + * + * Returned Value: + * Zero (OK) on success; a negated errno value on failure. -ENOMEM is + * returned if there are not enough free pages to hold the copy, in which + * case nothing is left allocated. + * + ****************************************************************************/ + +#ifdef CONFIG_ARCH_HAVE_FORK +int up_addrenv_fork(FAR const arch_addrenv_t *src, + FAR arch_addrenv_t *dest); +#endif + /**************************************************************************** * Name: up_addrenv_attach * diff --git a/include/nuttx/sched.h b/include/nuttx/sched.h index 76630d3f10110..9778282fecd94 100644 --- a/include/nuttx/sched.h +++ b/include/nuttx/sched.h @@ -650,6 +650,14 @@ struct tcb_s /* after the frame has been */ /* removed from the stack. */ + /* vfork() Support ********************************************************/ + +#ifdef CONFIG_ARCH_HAVE_VFORK + FAR sem_t *vfork_rel; /* Non-NULL in a vfork() child: */ + /* the suspended parent to release */ + /* when this task is torn down. */ +#endif + /* External Module Support ************************************************/ #ifdef CONFIG_PIC @@ -1134,23 +1142,35 @@ void nxtask_startup(main_t entrypt, int argc, FAR char *argv[]); #endif /**************************************************************************** - * Internal fork support. The overall sequence is: - * - * 1) User code calls fork(). fork() is provided in architecture-specific - * code. - * 2) fork()and calls nxtask_setup_fork(). + * Internal support for the two cloning primitives, vfork() and fork(). The + * sequence below is common to both, and `vfork' says which one was called: + * + * vfork() the child shares the parent's memory and the parent is + * suspended until the child _exit()s or exec()s. + * fork() the child gets its own copy of the parent's memory at the same + * virtual addresses, and both run. + * + * 1) User code calls vfork() or fork(). Both are libc wrappers around + * up_fork(), which is provided in architecture-specific code. + * 2) The architecture-specific code snapshots the caller's registers and + * calls nxtask_setup_fork(). * 3) nxtask_setup_fork() allocates and configures the child task's TCB. * This consists of: * - Allocation of the child task's TCB. * - Initialization of file descriptors and streams * - Configuration of environment variables - * - Allocate and initialize the stack + * - Establishing the child's address environment: joined to the parent's + * for vfork(), duplicated from it for fork() + * - Allocating the stack, or inheriting the parent's for fork() * - Setup the input parameters for the task. * - Initialization of the TCB (including call to up_initial_state()) - * 4) fork() provides any additional operating context. fork must: + * 4) The architecture-specific code provides any additional operating + * context: * - Initialize special values in any CPU registers that were not * already configured by up_initial_state() - * 5) fork() then calls nxtask_start_fork() + * - Relocate the copied stack, unless the child shares the parent's + * 5) It then calls nxtask_start_fork(), which for vfork() additionally + * suspends the caller. * 6) nxtask_start_fork() then executes the child thread. * * nxtask_abort_fork() may be called if an error occurs between @@ -1158,8 +1178,8 @@ void nxtask_startup(main_t entrypt, int argc, FAR char *argv[]); * ****************************************************************************/ -FAR struct tcb_s *nxtask_setup_fork(start_t retaddr); -pid_t nxtask_start_fork(FAR struct tcb_s *child); +FAR struct tcb_s *nxtask_setup_fork(start_t retaddr, bool vfork); +pid_t nxtask_start_fork(FAR struct tcb_s *child, bool vfork); void nxtask_abort_fork(FAR struct tcb_s *child, int errcode); /**************************************************************************** diff --git a/include/sys/syscall_lookup.h b/include/sys/syscall_lookup.h index 97b1cee5b0532..dc85c2d76771a 100644 --- a/include/sys/syscall_lookup.h +++ b/include/sys/syscall_lookup.h @@ -116,8 +116,8 @@ SYSCALL_LOOKUP(nxsem_wait_slow, 1) /* The following can be individually enabled */ -#ifdef CONFIG_ARCH_HAVE_FORK - SYSCALL_LOOKUP(up_fork, 0) +#if defined(CONFIG_ARCH_HAVE_VFORK) || defined(CONFIG_ARCH_HAVE_FORK) + SYSCALL_LOOKUP(up_fork, 1) #endif #ifdef CONFIG_SCHED_WAITPID diff --git a/include/unistd.h b/include/unistd.h index e9d1686248c9f..885bbed12a532 100644 --- a/include/unistd.h +++ b/include/unistd.h @@ -347,8 +347,17 @@ extern "C" /* Task Control Interfaces */ +/* fork() is declared only where POSIX fork() semantics can be provided, so + * that calling it elsewhere is a build error rather than a silent change of + * meaning. + */ + +#ifdef CONFIG_ARCH_HAVE_FORK pid_t fork(void); +#endif +#ifdef CONFIG_ARCH_HAVE_VFORK pid_t vfork(void); +#endif pid_t getpid(void); pid_t getpgid(pid_t pid); pid_t getpgrp(void); diff --git a/libs/libbuiltin/libgcc/gcov.c b/libs/libbuiltin/libgcc/gcov.c index c73aa620b2301..134b6aa79ac3c 100644 --- a/libs/libbuiltin/libgcc/gcov.c +++ b/libs/libbuiltin/libgcc/gcov.c @@ -468,10 +468,16 @@ void __gcov_execle(void) { } +/* GCC redirects fork() in instrumented code to __gcov_fork(), so this is + * reachable only where unistd.h declares fork() at all. + */ + +#ifdef CONFIG_ARCH_HAVE_FORK pid_t __gcov_fork(void) { return fork(); } +#endif void __gcov_dump(void) { diff --git a/libs/libc/libc.csv b/libs/libc/libc.csv index 4600f5329f96a..2e2d13506cfc9 100644 --- a/libs/libc/libc.csv +++ b/libs/libc/libc.csv @@ -348,6 +348,7 @@ "usleep","unistd.h","","int","useconds_t" "vasprintf","stdio.h","","int","FAR char **","FAR const IPTR char *","va_list" "versionsort","dirent.h","","int","FAR const struct dirent **","FAR const struct dirent **" +"vfork","unistd.h","!defined(CONFIG_BUILD_KERNEL) && defined(CONFIG_ARCH_HAVE_VFORK)","pid_t" "vfprintf","stdio.h","defined(CONFIG_FILE_STREAM)","int","FAR FILE *","FAR const IPTR char *","va_list" "vprintf","stdio.h","","int","FAR const IPTR char *","va_list" "vscanf","stdio.h","defined(CONFIG_FILE_STREAM)","int","FAR const IPTR char *","va_list" diff --git a/libs/libc/unistd/CMakeLists.txt b/libs/libc/unistd/CMakeLists.txt index 3521d45f607eb..3bcc3a6004e49 100644 --- a/libs/libc/unistd/CMakeLists.txt +++ b/libs/libc/unistd/CMakeLists.txt @@ -104,7 +104,7 @@ if(NOT CONFIG_DISABLE_MOUNTPOINTS) list(APPEND SRCS lib_truncate.c lib_posix_fallocate.c) endif() -if(CONFIG_ARCH_HAVE_FORK) +if(CONFIG_ARCH_HAVE_VFORK OR CONFIG_ARCH_HAVE_FORK) list(APPEND SRCS lib_fork.c) endif() diff --git a/libs/libc/unistd/Make.defs b/libs/libc/unistd/Make.defs index 602887be4c33a..a2ea08697a0e2 100644 --- a/libs/libc/unistd/Make.defs +++ b/libs/libc/unistd/Make.defs @@ -54,7 +54,7 @@ ifneq ($(CONFIG_DISABLE_MOUNTPOINTS),y) CSRCS += lib_truncate.c lib_posix_fallocate.c endif -ifeq ($(CONFIG_ARCH_HAVE_FORK),y) +ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),) CSRCS += lib_fork.c endif diff --git a/libs/libc/unistd/lib_fork.c b/libs/libc/unistd/lib_fork.c index 53c2bfe0e9b2c..84813db3265b5 100644 --- a/libs/libc/unistd/lib_fork.c +++ b/libs/libc/unistd/lib_fork.c @@ -34,8 +34,6 @@ #include #include -#if defined(CONFIG_ARCH_HAVE_FORK) - /**************************************************************************** * Private Functions ****************************************************************************/ @@ -137,27 +135,38 @@ static void atfork_parent(void) ****************************************************************************/ /**************************************************************************** - * Name: fork + * Name: vfork * * Description: - * The fork() function is a wrapper of up_fork() syscall + * The vfork() function is equivalent to fork(), except that the behavior + * is undefined if the process created by vfork() either modifies any data + * other than a variable of type pid_t used to store the return value from + * vfork(), or returns from the function in which vfork() was called, or + * calls any other function before successfully calling _exit() or one of + * the exec family of functions. + * + * The child shares the parent's memory and the parent is suspended until + * the child _exit()s or exec()s. The suspension lives in the kernel, so + * vfork() does not depend on CONFIG_SCHED_WAITPID. Wrapper of the + * up_fork() syscall. * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and + * Upon successful completion, vfork() returns 0 to the child process and * returns the process ID of the child process to the parent process. * Otherwise, -1 is returned to the parent, no child process is created, * and errno is set to indicate the error. * ****************************************************************************/ -pid_t fork(void) +#ifdef CONFIG_ARCH_HAVE_VFORK +pid_t vfork(void) { pid_t pid; #ifdef CONFIG_PTHREAD_ATFORK atfork_prepare(); #endif - pid = up_fork(); + pid = up_fork(true); #ifdef CONFIG_PTHREAD_ATFORK if (pid == 0) @@ -172,39 +181,38 @@ pid_t fork(void) return pid; } - -#if defined(CONFIG_SCHED_WAITPID) +#endif /* CONFIG_ARCH_HAVE_VFORK */ /**************************************************************************** - * Public Functions - ****************************************************************************/ - -/**************************************************************************** - * Name: vfork + * Name: fork * * Description: - * The vfork() function is implemented based on fork() function, on - * vfork(), the parent task need to wait until the child task is performing - * exec or running finished. + * POSIX fork(). The child receives its own copy of the parent's memory, + * at the same virtual addresses. It may modify anything, call anything, + * return from the function that called fork(), and it runs concurrently + * with the parent. None of vfork()'s restrictions apply. + * + * Provided only where CONFIG_ARCH_HAVE_FORK is selected; elsewhere fork() + * is not declared at all, so calling it is a build error. Wrapper of the + * up_fork() syscall. * * Returned Value: - * Upon successful completion, vfork() returns 0 to the child process and + * Upon successful completion, fork() returns 0 to the child process and * returns the process ID of the child process to the parent process. * Otherwise, -1 is returned to the parent, no child process is created, * and errno is set to indicate the error. * ****************************************************************************/ -pid_t vfork(void) +#ifdef CONFIG_ARCH_HAVE_FORK +pid_t fork(void) { - int status = 0; - int ret; pid_t pid; #ifdef CONFIG_PTHREAD_ATFORK atfork_prepare(); #endif - pid = up_fork(); + pid = up_fork(false); #ifdef CONFIG_PTHREAD_ATFORK if (pid == 0) @@ -217,22 +225,6 @@ pid_t vfork(void) } #endif - if (pid != 0) - { - /* we are in parent task, and we need to wait the child task - * until running finished or performing exec - */ - - ret = waitpid(pid, &status, WNOWAIT); - if (ret < 0) - { - serr("ERROR: waitpid failed: %d\n", get_errno()); - } - } - return pid; } - -#endif /* CONFIG_SCHED_WAITPID */ - #endif /* CONFIG_ARCH_HAVE_FORK */ diff --git a/sched/addrenv/addrenv.c b/sched/addrenv/addrenv.c index 9936a105c774c..32e6cba247c67 100644 --- a/sched/addrenv/addrenv.c +++ b/sched/addrenv/addrenv.c @@ -27,6 +27,7 @@ #include #include +#include #include #include @@ -292,6 +293,70 @@ int addrenv_join(FAR struct tcb_s *ptcb, FAR struct tcb_s *tcb) return OK; } +#ifdef CONFIG_ARCH_HAVE_FORK +/**************************************************************************** + * Name: addrenv_fork + * + * Description: + * Duplicate the parent process's address environment for a POSIX fork() + * child, and attach the duplicate to the child. + * + * This is the counterpart of addrenv_join(): where join gives the child + * the parent's memory, fork gives it a copy -- its own pages, holding a + * snapshot of the parent's contents, mapped at the same virtual addresses. + * Mapping at the same addresses is what lets the copy be exact: every + * pointer the parent held into its own memory remains valid in the child, + * including the pointers inside the copied heap's own metadata. + * + * The copy is eager -- there is no copy-on-write, because NuttX has no + * demand paging to build it on -- so forking a large process needs as much + * free memory as the process occupies, and fails with -ENOMEM if that is + * not available. That is the nature of the primitive on this class of + * system, not a defect of this implementation; spawn-heavy code should + * prefer posix_spawn() or vfork(). + * + * Input Parameters: + * ptcb - The tcb of the parent process + * tcb - The tcb of the child process + * + * Returned Value: + * This is a NuttX internal function so it follows the convention that + * 0 (OK) is returned on success and a negated errno is returned on + * failure. + * + ****************************************************************************/ + +int addrenv_fork(FAR struct tcb_s *ptcb, FAR struct tcb_s *tcb) +{ + FAR struct addrenv_s *addrenv; + int ret; + + DEBUGASSERT(ptcb->addrenv_own != NULL); + + addrenv = addrenv_allocate(); + if (addrenv == NULL) + { + return -ENOMEM; + } + + /* Duplicate the parent's regions into freshly allocated pages, mapped at + * the same virtual addresses. + */ + + ret = up_addrenv_fork(&ptcb->addrenv_own->addrenv, &addrenv->addrenv); + if (ret < 0) + { + berr("ERROR: up_addrenv_fork failed: %d\n", ret); + addrenv_drop(addrenv, false); + return ret; + } + + /* Hand the reference taken by addrenv_allocate() to the child */ + + return addrenv_attach(tcb, addrenv); +} +#endif /* CONFIG_ARCH_HAVE_FORK */ + /**************************************************************************** * Name: addrenv_leave * diff --git a/sched/sched/sched.h b/sched/sched/sched.h index 8c3aff1d4265d..b06cd97cbc2cd 100644 --- a/sched/sched/sched.h +++ b/sched/sched/sched.h @@ -319,6 +319,16 @@ void nxsched_remove_self(FAR struct tcb_s *rtrtcb); void nxsched_add_blocked(FAR struct tcb_s *btcb, tstate_t task_state); void nxsched_remove_blocked(FAR struct tcb_s *btcb); int nxsched_set_priority(FAR struct tcb_s *tcb, int sched_priority); + +/* Release the vfork() parent suspended on this child, if there is one. + * Called from nxsched_release_tcb(), the last point in the child's life -- + * by which time an exec()ing child has already handed its pid to the + * program it loaded. + */ + +#ifdef CONFIG_ARCH_HAVE_VFORK +void nxtask_resume_vfork(FAR struct tcb_s *child); +#endif #ifndef CONFIG_SMP bool nxsched_merge_pending(void); bool nxsched_reprioritize_rtr(FAR struct tcb_s *tcb, int priority); diff --git a/sched/sched/sched_releasetcb.c b/sched/sched/sched_releasetcb.c index 96b3e736e79cd..1f9d8fe1293f8 100644 --- a/sched/sched/sched_releasetcb.c +++ b/sched/sched/sched_releasetcb.c @@ -174,6 +174,15 @@ int nxsched_release_tcb(FAR struct tcb_s *tcb, uint8_t ttype) nxtask_joindestroy(tcb); #endif +#ifdef CONFIG_ARCH_HAVE_VFORK + /* Release a suspended vfork() parent here, the last point in the + * child's life: exec_swap() has already handed its pid to any + * program it loaded. + */ + + nxtask_resume_vfork(tcb); +#endif + /* And, finally, release the TCB itself */ if (tcb->flags & TCB_FLAG_FREE_TCB) diff --git a/sched/task/CMakeLists.txt b/sched/task/CMakeLists.txt index fdc19fdc589b1..7f6817e3dfcd9 100644 --- a/sched/task/CMakeLists.txt +++ b/sched/task/CMakeLists.txt @@ -46,7 +46,7 @@ if(CONFIG_SCHED_HAVE_PARENT) list(APPEND SRCS task_getppid.c task_reparent.c) endif() -if(CONFIG_ARCH_HAVE_FORK) +if(CONFIG_ARCH_HAVE_VFORK OR CONFIG_ARCH_HAVE_FORK) list(APPEND SRCS task_fork.c) endif() diff --git a/sched/task/Make.defs b/sched/task/Make.defs index 1fd24403b51ae..6c5857f63017e 100644 --- a/sched/task/Make.defs +++ b/sched/task/Make.defs @@ -30,7 +30,7 @@ ifeq ($(CONFIG_SCHED_HAVE_PARENT),y) CSRCS += task_getppid.c task_reparent.c endif -ifeq ($(CONFIG_ARCH_HAVE_FORK),y) +ifneq ($(CONFIG_ARCH_HAVE_VFORK)$(CONFIG_ARCH_HAVE_FORK),) CSRCS += task_fork.c endif diff --git a/sched/task/task_exit.c b/sched/task/task_exit.c index e79c4f812504f..892adfcfda942 100644 --- a/sched/task/task_exit.c +++ b/sched/task/task_exit.c @@ -157,5 +157,19 @@ int nxtask_exit(void) rtcb->lockcount--; + /* Publish anything woken while the TCB was being released. lockcount was + * raised directly rather than through sched_lock(), so the matching + * decrement above does not publish the way sched_unlock() would, and a + * vfork() parent released by nxsched_release_tcb() would be stranded -- + * in g_pendingtasks, or in g_readytorun on SMP. This mirrors what + * sched_unlock() does for each case. + */ + +#ifdef CONFIG_SMP + nxsched_deliver_task(this_cpu(), rtcb->cpu, SWITCH_HIGHER); +#else + nxsched_merge_pending(); +#endif + return ret; } diff --git a/sched/task/task_fork.c b/sched/task/task_fork.c index 5e2b133744264..ca7ff728636aa 100644 --- a/sched/task/task_fork.c +++ b/sched/task/task_fork.c @@ -34,7 +34,9 @@ #include #include +#include #include +#include #include "sched/sched.h" #include "environ/environ.h" @@ -42,9 +44,101 @@ #include "task/task.h" #include "tls/tls.h" -/* fork() requires architecture-specific support as well as waipid(). */ +/**************************************************************************** + * Private Function Prototypes + ****************************************************************************/ -#ifdef CONFIG_ARCH_HAVE_FORK +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) +static void fork_inherit_stack(FAR struct tcb_s *parent, + FAR struct tcb_s *child); +static void fork_inherit_tls(FAR struct tcb_s *child); +static void fork_restore_parent_env(void); +#endif + +/**************************************************************************** + * Private Functions + ****************************************************************************/ + +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) +/**************************************************************************** + * Name: fork_inherit_stack + * + * Description: + * Give the fork() child the parent's stack at the parent's virtual + * address rather than a relocated copy. The child's address environment + * is a duplicate, so the parent's stack is already there -- same contents, + * same address, its own pages -- and nothing needs allocating or copying. + * + * A relocated stack would break plain C: a pointer to a local taken + * before the fork would name the parent's copy, not the child's live + * object. + * + * TCB_FLAG_FREE_STACK is left clear: the stack belongs to the duplicated + * image and is released with it, so up_release_stack() must not free it. + * + * Input Parameters: + * parent - The parent task's TCB + * child - The child task's TCB + * + ****************************************************************************/ + +static void fork_inherit_stack(FAR struct tcb_s *parent, + FAR struct tcb_s *child) +{ + child->stack_alloc_ptr = parent->stack_alloc_ptr; + child->stack_base_ptr = parent->stack_base_ptr; + child->adj_stack_size = parent->adj_stack_size; + child->flags &= ~TCB_FLAG_FREE_STACK; +} + +/**************************************************************************** + * Name: fork_inherit_tls + * + * Description: + * Retarget the thread-local storage the fork() child inherited. + * + * tls_dup_info() cannot be used: it carves a fresh TLS block off the + * stack, which on an inherited stack would carve a second one and shift + * stack_base_ptr away from the parent's. The child's copy is already in + * place, so only the fields naming the task itself need correcting. + * + * The write lands in user memory at an address the parent also occupies, + * so the child's address environment must be current for it -- otherwise + * the parent's own TLS is what gets modified. + * + * Input Parameters: + * child - The child task's TCB + * + * Returned Value: + * Zero (OK) on success; a negated errno value on failure. + * + ****************************************************************************/ + +static void fork_inherit_tls(FAR struct tcb_s *child) +{ + FAR struct tls_info_s *info = (FAR struct tls_info_s *) + child->stack_alloc_ptr; + + info->tl_task = child->group->tg_info; + info->tl_tid = child->pid; +} + +/**************************************************************************** + * Name: fork_restore_parent_env + * + * Description: + * Undo the addrenv_select() that nxtask_setup_fork() made on the child's + * behalf, putting the caller back in its own address environment. The + * environment to go back to does not have to be remembered: the caller is + * the parent, and what was current before was the parent's own. + * + ****************************************************************************/ + +static void fork_restore_parent_env(void) +{ + addrenv_restore(this_task()->addrenv_own); +} +#endif /* CONFIG_ARCH_ADDRENV && CONFIG_ARCH_HAVE_FORK */ /**************************************************************************** * Public Functions @@ -54,37 +148,22 @@ * Name: nxtask_setup_fork * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. - * - * This function provides one step in the overall fork() sequence: It - * Allocates and initializes the child task's TCB. The overall sequence - * is: - * - * 1) User code calls fork(). fork() is provided in - * architecture-specific code. - * 2) fork()and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) up_fork() provides any additional operating context. up_fork must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) up_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * Allocate and initialize the child task's TCB. This is one step in the + * sequence common to vfork() and fork(); see the comment above the + * prototype in include/nuttx/sched.h for the whole sequence and for what + * the two primitives mean. + * + * Exactly two things depend on `vfork': + * + * - the address environment: vfork() joins the parent's, fork() + * duplicates it. + * - the stack: a vfork() child gets its own, which the architecture code + * fills with a relocated copy; a fork() child inherits the parent's + * address (fork_inherit_stack()). * * Input Parameters: - * retaddr - Return address - * argsize - Location to return the argument size + * retaddr - Address at which the child resumes + * vfork - true for vfork(), false for fork() * * Returned Value: * Upon successful completion, nxtask_setup_fork() returns a pointer to @@ -93,7 +172,7 @@ * ****************************************************************************/ -FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) +FAR struct tcb_s *nxtask_setup_fork(start_t retaddr, bool vfork) { FAR struct tcb_s *ptcb = this_task(); FAR struct tcb_s *parent; @@ -160,16 +239,80 @@ FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) } #if defined(CONFIG_ARCH_ADDRENV) - /* Join the parent address environment */ - if (ttype != TCB_FLAG_TTYPE_KERNEL) { - ret = addrenv_join(parent, child); + if (vfork) + { + /* vfork(): join the parent address environment, exactly as + * pthread_create() does. The child shares .data, .bss and the + * heap. + */ + + ret = addrenv_join(parent, child); + } +#ifdef CONFIG_ARCH_HAVE_FORK + else + { + /* POSIX fork(): duplicate the parent's address environment now, + * before anything else is set up. The duplicate holds a copy of + * the parent's contents -- including its stack -- at the parent's + * virtual addresses, which is what lets the child go on to inherit + * the stack address rather than be given a relocated copy. See + * fork_inherit_stack(). + */ + + ret = addrenv_fork(parent, child); + if (ret >= 0) + { + /* Make the child's address environment current for the rest of + * the setup, and for the architecture code that runs after it. + * + * From here on, everything written on the child's behalf + * has to land in the child's image rather than the parent's, + * because + * the two occupy the same virtual addresses: its thread-local + * storage, and -- on architectures that keep the register save + * area on the user stack rather than on a kernel stack -- the + * register context the child is resumed from. Writing those + * under the parent's environment corrupts the parent and + * leaves the child reading whatever the snapshot happened to + * contain. + * + * Reads are unaffected: everything the setup reads from the + * parent -- environ, the argument vector -- is legible at the + * same address in the child, precisely because it is a copy. + * + * nxtask_start_fork() puts the parent's environment back. + */ + + FAR struct addrenv_s *oldenv; + + ret = addrenv_select(child->addrenv_own, &oldenv); + } + } +#else + /* An address environment without ARCH_HAVE_FORK -- a protected build + * over an MMU, for instance. There is an address environment to join, + * but no POSIX fork() to duplicate it for, so the branch above is not + * compiled and `vfork' is always true here. + */ + + DEBUGASSERT(vfork); +#endif + if (ret < 0) { goto errout_with_tcb; } } +#else + /* Without address environments there is only one address space, so + * everything except the stack is shared no matter which primitive was + * called. POSIX fork() cannot be provided at all, and CONFIG_ARCH_HAVE_ + * FORK is not selected, so `vfork' is always true here. + */ + + DEBUGASSERT(vfork); #endif /* Duplicate the parent tasks environment */ @@ -193,12 +336,27 @@ FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) argv = nxsched_get_stackargs(parent); nxtask_setup_name(child, argv[0]); - /* Allocate the stack for the TCB */ + /* Allocate the stack for the TCB, or inherit the parent's */ + +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) + if (!vfork && ttype != TCB_FLAG_TTYPE_KERNEL) + { + /* The child's copy of the parent's stack is already in place, at the + * parent's address, courtesy of the duplication above. + */ + + fork_inherit_stack(parent, child); + ret = OK; + } + else +#endif + { + stack_size = (uintptr_t)ptcb->stack_base_ptr - + (uintptr_t)ptcb->stack_alloc_ptr + ptcb->adj_stack_size; - stack_size = (uintptr_t)ptcb->stack_base_ptr - - (uintptr_t)ptcb->stack_alloc_ptr + ptcb->adj_stack_size; + ret = up_create_stack(child, stack_size, ttype); + } - ret = up_create_stack(child, stack_size, ttype); if (ret < OK) { goto errout_with_tcb; @@ -235,20 +393,35 @@ FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) goto errout_with_tcb; } - /* Setup thread local storage */ - - ret = tls_dup_info(child, parent); - if (ret < OK) + /* Set up thread local storage and the argument vector. + * + * A fork() child that inherited its stack already has both, byte for + * byte, at the addresses the parent has them at -- they came across with + * the rest of the image. Re-creating them would carve fresh frames off a + * stack that already contains them, moving stack_base_ptr away from the + * parent's and undoing the inheritance. Only the TLS fields that name + * the task itself need correcting. + */ + +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) + if (!vfork && ttype != TCB_FLAG_TTYPE_KERNEL) { - goto errout_with_tcb; + fork_inherit_tls(child); } - - /* Setup to pass parameters to the new task */ - - ret = nxtask_setup_stackargs(child, argv[0], &argv[1]); - if (ret < OK) + else +#endif { - goto errout_with_tcb; + ret = tls_dup_info(child, parent); + if (ret < OK) + { + goto errout_with_tcb; + } + + ret = nxtask_setup_stackargs(child, argv[0], &argv[1]); + if (ret < OK) + { + goto errout_with_tcb; + } } /* Now we have enough in place that we can join the group */ @@ -258,6 +431,18 @@ FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) return child; errout_with_tcb: +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) + /* Get back into the parent's address environment before unwinding. If the + * duplication above never happened this is the environment we are already + * in, and addrenv_restore() is then a no-op. + */ + + if (!vfork && ttype != TCB_FLAG_TTYPE_KERNEL) + { + fork_restore_parent_env(); + } +#endif + nxsched_release_tcb((FAR struct tcb_s *)child, ttype); errout: set_errno(-ret); @@ -268,65 +453,129 @@ FAR struct tcb_s *nxtask_setup_fork(start_t retaddr) * Name: nxtask_start_fork * * Description: - * The fork() function has the same effect as fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. - * - * This function provides one step in the overall fork() sequence: It - * starts execution of the previously initialized TCB. The overall - * sequence is: - * - * 1) User code calls fork() - * 2) Architecture-specific code provides fork()and calls - * nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) fork() provides any additional operating context. fork must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * The last step of both primitives: finish the child and run it. The + * architecture-specific code calls this once it has built the child's + * register context and stack. + * + * A vfork() additionally suspends the caller until the child calls _exit() + * or one of the exec family of functions. The suspension lives here, in + * the kernel primitive, rather than in a libc waitpid() as it once did. + * Two things follow from that. The parent is released when the child's + * TCB is torn down (see nxtask_resume_vfork()), which for an exec()ing + * child is immediately after exec_swap() has handed the child's pid to the + * program it loaded -- so the parent resumes at exec(), holding a pid that + * names the running program, as POSIX requires. And vfork() no longer + * depends on CONFIG_SCHED_WAITPID. * * Input Parameters: - * child - The tcb_s struct instance that created by - * nxtask_setup_fork() method - * wait_child - whether need to wait until the child is running finished + * child - The tcb_s struct instance created by nxtask_setup_fork() + * vfork - true for vfork(), false for fork() * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * The process ID of the child, or ERROR on failure. * ****************************************************************************/ -pid_t nxtask_start_fork(FAR struct tcb_s *child) +pid_t nxtask_start_fork(FAR struct tcb_s *child, bool vfork) { +#ifdef CONFIG_ARCH_HAVE_VFORK + /* The rendezvous between the suspended parent and the child lives in this + * frame: the parent is blocked here for the whole lifetime of the child, + * so the storage is alive exactly as long as it is needed, and no + * allocation is required on a path that must not fail. + */ + + sem_t rel; + int ret; +#endif pid_t pid; - sinfo("Starting Child TCB=%p\n", child); + sinfo("Starting Child TCB=%p vfork=%d\n", child, vfork); DEBUGASSERT(child); +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) + /* The architecture code has finished writing the child's image, so put the + * parent back in its own address environment. See nxtask_setup_fork(). + */ + + if (!vfork && + (child->flags & TCB_FLAG_TTYPE_MASK) != TCB_FLAG_TTYPE_KERNEL) + { + fork_restore_parent_env(); + } +#endif + /* Get the assigned pid before we start the task */ pid = child->pid; +#ifdef CONFIG_ARCH_HAVE_VFORK + if (vfork) + { + nxsem_init(&rel, 0, 0); + child->vfork_rel = &rel; + } +#endif + /* Activate the task */ nxtask_activate(child); +#ifdef CONFIG_ARCH_HAVE_VFORK + if (vfork) + { + /* Wait for the child to _exit() or exec(). This is not a cancellation + * point and must not be interrupted by a signal: the child may be + * running on our stack, so returning early would corrupt it. + */ + + do + { + ret = nxsem_wait_uninterruptible(&rel); + } + while (ret == -EINTR); + + nxsem_destroy(&rel); + } +#endif + return pid; } +#ifdef CONFIG_ARCH_HAVE_VFORK +/**************************************************************************** + * Name: nxtask_resume_vfork + * + * Description: + * Release the vfork() parent suspended on this child, if there is one. + * + * Called from nxsched_release_tcb(), the last point in the child's life, + * by which time an exec()ing child has already handed its pid to the + * program it loaded. nxtask_abort_fork() reaches it too, so a fork that + * fails after the rendezvous also releases the parent. + * + * Input Parameters: + * child - The TCB being torn down + * + * Returned Value: + * None + * + ****************************************************************************/ + +void nxtask_resume_vfork(FAR struct tcb_s *child) +{ + FAR sem_t *rel = child->vfork_rel; + + if (rel != NULL) + { + /* Clearing the pointer first is what makes this once-only. */ + + child->vfork_rel = NULL; + nxsem_post(rel); + } +} +#endif /* CONFIG_ARCH_HAVE_VFORK */ + /**************************************************************************** * Name: nxtask_abort_fork * @@ -340,6 +589,20 @@ pid_t nxtask_start_fork(FAR struct tcb_s *child) void nxtask_abort_fork(FAR struct tcb_s *child, int errcode) { +#if defined(CONFIG_ARCH_ADDRENV) && defined(CONFIG_ARCH_HAVE_FORK) + /* A child holding an address environment of its own, rather than a + * reference to the caller's, is a fork() child, and nxtask_setup_fork() + * left that environment selected. Get back into the parent's before + * unwinding. See nxtask_setup_fork(). + */ + + if (child->addrenv_own != NULL && + child->addrenv_own != this_task()->addrenv_own) + { + fork_restore_parent_env(); + } +#endif + /* The TCB was added to the active task list by nxtask_setup_scheduler() */ dq_rem((FAR dq_entry_t *)child, list_inactivetasks()); @@ -349,5 +612,3 @@ void nxtask_abort_fork(FAR struct tcb_s *child, int errcode) nxsched_release_tcb(child, child->flags & TCB_FLAG_TTYPE_MASK); set_errno(errcode); } - -#endif /* CONFIG_ARCH_HAVE_FORK */ diff --git a/syscall/syscall.csv b/syscall/syscall.csv index 582deb3907f4f..98c99dff57ad4 100644 --- a/syscall/syscall.csv +++ b/syscall/syscall.csv @@ -202,7 +202,7 @@ "umount2","sys/mount.h","!defined(CONFIG_DISABLE_MOUNTPOINT)","int","FAR const char *","unsigned int" "unlink","unistd.h","!defined(CONFIG_DISABLE_MOUNTPOINT)","int","FAR const char *" "unsetenv","stdlib.h","!defined(CONFIG_DISABLE_ENVIRON)","int","FAR const char *" -"up_fork","nuttx/arch.h","defined(CONFIG_ARCH_HAVE_FORK)","pid_t" +"up_fork","nuttx/arch.h","defined(CONFIG_ARCH_HAVE_VFORK) || defined(CONFIG_ARCH_HAVE_FORK)","pid_t","bool" "utimens","sys/stat.h","","int","FAR const char *","const struct timespec [2]|FAR const struct timespec *" "wait","sys/wait.h","defined(CONFIG_SCHED_WAITPID) && defined(CONFIG_SCHED_HAVE_PARENT)","pid_t","FAR int *" "waitid","sys/wait.h","defined(CONFIG_SCHED_WAITPID) && defined(CONFIG_SCHED_HAVE_PARENT)","int","idtype_t","id_t"," FAR siginfo_t *","int" From fcf49da4c542124362c2533132ec66253a0d4400 Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Fri, 7 Aug 2026 12:13:14 +0200 Subject: [PATCH 2/6] !Documentation: Describe the fork()/vfork() split. Documentation/guides/fork_vfork_migration.rst is new. It says what changed and why, gives the two primitives as a table, states plainly what breaks, and answers "which replacement do I want?" from the reader's own reason for having called fork() -- posix_spawn() or vfork() to run a program, pthread_create() for a second flow of control that shares memory, fork() itself for an independent copy. It also documents the two configuration symbols, what an architecture has to implement to gain real fork(), and the one visible consequence of moving the vfork() suspension into the kernel: a waitpid() after a child that _exit()s can only report status where CONFIG_SCHED_CHILD_STATUS is enabled. reference/user/01_task_control.rst gains an entry for fork() and rewrites the one for vfork(), which described NuttX's limitations rather than the interface's contract. standards/posix.rst moves fork() from "No" to "Cond." and vfork() from "Yes" to "Cond.", both being conditional on the configuration now. implementation/memory_configurations.rst no longer lists fork() as unimplementable in the presence of address environments, which was the whole point of that section's wish list. Three long-standing typos in that file are corrected while touching it, since codespell checks the whole of any file a patch modifies. BREAKING CHANGE: this commit carries no code; it is the migration guide for the fork() withdrawal in the commit before it, and is marked so that every commit in the series carries the marker CONTRIBUTING.md 1.13 requires. The quick fixes are in Documentation/guides/fork_vfork_migration.rst. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- Documentation/guides/fork_vfork_migration.rst | 197 ++++++++++++++++++ Documentation/guides/index.rst | 1 + .../implementation/memory_configurations.rst | 21 +- .../reference/user/01_task_control.rst | 71 ++++++- Documentation/standards/posix.rst | 4 +- 5 files changed, 275 insertions(+), 19 deletions(-) create mode 100644 Documentation/guides/fork_vfork_migration.rst diff --git a/Documentation/guides/fork_vfork_migration.rst b/Documentation/guides/fork_vfork_migration.rst new file mode 100644 index 0000000000000..95c22f701f6b8 --- /dev/null +++ b/Documentation/guides/fork_vfork_migration.rst @@ -0,0 +1,197 @@ +========================================= +Migrating to separate ``fork``/``vfork`` +========================================= + +What changed +============ + +NuttX used to implement ``fork()`` and ``vfork()`` as the same function. Both +were thin libc wrappers around a single ``up_fork()`` syscall; ``vfork()`` +differed only by a trailing ``waitpid()``. Underneath, the child joined the +parent's address environment -- the same ``addrenv_join()`` that +``pthread_create()`` uses -- and got a private *copy of the stack*. So the +child shared ``.data``, ``.bss`` and the heap with its parent, and ran +concurrently with it. + +That was not ``fork()``. It was ``vfork()``-with-a-private-stack published +under ``fork()``'s name. The history says so plainly: the ``fork()`` this +replaces was NuttX's old ``vfork()``, renamed in 2023 without any change of +behaviour. And the consequence was silent: a program written against POSIX +``fork()`` compiled and ran, and its child's writes quietly landed in the +parent's variables. + +There are now two distinct primitives, and each means what its name says: + +.. list-table:: + :header-rows: 1 + :widths: 14 30 26 30 + + * - API + - Memory + - Parent + - Availability + * - ``fork()`` + - child gets **its own copy** at the same virtual addresses + - runs concurrently + - ``CONFIG_ARCH_HAVE_FORK`` -- only where an address environment can be + duplicated + * - ``vfork()`` + - child **shares** the parent's memory + - **suspended** until the child ``_exit()``\ s or ``exec()``\ s + - ``CONFIG_ARCH_HAVE_VFORK`` -- no address environment needed + +.. note:: + + ``fork()`` is provided only where the architecture implements + ``up_addrenv_fork()`` and therefore selects ``CONFIG_ARCH_HAVE_FORK``; it + becomes available architecture by architecture as that hook lands. Check + ``CONFIG_ARCH_HAVE_FORK`` in your own configuration rather than assuming + either way. Where it is unset, ``vfork()`` is what the configuration + offers. + +This is a breaking change +========================= + +Two things break, and they break loudly rather than quietly: + +**Code calling** ``fork()`` **on a target without a duplicable address +environment no longer builds.** ``fork()`` is not declared in ``unistd.h`` +there, so you get a compile error naming the function. That is the intended +outcome: a build error is strictly better than the silent wrongness it +replaces. Today that is every in-tree architecture, so every caller of +``fork()`` has to be looked at. + +**Code calling** ``fork()`` **on a target that does have real** ``fork()`` +**changes behaviour** -- from sharing to copying. Code that (perhaps +unknowingly) relied on the sharing will now see the parent and child diverge. + +Which replacement do I want? +============================ + +Answer the question "why did I call ``fork()``?". + +*I want the child to run a different program.* + Use :c:func:`posix_spawn` or ``task_spawn()``. This is the single most + common reason to call ``fork()``, NuttX has always provided a better answer + for it, and that answer does not have the pid discontinuity that + ``fork()``\ +\ ``exec()`` has. If you must keep the two-step idiom, use + ``vfork()`` + ``exec*()``: that is exactly what ``vfork()`` is for, and + unlike ``fork()`` it needs no duplicable address environment. + + .. code-block:: c + + #include + + pid = vfork(); /* was: pid = fork(); */ + if (pid == 0) + { + execv(path, argv); /* or _exit() on failure */ + _exit(EXIT_FAILURE); + } + + Note the restriction that comes with it: between the ``vfork()`` and the + ``exec()`` the child shares the parent's memory and runs on the parent's + behalf, so it must not modify anything, must not return from the calling + function, and must not call anything other than ``_exit()`` or an ``exec`` + family function. + +*I want a second flow of control that shares my memory.* + Use ``pthread_create()``. That is the same memory relationship the old + ``fork()`` gave you, spelled clearly, with a normal entry point instead of a + function that returns twice. There is no longer a returns-twice primitive + with concurrent sharing semantics: the old behaviour was not POSIX, and + ``vfork()`` is not a drop-in for it -- the parent is suspended, so parent and + child never run concurrently. + +*I want a genuinely independent copy of this process.* + Keep calling ``fork()``, and make sure your configuration selects + ``CONFIG_ARCH_HAVE_FORK``. Be aware there is no copy-on-write: the copy is + eager, so forking a large process needs as much free memory as the process + occupies and fails with ``ENOMEM`` otherwise. + +Configuration symbols +===================== + +``CONFIG_ARCH_HAVE_VFORK`` + Hidden. The architecture can implement POSIX ``vfork()``. Selected exactly + where ``ARCH_HAVE_FORK`` used to be, so every configuration that had the old + ``fork()`` has ``vfork()``. + +``CONFIG_ARCH_HAVE_FORK`` + Hidden, ``depends on ARCH_ADDRENV``. It no longer means "``fork()`` + exists"; it means "this configuration can provide POSIX ``fork()`` + semantics", which requires an address environment and an + ``up_addrenv_fork()`` to duplicate it with. + +Notes for architecture maintainers +================================== + +The register/stack snapshot machinery is common to both primitives. Each +architecture exposes one entry point, ``up_fork(bool vfork)``, whose argument +says which primitive the caller used and is handed to ``nxtask_setup_fork()``. +That is where the memory semantics are decided: ``addrenv_join()`` for +``vfork()``, ``addrenv_fork()`` for ``fork()``. + +Adding real ``fork()`` to an architecture +----------------------------------------- + +Two things are needed, and the second is the one that is easy to miss. + +**Implement** ``up_addrenv_fork()``. It duplicates an address environment: +allocate fresh pages, copy the parent's contents into them, and map them at the +*same* virtual addresses. ``up_addrenv_clone()`` is not this -- it copies only +the representation and leaves both processes pointing at one set of page +tables. Then give ``ARCH_HAVE_FORK`` a ``default y if `` line in +``arch/Kconfig``. + +**Build the child from the caller's saved system call frame.** In a kernel +build ``fork()`` is reached through a system call, so the return address and +stack pointer the architecture's fork entry point can observe for itself belong +to the *kernel*, not to the caller; a child built from those resumes at a +kernel address on a kernel stack. The architecture must record the caller's +exception frame when it traps -- ``xcp.sregs`` is the field that exists for +this -- and build the child from that instead, while a kernel thread that calls +the entry point directly still takes the ordinary path. + +Nothing else is required: the ``up_fork()`` entry point and the libc wrapper +are already there and become live automatically. + +Note that ``ARCH_HAVE_FORK`` is about a *per-process* address environment. A +protected build has one address space carved up once at boot, whether the +boundaries are drawn by an MPU or by a fixed set of MMU mappings; its +``up_addrenv_*()`` are stubs, and there is no mapping to duplicate at the same +virtual addresses. ``CONFIG_ARCH_ADDRENV`` being set is therefore not by +itself evidence that ``fork()`` can be provided. ``vfork()``, which shares the +parent's memory, works there as everywhere else. + +Known gaps +========== + +``fork()`` **is gained one architecture at a time.** The generic machinery is +complete -- ``addrenv_fork()``, the ``up_addrenv_fork()`` hook, the syscall, the +libc wrapper and the ``ostest`` case -- so an architecture provides ``fork()`` +by implementing ``up_addrenv_fork()`` and selecting ``CONFIG_ARCH_HAVE_FORK``, +with no further generic work. + +**A windowed ABI needs its stack rebased, not just copied.** On Xtensa, +giving a child a relocated copy of the parent's stack takes more than the copy: +the register-window save areas embedded in the stack hold absolute stack +pointers, so each one has to be rebased along with the copy, or the child +reloads a pointer into the *parent's* stack on its very first window underflow. +That rebasing is architecture-specific and belongs with the Xtensa entry points +rather than here. + +Note also on ``waitpid()`` after ``vfork()`` +============================================ + +The ``vfork()`` parent is now resumed when the child's TCB is torn down, so by +the time it runs the child is completely gone. Where the child called +``exec()`` this makes no difference -- ``exec_swap()`` has already given the +loaded program the child's pid, and that program is still running, so +``waitpid()`` behaves normally. Where the child called ``_exit()``, +``waitpid()`` can only return its status if ``CONFIG_SCHED_CHILD_STATUS`` is +enabled; otherwise it returns ``ECHILD``, because NuttX does not retain the +status of a task that no longer exists. That is a pre-existing property of +that configuration, not a change: the previous implementation blocked in a +libc ``waitpid(WNOWAIT)`` and an application's own ``waitpid()`` afterwards hit +the same wall. diff --git a/Documentation/guides/index.rst b/Documentation/guides/index.rst index acbb2a118b554..9b6114409aa00 100644 --- a/Documentation/guides/index.rst +++ b/Documentation/guides/index.rst @@ -37,6 +37,7 @@ Guides logging_rambuffer.rst ipv6.rst integrate_newlib.rst + fork_vfork_migration.rst protected_build.rst platform_directories.rst port_drivers_to_stm32f7.rst diff --git a/Documentation/implementation/memory_configurations.rst b/Documentation/implementation/memory_configurations.rst index 6ad536df5d6c3..f8f5b8b9b97de 100644 --- a/Documentation/implementation/memory_configurations.rst +++ b/Documentation/implementation/memory_configurations.rst @@ -30,7 +30,7 @@ On-Demand Paging NuttX also supports on-demand paging via ``CONFIG_PAGING``. On-demand paging is a method of virtual memory management and requires -the the CPU architecutre support a MMU. +the the CPU architecture support a MMU. In a system that uses on-demand paging, the OS responds to a page fault by copying data from some storage media into physical memory and setting up @@ -410,7 +410,7 @@ of functions that: 1. Have only one ``.text`` space in RAM, but 2. Separate ``.data`` and ``.bass`` space, and are -3. Separately linked into with the program in each address environmnet. +3. Separately linked into with the program in each address environment. (not implemented). @@ -484,7 +484,7 @@ at least in its current form. That full implementation of ``mmap()`` plus the minor changes to the NuttX ELF loader are all that are required to support fully share-able ``.text`` sections – as well as the memory savings -from not carrying aroung the relocation and symbol information +from not carrying around the relocation and symbol information (Not implemented). @@ -632,10 +632,13 @@ the contemplate in any real detail: and swap the state into physical memory as needed?(not implemented). * ``mmap()``. True shared memory and true file mapping could be supported. I am repeating myself (not implemented). -* ``fork()``. The ``fork()`` interface could be supported. NuttX currently - supports the "crippled" version, ``vfork()`` but with these process address - environments, the real ``fork()`` interface could be supported. - (not implemented). +* ``fork()``. The real ``fork()`` interface can be supported on configurations + with a duplicable process address environment: an architecture implements + ``up_addrenv_fork()`` and selects ``CONFIG_ARCH_HAVE_FORK``. What is not + implemented is + copy-on-write: the duplication copies the parent's pages eagerly, which + needs as much free memory as the parent occupies. Demand paging would fix + that. * Dynamic Stack Allocation. Completely eliminate the need for constant tuning of static stack sizes.(not implemented). * Shared Libraries. Am I repeating myself again?(not implemented). @@ -688,8 +691,8 @@ There are two problems here: So how do you create new tasks/processes in such a context. There is only one way possible; by using an interface that takes a file name as an argument (rather than absolute address). -New processes started with ``vfork()`` and ``exec()`` or with -``posix_spawn()`` should not have any of these issues. +New processes started with ``fork()`` or ``vfork()`` and ``exec()``, or with +``posix_spawn()``, should not have any of these issues. ARM Memory Management diff --git a/Documentation/reference/user/01_task_control.rst b/Documentation/reference/user/01_task_control.rst index 163030776c35b..05c064d4ecdcc 100644 --- a/Documentation/reference/user/01_task_control.rst +++ b/Documentation/reference/user/01_task_control.rst @@ -59,8 +59,9 @@ Standard interfaces - :c:func:`exit` - :c:func:`getpid` -Standard ``vfork`` and ``exec[v|l]`` interfaces: +Standard ``fork``/``vfork`` and ``exec[v|l]`` interfaces: + - :c:func:`fork` - :c:func:`vfork` - :c:func:`exec` - :c:func:`execv` @@ -347,6 +348,46 @@ Functions **POSIX Compatibility:** Compatible with the POSIX interface of the same name. +.. c:function:: pid_t fork(void) + + ``fork()`` creates a new process. The child process is an exact copy of + the calling process: it receives **its own copy** of the parent's memory, + at the same virtual addresses. Writes by the child are invisible to the + parent and writes by the parent are invisible to the child. The child may + modify anything, call any function, return from the function in which + ``fork()`` was called, and run indefinitely; it runs concurrently with the + parent. None of ``vfork()``'s restrictions apply. + + NOTE: ``fork()`` requires an address environment that can be + duplicated, and so it is available only where ``CONFIG_ARCH_HAVE_FORK`` + is selected -- which in turn requires ``CONFIG_ARCH_ADDRENV`` and an + architecture that implements ``up_addrenv_fork()``. **Where it cannot + be provided it is not provided at all**: the declaration is + absent from ``unistd.h`` and code that calls it fails to build. That + is deliberate. A build error naming the function is strictly better + than a ``fork()`` that silently gives the child the parent's memory. + See :doc:`/guides/fork_vfork_migration` for how to move code that + relied on the previous behaviour. + + There is no copy-on-write, because NuttX has no demand paging to build + it on, so the copy is eager: forking a large process needs as much free + memory as the process occupies and fails with ``ENOMEM`` otherwise. + Spawn-heavy code should prefer :c:func:`posix_spawn` or + :c:func:`vfork`, on NuttX as anywhere. + + Applications that relied on the historical NuttX ``fork()`` behaviour -- + shared memory, private stack, both running -- want + :c:func:`pthread_create`, which is that memory relationship spelled + clearly, or :c:func:`posix_spawn`. + + :return: Upon successful completion, ``fork()`` returns 0 to the child + process and returns the process ID of the child process to the parent + process. Otherwise, -1 is returned to the parent, no child process is + created, and ``errno`` is set to indicate the error. + + **POSIX Compatibility:** Compatible with the POSIX interface of the same + name. + .. c:function:: pid_t vfork(void) The ``vfork()`` function has the same effect as @@ -357,12 +398,18 @@ Functions function before successfully calling ``_exit()`` or one of the ``exec`` family of functions. - NOTE: ``vfork()`` is not an independent NuttX feature, but is - implemented in architecture-specific logic (using only helper - functions from the NuttX core logic). As a result, ``vfork()`` may - not be available on all architectures. The current implementation in - NuttX arm64 only guarantees that ``vfork()`` works when - CONFIG_BUILD_FLAT=y. + The child **shares** the parent's memory -- nothing is copied, which is the + entire point of ``vfork()`` and the reason a caller chooses it -- and the + parent is **suspended** until the child calls ``_exit()`` or one of the + ``exec`` family. That suspension is what makes the sharing safe, and it is + also the price: the restrictions above exist because the child is running + in the parent's address space on borrowed time. + + NOTE: ``vfork()`` is implementable with or without an MMU and is + available wherever ``CONFIG_ARCH_HAVE_VFORK`` is selected. The + suspension lives in the kernel primitive rather than in a libc + ``waitpid()``, so the parent is resumed at ``exec()`` as POSIX requires, + and ``vfork()`` does not depend on ``CONFIG_SCHED_WAITPID``. :return: Upon successful completion, ``vfork()`` returns 0 to the child process and returns the process ID of the child process to the @@ -449,7 +496,15 @@ Functions thread, then (2) call ``execv()`` or ``execl()`` to replace the new thread with a program from the file system. Since the new thread will be terminated by the ``execv()`` or ``execl()`` call, it really served no - purpose other than to support POSIX compatibility. + purpose other than to support POSIX compatibility. :c:func:`posix_spawn` + does the same job in one step and should be preferred. + + Note also that ``exec()`` does not overlay the calling process: it starts + the new program as a separate task. ``exec_swap()`` then exchanges the two + pids, so that from the parent's point of view the pid ``vfork()`` returned + does name the running program -- but the ``vfork()`` stub itself exits, and + it is that exit which releases the suspended ``vfork()`` parent. The parent + therefore resumes at ``exec()``, as POSIX requires. The non-standard binfmt function ``exec()`` needs to have (1) a symbol table that provides the list of symbols exported by the base code, and diff --git a/Documentation/standards/posix.rst b/Documentation/standards/posix.rst index cdca6eb748ea9..47622ec7921c2 100644 --- a/Documentation/standards/posix.rst +++ b/Documentation/standards/posix.rst @@ -1421,7 +1421,7 @@ Multiple Processes: +--------------------------------+---------+ | :c:func:`exit` | Yes | +--------------------------------+---------+ -| fork() | No | +| :c:func:`fork` | Cond. | +--------------------------------+---------+ | :c:func:`getpgrp` | Yes | +--------------------------------+---------+ @@ -2364,7 +2364,7 @@ XSI Multiple Process: +--------------------------------+---------+ | :c:func:`usleep` | Yes | +--------------------------------+---------+ -| :c:func:`vfork` | Yes | +| :c:func:`vfork` | Cond. | +--------------------------------+---------+ | :c:func:`waitid` | Yes | +--------------------------------+---------+ From 85f509ecee9c1a038ee0b58be520f26c7be16495 Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Sun, 9 Aug 2026 14:53:32 +0200 Subject: [PATCH 3/6] arch/arm: Build fork() children from the caller's syscall frame on armv7-a. In a kernel build the cloning primitives are reached through a system call, and armv7-a dispatches one by re-pointing the caller's own exception frame at dispatch_syscall() and switching to the task's kernel stack. The snapshot the entry point in fork.S takes for itself therefore describes the kernel-side stub, and the frames below it are on a stack the child gets no copy of: a child built from that snapshot resumes at a kernel address with a stack pointer into its own user stack. It faulted with a prefetch abort at PC 0 on qemu-armv7a:knsh, which is why the fork family had never been run there. Record what the caller was actually doing instead. arm_syscall() stores the exception frame of the outermost system call in xcp.sregs, mirroring riscv_swint.c, and arm_fork() chooses where the caller's registers live: arm_fork_syscall() when a user stack pointer is saved, rebuilding the child from xcp.sregs so that it returns from the very same SVC as the parent, in the same mode, on its own stack and with no inherited system call nesting; arm_fork_direct() otherwise -- the flat build, a kernel thread in any build, and a build without a kernel stack, where the call is dispatched on the caller's own stack so the caller's frames are copied along with the kernel-side ones. Note that the discriminator is xcp.ustkptr rather than TCB_FLAG_SYSCALL. On armv7-a the caller is the task that runs the kernel side of its own system call, so being in a system call is not by itself a reason to distrust the snapshot; the switch to the kernel stack is. Because arm_syscall() has already re-pointed the frame by the time arm_fork() runs, the caller's PC, CPSR and SP come from where arm_syscall() put them -- syscall[0].sysreturn, syscall[0].cpsr and ustkptr -- and the rest from the frame itself. Nothing selects the primitives on an ARM kernel build yet, so this commit changes no configuration; it is what the next one needs to be correct. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- Documentation/guides/fork_vfork_migration.rst | 20 ++ arch/arm/include/armv7-a/irq.h | 16 + arch/arm/src/armv7-a/arm_syscall.c | 14 + arch/arm/src/common/arm_fork.c | 288 +++++++++++++++--- 4 files changed, 291 insertions(+), 47 deletions(-) diff --git a/Documentation/guides/fork_vfork_migration.rst b/Documentation/guides/fork_vfork_migration.rst index 95c22f701f6b8..8420fbe5b7ce1 100644 --- a/Documentation/guides/fork_vfork_migration.rst +++ b/Documentation/guides/fork_vfork_migration.rst @@ -153,6 +153,26 @@ exception frame when it traps -- ``xcp.sregs`` is the field that exists for this -- and build the child from that instead, while a kernel thread that calls the entry point directly still takes the ordinary path. +Two architectures do it, and they are worth copying: + +* RISC-V: ``riscv_swint.c`` stores the frame in ``xcp.sregs``, and + ``riscv_fork.c`` rebuilds the child from it. +* armv7-a: ``arm_syscall.c`` stores the frame in ``xcp.sregs``, and + ``arm_fork()`` dispatches to ``arm_fork_syscall()`` or + ``arm_fork_direct()``. The discriminator here is a saved user stack + pointer, ``xcp.ustkptr``, rather than ``TCB_FLAG_SYSCALL``: armv7-a + dispatches a system call by re-pointing the caller's own exception frame at + ``dispatch_syscall()``, so the caller *is* the task that runs the kernel + side of the call. What makes its snapshot useless is not the system call + as such but the switch to the kernel stack, which leaves the kernel-side + frames on a stack the child gets no copy of. A build without a kernel + stack dispatches on the caller's own stack, so there the frames are copied + along with the caller's and ``arm_fork_direct()`` remains correct. Because + ``arm_syscall()`` has already re-pointed the frame by the time + ``arm_fork()`` runs, its PC, CPSR and SP are the kernel's; the caller's are + read from where ``arm_syscall()`` put them -- ``syscall[0].sysreturn``, + ``syscall[0].cpsr`` and ``ustkptr``. + Nothing else is required: the ``up_fork()`` entry point and the libc wrapper are already there and become live automatically. diff --git a/arch/arm/include/armv7-a/irq.h b/arch/arm/include/armv7-a/irq.h index b9f973b627bce..096f9ea99ba08 100644 --- a/arch/arm/include/armv7-a/irq.h +++ b/arch/arm/include/armv7-a/irq.h @@ -301,6 +301,22 @@ struct xcptcontext uint8_t nsyscalls; struct xcpt_syscall_s syscall[CONFIG_SYS_NNEST]; + + /* Where the register save area of the caller of the outermost system call + * is, which is the exception frame arm_vectorsvc built on the caller's own + * stack. It is recorded by arm_syscall() and is what the cloning + * primitives build the child's context from: a fork() or vfork() reached + * through a system call has to give the child the registers of the task + * that trapped, not those of the kernel-side stub that arm_fork() is + * called from. See arm_fork(). + * + * The frame is the one arm_syscall() has already re-pointed at + * dispatch_syscall(): its PC, CPSR, R0 and SP are the kernel's. The + * caller's own values are in syscall[0].sysreturn, syscall[0].cpsr and + * ustkptr respectively; R0 is the return value and belongs to neither. + */ + + uint32_t *sregs; #endif #ifdef CONFIG_ARCH_ADDRENV diff --git a/arch/arm/src/armv7-a/arm_syscall.c b/arch/arm/src/armv7-a/arm_syscall.c index 5c0039bf9b7ac..bcc9c187be532 100644 --- a/arch/arm/src/armv7-a/arm_syscall.c +++ b/arch/arm/src/armv7-a/arm_syscall.c @@ -513,6 +513,20 @@ uint32_t *arm_syscall(uint32_t *regs) rtcb->xcp.syscall[index].cpsr = regs[REG_CPSR]; #endif + /* Remember where the caller's registers are. The cloning + * primitives need them: the child of a fork() or vfork() made + * from user space resumes from this very SVC, so it is built from + * this frame and not from the registers the kernel-side stub + * happens to be running with. Only the outermost system call is + * of interest, since that is the one the caller made. See + * arm_fork(). + */ + + if (index == 0) + { + rtcb->xcp.sregs = regs; + } + regs[REG_PC] = (uint32_t)dispatch_syscall; #ifdef CONFIG_BUILD_KERNEL cpsr = regs[REG_CPSR] & ~PSR_MODE_MASK; diff --git a/arch/arm/src/common/arm_fork.c b/arch/arm/src/common/arm_fork.c index db822caee7570..55adec0510fa6 100644 --- a/arch/arm/src/common/arm_fork.c +++ b/arch/arm/src/common/arm_fork.c @@ -42,75 +42,53 @@ #include "sched/sched.h" /**************************************************************************** - * Public Functions + * Private Functions ****************************************************************************/ /**************************************************************************** - * Name: arm_fork + * Name: arm_fork_direct * * Description: - * The common ARM worker behind up_fork(). vfork() and fork() snapshot - * the caller's registers identically; `vfork' says which primitive was - * called, and is passed straight through to nxtask_setup_fork(), which is - * where the memory semantics are decided. - * - * What differs here is only the stack. Normally the child has a stack of - * its own, and this function fills it with a relocated copy of the - * parent's, rebasing the stack and frame pointers to match. When the - * child shares the parent's stack addresses -- a fork() child, inside its - * duplicated address environment -- there is nothing to relocate and the - * pointers are carried over unchanged. - * - * The overall sequence is: - * - * 1) User code calls vfork() or fork(). The libc wrapper enters - * up_fork(), which collects context information and transfers control - * to arm_fork(). - * 2) arm_fork() calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Establishing the child's address environment: joined to the - * parent's for vfork(), duplicated from it for fork() - * - Allocating the stack, or inheriting the parent's for fork() - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) arm_fork() provides any additional operating context. arm_fork must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) arm_fork() then calls nxtask_start_fork(), which for vfork() - * additionally suspends the caller. - * 6) which executes the child thread. - * - * nxtask_abort_fork() may be called if an error occurs between steps 3 and - * 6. + * Clone a caller that reached up_fork() by an ordinary function call, so + * that the register snapshot the entry point in fork.S took describes the + * caller itself. That is the case in a flat build, in a protected build + * -- where a system call is dispatched on the caller's own stack, so the + * caller's frames are copied along with the kernel-side ones and the child + * unwinds back through them -- and for a kernel thread in any build. * * Input Parameters: * vfork - true for vfork(), false for fork() + * parent - The calling task's TCB * context - Caller context information saved by the entry point * * Returned Value: - * Upon successful completion, 0 is returned to the child and the process - * ID of the child is returned to the parent. Otherwise, -1 is returned to - * the parent, no child is created, and errno is set to indicate the error. + * The process ID of the child, or ERROR on failure. * ****************************************************************************/ -pid_t arm_fork(bool vfork, const struct fork_s *context) +static pid_t arm_fork_direct(bool vfork, struct tcb_s *parent, + const struct fork_s *context) { - struct tcb_s *parent = this_task(); struct tcb_s *child; uint32_t newsp; uint32_t newfp; uint32_t newtop; uint32_t stacktop; uint32_t stackutil; -#ifdef CONFIG_ARCH_KERNEL_STACK - uint32_t oldsp = (uint32_t)parent->xcp.ustkptr; -#else uint32_t oldsp = context->sp; + +#ifdef CONFIG_ARCH_KERNEL_STACK + /* A caller that trapped into a system call and was switched onto its + * kernel stack left its own stack pointer here; the snapshot in `context' + * is the kernel-side stub's. Where the whole exception frame is available + * arm_fork_syscall() has already taken the call, so this is reached only + * where the caller's frames are copied along with the kernel-side ones. + */ + + if (parent->xcp.ustkptr != NULL) + { + oldsp = (uint32_t)parent->xcp.ustkptr; + } #endif sinfo("fork context [%p]:\n", context); @@ -277,3 +255,219 @@ pid_t arm_fork(bool vfork, const struct fork_s *context) return nxtask_start_fork(child, vfork); } + +#if defined(CONFIG_ARCH_ARMV7A) && defined(CONFIG_ARCH_KERNEL_STACK) + +/**************************************************************************** + * Name: arm_fork_syscall + * + * Description: + * Clone a caller that reached up_fork() through a system call that was + * switched onto a kernel stack. The register snapshot fork.S took is + * useless here: it describes the kernel-side stub, and the frames below + * it are on a stack the child does not get a copy of, so a child built + * from it would resume at a kernel address with a stack pointer into its + * own user stack. + * + * What the caller was actually doing is the exception frame arm_vectorsvc + * built on the caller's stack and arm_syscall() recorded in xcp.sregs; the + * child is built from that. It therefore returns from the very same SVC + * instruction as the parent, in the same mode, differing only in that it + * sees 0 as the return value and runs on its own stack. The child is not + * in a system call at all, so it inherits none of the parent's nesting + * state. + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * parent - The calling task's TCB + * + * Returned Value: + * The process ID of the child, or ERROR on failure. + * + ****************************************************************************/ + +static pid_t arm_fork_syscall(bool vfork, struct tcb_s *parent) +{ + uint32_t *sregs = parent->xcp.sregs; + struct tcb_s *child; + uint32_t newsp; + uint32_t newfp; + uint32_t newtop; + uint32_t stacktop; + uint32_t stackutil; + uint32_t oldsp = (uint32_t)parent->xcp.ustkptr; + uint32_t sysreturn; + uint32_t cpsr; + + DEBUGASSERT(sregs != NULL && parent->xcp.nsyscalls > 0); + + /* Where the caller resumes, and in which mode. arm_syscall() re-pointed + * the frame at dispatch_syscall() before this was reached, so these two + * come from where it put the originals rather than from the frame. In a + * protected build there is no mode change, so the frame still holds the + * caller's CPSR. + */ + + sysreturn = parent->xcp.syscall[0].sysreturn; +#ifdef CONFIG_BUILD_KERNEL + cpsr = parent->xcp.syscall[0].cpsr; +#else + cpsr = sregs[REG_CPSR]; +#endif + + /* Allocate and initialize a TCB for the child task. */ + + child = nxtask_setup_fork((start_t)(sysreturn & ~1), vfork); + if (!child) + { + serr("ERROR: nxtask_setup_fork failed\n"); + return (pid_t)ERROR; + } + + stacktop = (uint32_t)parent->stack_base_ptr + + parent->adj_stack_size; + DEBUGASSERT(stacktop > oldsp && oldsp >= (uint32_t)parent->stack_base_ptr); + stackutil = stacktop - oldsp; + + if (child->stack_base_ptr == parent->stack_base_ptr) + { + /* The child is running at the parent's stack addresses, inside its + * own duplicated address environment. There is nothing to relocate; + * see the same case in arm_fork_direct(). + */ + + newsp = oldsp; + newfp = sregs[REG_FP]; + } + else + { + newtop = (uint32_t)child->stack_base_ptr + + child->adj_stack_size; + newsp = newtop - stackutil; + + /* Put the child's register save area where the parent's is: just + * below the stack the caller was using. It cannot be left at the top + * of the child's stack, which is where up_initial_state() put it, + * because the copy of the parent's stack below is about to land there. + */ + + child->xcp.regs = (uint32_t *)(newsp - XCPTCONTEXT_SIZE); + + memcpy((void *)newsp, (const void *)oldsp, stackutil); + + /* Was there a frame pointer in place before? */ + + if (sregs[REG_FP] >= oldsp && sregs[REG_FP] < stacktop) + { + uint32_t frameutil = stacktop - sregs[REG_FP]; + newfp = newtop - frameutil; + } + else + { + newfp = sregs[REG_FP]; + } + } + + /* Inherit the caller's whole exception frame, integer and floating point + * registers alike, then fix up only what has to differ: the child sees 0 + * as the return value and runs on its own stack. + */ + + memcpy(child->xcp.regs, sregs, XCPTCONTEXT_SIZE); + + child->xcp.regs[REG_R0] = 0; + child->xcp.regs[REG_FP] = newfp; + child->xcp.regs[REG_SP] = newsp; + child->xcp.regs[REG_PC] = sysreturn; + child->xcp.regs[REG_CPSR] = cpsr; + + /* And, finally, start the child task. A vfork() additionally suspends us + * until the child calls _exit() or exec(). + */ + + return nxtask_start_fork(child, vfork); +} + +#endif /* CONFIG_ARCH_ARMV7A && CONFIG_ARCH_KERNEL_STACK */ + +/**************************************************************************** + * Public Functions + ****************************************************************************/ + +/**************************************************************************** + * Name: arm_fork + * + * Description: + * The common ARM worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. + * + * What differs here is only the stack. Normally the child has a stack of + * its own, and this function fills it with a relocated copy of the + * parent's, rebasing the stack and frame pointers to match. When the + * child shares the parent's stack addresses -- a fork() child, inside its + * duplicated address environment -- there is nothing to relocate and the + * pointers are carried over unchanged. + * + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). The libc wrapper enters + * up_fork(), which collects context information and transfers control + * to arm_fork(). + * 2) arm_fork() calls nxtask_setup_fork(). + * 3) nxtask_setup_fork() allocates and configures the child task's TCB. + * This consists of: + * - Allocation of the child task's TCB. + * - Initialization of file descriptors and streams + * - Configuration of environment variables + * - Establishing the child's address environment: joined to the + * parent's for vfork(), duplicated from it for fork() + * - Allocating the stack, or inheriting the parent's for fork() + * - Setup the input parameters for the task. + * - Initialization of the TCB (including call to up_initial_state()) + * 4) arm_fork() provides any additional operating context. arm_fork must: + * - Initialize special values in any CPU registers that were not + * already configured by up_initial_state() + * 5) arm_fork() then calls nxtask_start_fork(), which for vfork() + * additionally suspends the caller. + * 6) which executes the child thread. + * + * nxtask_abort_fork() may be called if an error occurs between steps 3 and + * 6. + * + * Everything above is common to the two ways this can be reached, which + * differ only in where the caller's registers are to be found -- see + * arm_fork_direct() and arm_fork_syscall(). + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by the entry point + * + * Returned Value: + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. + * + ****************************************************************************/ + +pid_t arm_fork(bool vfork, const struct fork_s *context) +{ + struct tcb_s *parent = this_task(); + +#if defined(CONFIG_ARCH_ARMV7A) && defined(CONFIG_ARCH_KERNEL_STACK) + /* A saved user stack pointer means this was reached through a system call + * that switched to the task's kernel stack, so the caller is the user task + * that trapped and not the code that called into fork.S. A kernel thread + * has no kernel stack to switch to and never gets here with one saved. + */ + + if (parent->xcp.ustkptr != NULL) + { + DEBUGASSERT((parent->flags & TCB_FLAG_SYSCALL) != 0); + return arm_fork_syscall(vfork, parent); + } +#endif + + return arm_fork_direct(vfork, parent, context); +} From 79d453cd5f9c5a6264b7c2bab72a1073a744b3b6 Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Sun, 9 Aug 2026 14:53:46 +0200 Subject: [PATCH 4/6] arch/arm: Restore vfork() on 32-bit ARM kernel builds. ARCH_ARM has selected ARCH_HAVE_VFORK only "if !BUILD_KERNEL" since the fork()/vfork() split. That condition was deliberate but temporary: it was added because the fork family had never worked on a 32-bit ARM kernel build -- the entry point in fork.S snapshots the kernel-side stub rather than the caller, so a child resumes at a kernel address -- and said in as many words that "arch/arm takes the condition off again in the patch that adds its saved-syscall-frame path". That patch is the one before this. arm_syscall() records the caller's exception frame in xcp.sregs and arm_fork() builds the child from it, so the condition has nothing left to protect against. Cortex-M is unaffected either way -- BUILD_KERNEL depends on ARCH_USE_MMU, which it does not have -- so the only configurations this changes are the MMU-capable ARM ports, which are exactly the ones the previous commit fixed. Verified on qemu-armv7a:knsh under qemu-system-arm: ostest's vfork_test passes, where before the change vfork() was absent. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- arch/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index 13e0db6ea983b..3a09b26fcfcc6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -11,7 +11,7 @@ config ARCH_ARM bool "ARM" select ARCH_HAVE_BACKTRACE select ARCH_HAVE_INTERRUPTSTACK - select ARCH_HAVE_VFORK if !BUILD_KERNEL + select ARCH_HAVE_VFORK select ARCH_HAVE_STACKCHECK select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_STDARG_H From 151331fd1515734ff312bbbbfe80091cc04de978 Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Sun, 9 Aug 2026 15:08:43 +0200 Subject: [PATCH 5/6] arch/arm64: Build fork() children from the caller's syscall frame. In a kernel or protected build vfork() is reached through a system call, so the return address and stack pointer that the entry point in arm64_fork_func.S can snapshot for itself belong to the kernel-side stub, not to the caller. A child built from that snapshot resumes at a kernel address on a kernel stack. This is why arm64 selected the fork family only for the flat build. Record what the caller was actually doing instead. arm64_sync_exc passes the exception frame to dispatch_syscall() in x7 -- x0-x6 carry the call number and its six parameters, so x7 is free -- and dispatch_syscall() stores it in xcp.sregs, mirroring what riscv_swint.c does. arm64_fork() then chooses where the caller's registers live: arm64_fork_syscall() when TCB_FLAG_SYSCALL is set, rebuilding the child from xcp.sregs so that it returns from the very same SVC as the parent; arm64_fork_direct() otherwise, which is the flat build and any kernel thread that calls the entry point as a plain function. The stack copy and the relocation of pointers into it are shared by both paths in arm64_fork_stack() and arm64_fork_reloc(). With that in place ARCH_ARM64 can select ARCH_HAVE_VFORK unconditionally. Verified on qemu-armv8a:knsh (BUILD_KERNEL), qemu-armv8a:nsh (BUILD_FLAT) and qemu-armv8a:citest_smp under qemu-system-aarch64: ostest's vfork_test passes on all three, and it was absent from knsh before the change. The protected configurations are build-verified only (fvp-armv8r:pnsh), there being no emulator for them here. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- Documentation/guides/fork_vfork_migration.rst | 7 +- arch/Kconfig | 2 +- arch/arm64/include/irq.h | 15 + arch/arm64/src/common/arm64_fork.c | 400 ++++++++++++++---- arch/arm64/src/common/arm64_syscall.c | 6 +- arch/arm64/src/common/arm64_vectors.S | 7 + 6 files changed, 341 insertions(+), 96 deletions(-) diff --git a/Documentation/guides/fork_vfork_migration.rst b/Documentation/guides/fork_vfork_migration.rst index 8420fbe5b7ce1..63e5f682a0373 100644 --- a/Documentation/guides/fork_vfork_migration.rst +++ b/Documentation/guides/fork_vfork_migration.rst @@ -153,10 +153,15 @@ exception frame when it traps -- ``xcp.sregs`` is the field that exists for this -- and build the child from that instead, while a kernel thread that calls the entry point directly still takes the ordinary path. -Two architectures do it, and they are worth copying: +Three architectures do it, and they are worth copying: * RISC-V: ``riscv_swint.c`` stores the frame in ``xcp.sregs``, and ``riscv_fork.c`` rebuilds the child from it. +* arm64: ``arm64_vectors.S`` hands the frame to ``dispatch_syscall()``, which + stores it in ``xcp.sregs``; ``arm64_fork()`` then dispatches to + ``arm64_fork_syscall()`` or ``arm64_fork_direct()`` according to whether + ``TCB_FLAG_SYSCALL`` is set, so a kernel thread that calls the entry point + directly still works. * armv7-a: ``arm_syscall.c`` stores the frame in ``xcp.sregs``, and ``arm_fork()`` dispatches to ``arm_fork_syscall()`` or ``arm_fork_direct()``. The discriminator here is a saved user stack diff --git a/arch/Kconfig b/arch/Kconfig index 3a09b26fcfcc6..3944e83d5a5f6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -29,7 +29,7 @@ config ARCH_ARM64 select ARCH_64BIT select ARCH_HAVE_BACKTRACE select ARCH_HAVE_INTERRUPTSTACK - select ARCH_HAVE_VFORK if !BUILD_KERNEL && !BUILD_PROTECTED + select ARCH_HAVE_VFORK select ARCH_HAVE_STACKCHECK select ARCH_HAVE_CUSTOMOPT select ARCH_HAVE_STDARG_H diff --git a/arch/arm64/include/irq.h b/arch/arm64/include/irq.h index 3fae95b1ff392..6dd9a4e282158 100644 --- a/arch/arm64/include/irq.h +++ b/arch/arm64/include/irq.h @@ -284,6 +284,21 @@ struct xcptcontext uint64_t *initregs; #endif +#ifdef CONFIG_LIB_SYSCALL + /* The caller's register context, as saved by the SVC exception entry, for + * the duration of a system call. This is what the *user* was doing when + * it trapped, as opposed to `regs' above, which during a system call + * describes the kernel. + * + * vfork() and fork() need it: they are reached through a system call, so + * the return address and stack pointer their architecture entry point can + * see for itself are the kernel's, and a child built from those would + * resume at a kernel address on a kernel stack. + */ + + uint64_t *sregs; +#endif + #ifdef CONFIG_ARCH_FPU uint64_t *fpu_regs; uint64_t *saved_fpu_regs; diff --git a/arch/arm64/src/common/arm64_fork.c b/arch/arm64/src/common/arm64_fork.c index b7d5af1f9f7d4..3566c64ded810 100644 --- a/arch/arm64/src/common/arm64_fork.c +++ b/arch/arm64/src/common/arm64_fork.c @@ -53,74 +53,131 @@ ****************************************************************************/ /**************************************************************************** - * Public Functions + * Private Functions ****************************************************************************/ -#ifdef CONFIG_ARCH_FPU +/**************************************************************************** + * Name: arm64_fork_stack + * + * Description: + * Give the child the part of the parent's stack that is in use, copied to + * the top of the child's own stack. + * + * The copy is aligned with the top of each stack rather than the bottom, + * so a single offset carries any address in the copied region from the + * parent's stack to the child's; that offset is what is returned, and + * arm64_fork_reloc() applies it. + * + * Input Parameters: + * parent - The parent task's TCB + * child - The child task's TCB + * sp - The parent's stack pointer where the primitive was called + * + * Returned Value: + * The offset from an address in the parent's stack to the same place in + * the child's copy of it. + * + ****************************************************************************/ -void arm64_fork_fpureg_save(struct fork_s *context) +static uint64_t arm64_fork_stack(struct tcb_s *parent, struct tcb_s *child, + uint64_t sp) { - /* Take a snapshot of the thread fpu reg context right now */ + uint64_t stacktop; + uint64_t stackutil; + uint64_t newtop; - arm64_fpu_save(context->fpu); - UP_DSB(); -} + /* How much of the parent's stack was utilized? The ARM uses a push-down + * stack so that the current stack pointer should be lower than the + * initial, adjusted stack pointer. The stack usage should be the + * difference between those two. + */ -#endif + stacktop = (uint64_t)parent->stack_base_ptr + parent->adj_stack_size; + DEBUGASSERT(stacktop > sp); + stackutil = stacktop - sp; + + /* Make some feeble effort to preserve the stack contents. This is + * feeble because the stack surely contains invalid pointers and other + * content that will not work in the child context. However, if the + * user follows all of the caveats of vfork() usage, even this feeble + * effort is overkill. + */ + + newtop = (uint64_t)child->stack_base_ptr + child->adj_stack_size; + memcpy((void *)(newtop - stackutil), (const void *)sp, stackutil); + + return newtop - stacktop; +} /**************************************************************************** - * Name: arm64_fork + * Name: arm64_fork_reloc * * Description: - * The common ARM64 worker behind up_fork(). vfork() and fork() snapshot - * the caller's registers identically; `vfork' says which primitive was - * called, and is passed straight through to nxtask_setup_fork(), which is - * where the memory semantics are decided. + * Carry one address from the parent's stack over to the child's copy of + * it. Addresses outside the region that arm64_fork_stack() copied are + * returned unchanged: they point somewhere the child shares with the + * parent, or somewhere that has no counterpart at all. * - * The overall sequence is: + * Input Parameters: + * parent - The parent task's TCB + * addr - The address to relocate + * sp - The parent's stack pointer where the primitive was called, + * which is the low end of the region that was copied + * offset - The offset returned by arm64_fork_stack() * - * 1) User code calls vfork() or fork(). up_fork() collects context - * information and transfers control to arm64_fork(). - * 2) arm64_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) arm64_fork() provides any additional operating context. arm64_fork - * must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) arm64_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * Returned Value: + * The relocated address. * - * nxtask_abort_fork() may be called if an error occurs between steps 3 and - * 6. + ****************************************************************************/ + +static uint64_t arm64_fork_reloc(struct tcb_s *parent, uint64_t addr, + uint64_t sp, uint64_t offset) +{ + uint64_t stacktop = (uint64_t)parent->stack_base_ptr + + parent->adj_stack_size; + + /* The top of the stack is included: a stack pointer resting there is one + * past the last byte copied, and still has to move with it. + */ + + if (addr >= sp && addr <= stacktop) + { + return addr + offset; + } + + return addr; +} + +/**************************************************************************** + * Name: arm64_fork_direct + * + * Description: + * Clone a caller that reached up_fork() by an ordinary function call, so + * that the register snapshot taken by arm64_fork_func.S describes the + * caller itself. That is the case in a flat build, and for a kernel + * thread in any build. + * + * The child has no exception frame to inherit, so one is synthesised: it + * resumes at the caller's return address, at the same privilege level, + * with the callee-saved registers the caller had. * * Input Parameters: * vfork - true for vfork(), false for fork() - * context - Caller context information saved by up_fork() + * parent - The calling task's TCB + * context - Caller context information saved by arm64_fork_func.S * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * The process ID of the child, or ERROR on failure. * ****************************************************************************/ -pid_t arm64_fork(bool vfork, const struct fork_s *context) +static pid_t arm64_fork_direct(bool vfork, struct tcb_s *parent, + const struct fork_s *context) { - struct tcb_s *parent = this_task(); struct tcb_s *child; + uint64_t offset; uint64_t newsp; uint64_t newfp; - uint64_t newtop; - uint64_t stacktop; - uint64_t stackutil; /* Allocate and initialize a TCB for the child task. */ @@ -131,57 +188,11 @@ pid_t arm64_fork(bool vfork, const struct fork_s *context) return (pid_t)ERROR; } - /* How much of the parent's stack was utilized? The ARM uses - * a push-down stack so that the current stack pointer should - * be lower than the initial, adjusted stack pointer. The - * stack usage should be the difference between those two. - */ - - stacktop = (uint64_t)parent->stack_base_ptr + - parent->adj_stack_size; - DEBUGASSERT(stacktop > context->sp); - stackutil = stacktop - context->sp; + /* Copy the parent's stack to the child and relocate the pointers into it */ - if (child->stack_base_ptr == parent->stack_base_ptr) - { - /* The child is running at the parent's stack addresses, inside its - * own duplicated address environment. There is nothing to relocate: - * every stack address the child inherits is still the address it - * names. - */ - - newsp = context->sp; - newfp = context->fp; - } - else - { - /* Make some feeble effort to preserve the stack contents. This is - * feeble because the stack surely contains invalid pointers and other - * content that will not work in the child context. However, if the - * user follows all of the caveats of vfork() usage, even this feeble - * effort is overkill. - * - * For a POSIX fork() child the stack contents are not merely a feeble - * effort: the child is entitled to use them, and it does. - */ - - newtop = (uint64_t)child->stack_base_ptr + - child->adj_stack_size; - newsp = newtop - stackutil; - memcpy((void *)newsp, (const void *)context->sp, stackutil); - - /* Was there a frame pointer in place before? */ - - if (context->fp >= context->sp && context->fp < stacktop) - { - uint64_t frameutil = stacktop - context->fp; - newfp = newtop - frameutil; - } - else - { - newfp = context->fp; - } - } + offset = arm64_fork_stack(parent, child, context->sp); + newsp = context->sp + offset; + newfp = arm64_fork_reloc(parent, context->fp, context->sp, offset); /* Update the stack pointer, frame pointer, and volatile registers. When * the child TCB was initialized, all of the values were set to zero. @@ -253,3 +264,206 @@ pid_t arm64_fork(bool vfork, const struct fork_s *context) return nxtask_start_fork(child, vfork); } + +#ifdef CONFIG_LIB_SYSCALL + +/**************************************************************************** + * Name: arm64_fork_syscall + * + * Description: + * Clone a caller that reached up_fork() through a system call. The + * register snapshot taken by arm64_fork_func.S is useless here: it + * describes the kernel-side stub, so a child built from it would resume at + * a kernel address on a kernel stack. What the caller was actually doing + * is the exception frame the SVC handler recorded in xcp.sregs; the child + * is built from that. + * + * The child therefore returns from the very same SVC instruction as the + * parent, at the same privilege level, differing only in that it sees 0 + * as the return value and runs on its own stack. + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * parent - The calling task's TCB + * + * Returned Value: + * The process ID of the child, or ERROR on failure. + * + ****************************************************************************/ + +static pid_t arm64_fork_syscall(bool vfork, struct tcb_s *parent) +{ + uint64_t *sregs = parent->xcp.sregs; + struct tcb_s *child; + uint64_t offset; + uint64_t newsp; + uint64_t newfp; + uint64_t regtop; + uint64_t sp; + + DEBUGASSERT(sregs != NULL); + + /* Which stack the caller was on depends on the level it trapped from: a + * user task uses SP_EL0, anything running in the kernel uses SP_ELx. + */ + + if ((sregs[REG_SPSR] & SPSR_MODE_MASK) == SPSR_MODE_EL0T) + { + sp = sregs[REG_SP_EL0]; + } + else + { + sp = sregs[REG_SP_ELX]; + } + + /* Allocate and initialize a TCB for the child task. The child resumes at + * the instruction after the SVC, which is where the parent resumes too. + */ + + child = nxtask_setup_fork((start_t)sregs[REG_ELR], vfork); + if (!child) + { + serr("ERROR: nxtask_setup_fork failed\n"); + return (pid_t)ERROR; + } + + /* Copy the parent's stack to the child and relocate the pointers into it */ + + offset = arm64_fork_stack(parent, child, sp); + newsp = sp + offset; + newfp = arm64_fork_reloc(parent, sregs[REG_FP], sp, offset); + + /* Where does the register save area the child is resumed from go? The + * parent's is wherever SP_ELx pointed when it trapped, so put the child's + * at the matching place: the top of its own kernel stack if it has one -- + * a user process in a kernel build -- or else the same offset into its + * copy of the parent's stack. + */ + +#ifdef CONFIG_ARCH_KERNEL_STACK + if (child->xcp.kstack) + { + regtop = (uint64_t)child->xcp.kstack + ARCH_KERNEL_STACKSIZE; + } + else +#endif + { + regtop = arm64_fork_reloc(parent, sregs[REG_SP_ELX], sp, offset); + } + + child->xcp.regs = (void *)(regtop - XCPTCONTEXT_SIZE); + + /* Inherit the parent's whole exception frame, integer and FPU registers + * alike, then fix up only what has to differ. + */ + + memcpy(child->xcp.regs, sregs, XCPTCONTEXT_SIZE); + +#ifdef CONFIG_ARCH_FPU + child->xcp.fpu_regs = (void *)(regtop - FPU_CONTEXT_SIZE); +#endif + + child->xcp.regs[REG_X0] = 0; + child->xcp.regs[REG_FP] = newfp; + child->xcp.regs[REG_EXE_DEPTH] = 0; + child->xcp.regs[REG_SP_ELX] = regtop - XCPTCONTEXT_SIZE; + + if ((sregs[REG_SPSR] & SPSR_MODE_MASK) == SPSR_MODE_EL0T) + { + child->xcp.regs[REG_SP_EL0] = newsp; +#ifdef CONFIG_ARCH_KERNEL_STACK + child->xcp.ustkptr = (uintptr_t *)newsp; +#endif + } + + /* And, finally, start the child task. On a failure, nxtask_start_fork() + * will discard the TCB by calling nxtask_abort_fork(). + */ + + return nxtask_start_fork(child, vfork); +} + +#endif /* CONFIG_LIB_SYSCALL */ + +/**************************************************************************** + * Public Functions + ****************************************************************************/ + +#ifdef CONFIG_ARCH_FPU + +void arm64_fork_fpureg_save(struct fork_s *context) +{ + /* Take a snapshot of the thread fpu reg context right now */ + + arm64_fpu_save(context->fpu); + UP_DSB(); +} + +#endif + +/**************************************************************************** + * Name: arm64_fork + * + * Description: + * The common ARM64 worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. + * + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). up_fork() collects context + * information and transfers control to arm64_fork(). + * 2) arm64_fork() and calls nxtask_setup_fork(). + * 3) nxtask_setup_fork() allocates and configures the child task's TCB. + * This consists of: + * - Allocation of the child task's TCB. + * - Initialization of file descriptors and streams + * - Configuration of environment variables + * - Allocate and initialize the stack + * - Setup the input parameters for the task. + * - Initialization of the TCB (including call to up_initial_state()) + * 4) arm64_fork() provides any additional operating context. arm64_fork + * must: + * - Initialize special values in any CPU registers that were not + * already configured by up_initial_state() + * 5) arm64_fork() then calls nxtask_start_fork() + * 6) nxtask_start_fork() then executes the child thread. + * + * nxtask_abort_fork() may be called if an error occurs between steps 3 and + * 6. + * + * Everything above is common to the two ways this can be reached, which + * differ only in where the caller's registers are to be found -- see + * arm64_fork_direct() and arm64_fork_syscall(). + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by up_fork() + * + * Returned Value: + * Upon successful completion, fork() returns 0 to the child process and + * returns the process ID of the child process to the parent process. + * Otherwise, -1 is returned to the parent, no child process is created, + * and errno is set to indicate the error. + * + ****************************************************************************/ + +pid_t arm64_fork(bool vfork, const struct fork_s *context) +{ + struct tcb_s *parent = this_task(); + +#ifdef CONFIG_LIB_SYSCALL + /* If a system call is in progress then this was reached from its kernel- + * side stub, and the caller is the user task that trapped, not the code + * that called into arm64_fork_func.S. + */ + + if ((parent->flags & TCB_FLAG_SYSCALL) != 0) + { + return arm64_fork_syscall(vfork, parent); + } +#endif + + return arm64_fork_direct(vfork, parent, context); +} diff --git a/arch/arm64/src/common/arm64_syscall.c b/arch/arm64/src/common/arm64_syscall.c index 8d503aa64e3dc..8e9bc569986da 100644 --- a/arch/arm64/src/common/arm64_syscall.c +++ b/arch/arm64/src/common/arm64_syscall.c @@ -95,7 +95,7 @@ static void arm64_dump_syscall(const char *tag, uint64_t cmd, uintptr_t dispatch_syscall(unsigned int nbr, uintptr_t parm1, uintptr_t parm2, uintptr_t parm3, uintptr_t parm4, uintptr_t parm5, - uintptr_t parm6) + uintptr_t parm6, void *context) { struct tcb_s *rtcb = this_task(); register long x0 asm("x0") = (long)(nbr); @@ -117,6 +117,10 @@ uintptr_t dispatch_syscall(unsigned int nbr, uintptr_t parm1, return -ENOSYS; } + /* Set the user register context to TCB */ + + rtcb->xcp.sregs = context; + /* Indicate that we are in a syscall handler */ rtcb->flags |= TCB_FLAG_SYSCALL; diff --git a/arch/arm64/src/common/arm64_vectors.S b/arch/arm64/src/common/arm64_vectors.S index eaedd7508bea2..2dae675644e95 100644 --- a/arch/arm64/src/common/arm64_vectors.S +++ b/arch/arm64/src/common/arm64_vectors.S @@ -172,6 +172,13 @@ SECTION_FUNC(text, arm64_sync_exc) msr daifclr, #IRQ_DAIF_MASK /* Re-enable interrupts */ 1: + /* Pass the caller's exception frame as the last argument. x0-x6 hold + * the system call number and its six parameters, so x7 is free. The + * cloning primitives need this frame to build the child's context; see + * dispatch_syscall() and arm64_fork(). + */ + + mov x7, sp bl dispatch_syscall msr daifset, #IRQ_DAIF_MASK /* Disable interrupts */ From bc2cf4cc142e4f468b24ded0fe7f6355fac7efcf Mon Sep 17 00:00:00 2001 From: Marco Casaroli Date: Sun, 9 Aug 2026 15:13:39 +0200 Subject: [PATCH 6/6] arch/x86_64: Build fork() children from the caller's syscall frame. In a kernel build vfork() is reached through a system call, so the return address and stack pointer the architecture's entry point can see for itself are the kernel's, not the caller's. A child built from those resumes at a kernel address, which is why x86_64 selected the fork family only for the flat build. x86_64_syscall() now publishes the caller's frame in xcp.sregs for the duration of the stub call, and x86_64_fork() builds the child from it: x86_64_fork_syscall() when xcp.sregs is non-NULL, so that the child returns from the very same `syscall' instruction as the parent, in user mode, on its own stack; x86_64_fork_direct() otherwise, which is the flat build and any kernel thread that calls the entry point as a plain function. The discriminator is xcp.sregs rather than TCB_FLAG_SYSCALL, which arm64 and RISC-V use: that flag also defers signal actions, x86_64 has never raised it, and its kernel-build signal path does not survive being made to -- a pre-existing problem that does not belong to this work. Two properties of SYSCALL/SYSRET shape the child's frame. The instruction leaves the caller's RIP and RFLAGS in RCX and R11 rather than on a stack, so they are moved into the RIP and RFLAGS slots of the interrupt frame the child is resumed from; and the hardware never records the caller's CS and SS at all, SYSRETQ reconstructing them from IA32_STAR, so the child's are filled in with the user code and data selectors at RPL 3. The frame is therefore not copied wholesale: the extended state and the general registers are inherited, while the segment registers and the thread pointer stay as up_initial_state() left them, the child's stack being a fresh allocation the parent's FS base does not describe. x86_64_fork_relocfp() is new and is not optional here. A function returns with `leave', which feeds the frame pointer into the stack pointer, so relocating only the RBP the child resumes with gets it exactly one frame: the next return loads a saved RBP still pointing into the parent's stack. With that in place ARCH_X86_64 can select ARCH_HAVE_VFORK unconditionally. Build-verified on qemu-intel64:knsh_romfs and qemu-intel64:ostest. NuttX on qemu-intel64 requires tsc-deadline and pcid, which TCG does not implement, so it cannot be run on this host. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli --- Documentation/guides/fork_vfork_migration.rst | 17 +- arch/Kconfig | 2 +- arch/x86_64/include/intel64/irq.h | 12 + arch/x86_64/src/common/x86_64_fork.c | 439 +++++++++++++++--- arch/x86_64/src/common/x86_64_syscall.c | 35 +- 5 files changed, 446 insertions(+), 59 deletions(-) diff --git a/Documentation/guides/fork_vfork_migration.rst b/Documentation/guides/fork_vfork_migration.rst index 63e5f682a0373..fa9a305d3c8e6 100644 --- a/Documentation/guides/fork_vfork_migration.rst +++ b/Documentation/guides/fork_vfork_migration.rst @@ -153,7 +153,7 @@ exception frame when it traps -- ``xcp.sregs`` is the field that exists for this -- and build the child from that instead, while a kernel thread that calls the entry point directly still takes the ordinary path. -Three architectures do it, and they are worth copying: +Four architectures do it, and they are worth copying: * RISC-V: ``riscv_swint.c`` stores the frame in ``xcp.sregs``, and ``riscv_fork.c`` rebuilds the child from it. @@ -177,6 +177,21 @@ Three architectures do it, and they are worth copying: ``arm_fork()`` runs, its PC, CPSR and SP are the kernel's; the caller's are read from where ``arm_syscall()`` put them -- ``syscall[0].sysreturn``, ``syscall[0].cpsr`` and ``ustkptr``. +* x86_64: ``x86_64_syscall()`` stores the frame in ``xcp.sregs``, and + ``x86_64_fork()`` dispatches to ``x86_64_fork_syscall()`` or + ``x86_64_fork_direct()``. The discriminator here is ``xcp.sregs`` itself + being non-NULL, because raising ``TCB_FLAG_SYSCALL`` would also defer signal + actions -- something x86_64 has never done and its kernel-build signal path + does not currently survive. Two properties of ``SYSCALL``/``SYSRET`` shape + the child's frame: the instruction leaves the caller's RIP and RFLAGS in + RCX and R11 rather than on a stack, so they have to be moved into the RIP + and RFLAGS slots of the interrupt frame the child is resumed from; and the + hardware never records the caller's CS and SS at all -- ``SYSRETQ`` + reconstructs them from ``IA32_STAR`` -- so the child's have to be filled in + with the user code and data selectors at RPL 3. For the same reason the + saved frame is not copied wholesale: only the extended state and the + general registers are inherited, and the segment registers and thread + pointer come from the frame ``up_initial_state()`` built for the child. Nothing else is required: the ``up_fork()`` entry point and the libc wrapper are already there and become live automatically. diff --git a/arch/Kconfig b/arch/Kconfig index 3944e83d5a5f6..60039f1991e4e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -147,7 +147,7 @@ config ARCH_X86_64 select PCI_LATE_DRIVERS_REGISTER if PCI select ARCH_TOOLCHAIN_GNU select ARCH_HAVE_BACKTRACE - select ARCH_HAVE_VFORK if !BUILD_KERNEL + select ARCH_HAVE_VFORK select ARCH_HAVE_SETJMP select ARCH_HAVE_PERF_EVENTS select ARCH_HAVE_POWEROFF diff --git a/arch/x86_64/include/intel64/irq.h b/arch/x86_64/include/intel64/irq.h index 24fd088d29cb7..84a04a4436fef 100644 --- a/arch/x86_64/include/intel64/irq.h +++ b/arch/x86_64/include/intel64/irq.h @@ -544,6 +544,18 @@ struct xcptcontext uint64_t *regs; +#ifdef CONFIG_LIB_SYSCALL + /* The register context of the user code that is currently in a system + * call, as x86_64_syscall_entry() saved it on the kernel stack. This is + * what the caller of a system call was doing, as opposed to xcp.regs, + * which during a system call describes the kernel side of it. + * x86_64_fork() needs it to build a child from the caller rather than + * from the stub. + */ + + uint64_t *sregs; +#endif + #ifdef CONFIG_ARCH_ADDRENV # ifdef CONFIG_ARCH_KERNEL_STACK /* In this configuration, all syscalls execute from an internal kernel diff --git a/arch/x86_64/src/common/x86_64_fork.c b/arch/x86_64/src/common/x86_64_fork.c index 05cd26016e8f0..ef5cfb83e4519 100644 --- a/arch/x86_64/src/common/x86_64_fork.c +++ b/arch/x86_64/src/common/x86_64_fork.c @@ -35,6 +35,7 @@ #include #include +#include #include #include "x86_64_fork.h" @@ -42,57 +43,156 @@ #include "sched/sched.h" /**************************************************************************** - * Public Functions + * Pre-processor Definitions ****************************************************************************/ +#ifdef CONFIG_LIB_SYSCALL + +/* Requested privilege level 3 in a segment selector. A caller that reached + * here through the `syscall' instruction was in user mode, and SYSRETQ is + * going to put it back there with the selectors IA32_STAR describes -- see + * x86_64_fork_syscall(). + */ + +# define X86_GDT_RPL_USER 3 + +#endif + /**************************************************************************** - * Name: x86_64_fork + * Private Functions + ****************************************************************************/ + +/**************************************************************************** + * Name: x86_64_fork_stacktop * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The high end of the part of a task's stack that a fork copies: the + * bottom of the register save area that up_initial_state() reserved at the + * very top of the stack. The save area itself is not stack and must not + * be copied over -- it is where the child's own resume frame is built. * - * The overall sequence is: + * Input Parameters: + * tcb - The task whose stack is in question * - * 1) User code calls fork(). fork() collects context information and - * transfers control up x86_64_fork(). - * 2) x86_64_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) x86_64_fork() provides any additional operating context. It must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) x86_64_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * Returned Value: + * The address one past the last byte of stack that is copied. * - * nxtask_abort_fork() may be called if an error occurs between steps 3 and - * 6. + ****************************************************************************/ + +static uint64_t x86_64_fork_stacktop(struct tcb_s *tcb) +{ + return (uint64_t)XCP_ALIGN_DOWN((uintptr_t)tcb->stack_base_ptr + + tcb->adj_stack_size - XCPTCONTEXT_SIZE); +} + +/**************************************************************************** + * Name: x86_64_fork_reloc + * + * Description: + * Carry one address from the parent's stack over to the child's copy of + * it. Addresses outside the copied region are returned unchanged: they + * point somewhere the child shares with the parent, or somewhere that has + * no counterpart at all. + * + * Input Parameters: + * addr - The address to relocate + * rsp - The parent's stack pointer where the primitive was called, + * which is the low end of the region that was copied + * stacktop - The high end of the region that was copied + * offset - The distance from the parent's stack to the child's copy + * + * Returned Value: + * The relocated address. + * + ****************************************************************************/ + +static uint64_t x86_64_fork_reloc(uint64_t addr, uint64_t rsp, + uint64_t stacktop, uint64_t offset) +{ + if (addr >= rsp && addr < stacktop) + { + return addr + offset; + } + + return addr; +} + +/**************************************************************************** + * Name: x86_64_fork_relocfp + * + * Description: + * Relocate the saved frame-pointer chain inside the child's copy of the + * parent's stack. + * + * This is x86_64-specific and it is not optional. A function returns here + * with `leave', which is `mov %rbp,%rsp' followed by `pop %rbp': the + * frame pointer feeds the *stack* pointer. Relocating only the RBP the + * child resumes with therefore gets it exactly one frame; the moment it + * returns through the next one it loads a saved RBP that still points + * into the parent's stack, and from then on the child runs on the + * parent's stack. It looks like the child is working -- it is even at the + * right offset -- until something returns through a slot the parent has + * since reused. + * + * The other architectures with this fork path do not need it: they return + * through a link register, so a stale frame pointer spoils a backtrace and + * nothing else. + * + * The walk stops at the first link that leaves the copied region -- the + * outermost frame's saved RBP does -- and refuses to move backwards, so a + * corrupt chain terminates it rather than looping. + * + * Input Parameters: + * rbp - The parent's frame pointer where the primitive was called + * rsp - The parent's stack pointer, the low end of the copied region + * stacktop - The high end of the copied region + * offset - The distance from the parent's stack to the child's copy + * + ****************************************************************************/ + +static void x86_64_fork_relocfp(uint64_t rbp, uint64_t rsp, + uint64_t stacktop, uint64_t offset) +{ + while (rbp >= rsp && rbp < stacktop) + { + uint64_t *slot = (uint64_t *)(rbp + offset); + uint64_t next = *slot; + + if (next <= rbp || next >= stacktop) + { + break; + } + + *slot = next + offset; + rbp = next; + } +} + +/**************************************************************************** + * Name: x86_64_fork_direct + * + * Description: + * Clone a caller that reached up_fork() by an ordinary function call, so + * that the register snapshot fork.S took describes the caller itself. + * That is the case in a flat build, and for a kernel thread in any build. + * + * The child has no exception frame to inherit, so one is synthesised: it + * resumes at the caller's return address, in the caller's own segments, + * with the callee-saved registers the caller had. * * Input Parameters: * vfork - true for vfork(), false for fork() - * context - Caller context information saved by up_fork() + * parent - The calling task's TCB + * context - Caller context information saved by fork.S * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * The process ID of the child, or ERROR on failure. * ****************************************************************************/ -pid_t x86_64_fork(bool vfork, const struct fork_s *context) +static pid_t x86_64_fork_direct(bool vfork, struct tcb_s *parent, + const struct fork_s *context) { - struct tcb_s *parent = this_task(); struct tcb_s *child; uint64_t newsp; uint64_t newfp; @@ -120,15 +220,13 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) sinfo("TCBs: Parent=%p Child=%p\n", parent, child); - /* How much of the parent's stack was utilized? The ARM uses - * a push-down stack so that the current stack pointer should - * be lower than the initial, adjusted stack pointer. The - * stack usage should be the difference between those two. + /* How much of the parent's stack was utilized? x86_64 uses a push-down + * stack so that the current stack pointer should be lower than the + * initial, adjusted stack pointer. The stack usage should be the + * difference between those two. */ - stacktop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)parent->stack_base_ptr + - parent->adj_stack_size - - XCPTCONTEXT_SIZE); + stacktop = x86_64_fork_stacktop(parent); DEBUGASSERT(stacktop > context->rsp); stackutil = stacktop - context->rsp; @@ -161,25 +259,17 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) * effort: the child is entitled to use them, and it does. */ - newtop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)child->stack_base_ptr + - child->adj_stack_size - - XCPTCONTEXT_SIZE); - - newsp = newtop - stackutil; + newtop = x86_64_fork_stacktop(child); + newsp = newtop - stackutil; memcpy((void *)newsp, (const void *)context->rsp, stackutil); /* Was there a frame pointer in place before? */ - if (context->rbp >= context->rsp && context->rbp < stacktop) - { - uint32_t frameutil = stacktop - context->rbp; - newfp = newtop - frameutil; - } - else - { - newfp = context->rbp; - } + newfp = x86_64_fork_reloc(context->rbp, context->rsp, stacktop, + newtop - stacktop); + x86_64_fork_relocfp(context->rbp, context->rsp, stacktop, + newtop - stacktop); sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 " RBP:%08" PRIx64 "\n", stacktop, context->rsp, context->rbp); @@ -213,3 +303,242 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) return nxtask_start_fork(child, vfork); } + +#ifdef CONFIG_LIB_SYSCALL + +/**************************************************************************** + * Name: x86_64_fork_syscall + * + * Description: + * Clone a caller that reached up_fork() through a system call. The + * register snapshot fork.S took is useless here: it describes the + * kernel-side stub, so a child built from it would resume at a kernel + * address on a kernel stack. What the caller was actually doing is the + * frame x86_64_syscall_entry() saved and x86_64_syscall() recorded in + * xcp.sregs; the child is built from that. + * + * The child therefore returns from the very same `syscall' instruction as + * the parent, in user mode, differing only in that it sees 0 as the return + * value and runs on its own stack. + * + * Two details of the SYSCALL/SYSRET pair shape this: + * + * 1. `syscall' does not save the caller's RIP and RFLAGS on a stack; it + * leaves them in RCX and R11, which is where the saved frame has them. + * The child is resumed by IRETQ (x86_64_fullcontextrestore()), so they + * have to be moved into the RIP and RFLAGS slots of its frame. + * 2. The hardware never tells the kernel which CS and SS the caller had -- + * SYSRETQ reconstructs them from IA32_STAR -- so those slots of the + * saved frame hold nothing, and the child's have to be filled with the + * selectors SYSRETQ would have produced, which is where the parent is + * about to return to. + * + * Everything the frame does hold -- the general registers and the extended + * (FPU/SSE) state -- is inherited. Everything it does not is taken from + * the frame up_initial_state() built for the child, so that the child + * keeps its own segment registers and, importantly, its own thread + * pointer: the child's stack is a fresh allocation at a different virtual + * address, so the parent's FS base does not describe it. + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * parent - The calling task's TCB + * + * Returned Value: + * The process ID of the child, or ERROR on failure. + * + ****************************************************************************/ + +static pid_t x86_64_fork_syscall(bool vfork, struct tcb_s *parent) +{ + uint64_t *sregs = parent->xcp.sregs; + struct tcb_s *child; + uint64_t newsp; + uint64_t newtop; + uint64_t offset; + uint64_t stacktop; + uint64_t stackutil; + uint64_t rsp; + uint64_t rip; + + DEBUGASSERT(sregs != NULL); + + /* Where the caller was and what it was doing */ + + rsp = sregs[REG_RSP]; + rip = sregs[REG_RCX]; + + sinfo("syscall frame [%p]: RSP:%08" PRIx64 " RIP:%08" PRIx64 "\n", + sregs, rsp, rip); + + /* Allocate and initialize a TCB for the child task. The child resumes at + * the instruction after the `syscall', which is where the parent resumes + * too. + */ + + child = nxtask_setup_fork((start_t)rip, vfork); + if (!child) + { + serr("ERROR: nxtask_setup_fork failed\n"); + return (pid_t)ERROR; + } + + sinfo("TCBs: Parent=%p Child=%p\n", parent, child); + + /* Give the child the part of the parent's stack that is in use, copied to + * the same place in its own stack. The copy is aligned with the top of + * each stack rather than the bottom, so a single offset carries any + * address in the copied region from one to the other. + */ + + stacktop = x86_64_fork_stacktop(parent); + DEBUGASSERT(stacktop > rsp); + stackutil = stacktop - rsp; + + newtop = x86_64_fork_stacktop(child); + newsp = newtop - stackutil; + offset = newtop - stacktop; + + memcpy((void *)newsp, (const void *)rsp, stackutil); + + sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", stacktop, rsp); + sinfo("New stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", newtop, newsp); + + /* Inherit the parent's extended (FPU/SSE) state, which + * x86_64_syscall_entry saved at the front of the frame, exactly where + * the child's belongs. + */ + + memcpy(child->xcp.regs, sregs, XCPTCONTEXT_XMM_AREA_SIZE); + + /* Inherit the general registers. RCX and R11 are included deliberately: + * SYSRETQ leaves the return address in RCX and RFLAGS in R11, so the + * parent resumes with those values and the child must too. + */ + + child->xcp.regs[REG_RBX] = sregs[REG_RBX]; + child->xcp.regs[REG_R8] = sregs[REG_R8]; + child->xcp.regs[REG_R9] = sregs[REG_R9]; + child->xcp.regs[REG_R10] = sregs[REG_R10]; + child->xcp.regs[REG_R11] = sregs[REG_R11]; + child->xcp.regs[REG_R12] = sregs[REG_R12]; + child->xcp.regs[REG_R13] = sregs[REG_R13]; + child->xcp.regs[REG_R14] = sregs[REG_R14]; + child->xcp.regs[REG_R15] = sregs[REG_R15]; + child->xcp.regs[REG_RCX] = sregs[REG_RCX]; + child->xcp.regs[REG_RDX] = sregs[REG_RDX]; + child->xcp.regs[REG_RSI] = sregs[REG_RSI]; + child->xcp.regs[REG_RDI] = sregs[REG_RDI]; + + /* The frame pointer moves with the stack it points into */ + + child->xcp.regs[REG_RBP] = x86_64_fork_reloc(sregs[REG_RBP], rsp, + stacktop, offset); + x86_64_fork_relocfp(sregs[REG_RBP], rsp, stacktop, offset); + + /* Build the interrupt frame the child is resumed from. RIP and RFLAGS + * come out of RCX and R11, and the selectors are the ones SYSRETQ derives + * from IA32_STAR: CS is the user code segment and SS the user data + * segment, both at RPL 3. See x86_64_cpu_priv_set(), which programs + * IA32_STAR. + */ + + child->xcp.regs[REG_RAX] = 0; + child->xcp.regs[REG_RIP] = rip; + child->xcp.regs[REG_RFLAGS] = sregs[REG_R11]; + child->xcp.regs[REG_RSP] = newsp; + child->xcp.regs[REG_CS] = X86_GDT_USERCODE_SEL | X86_GDT_RPL_USER; + child->xcp.regs[REG_SS] = X86_GDT_USERDATA_SEL | X86_GDT_RPL_USER; + +#ifdef CONFIG_ARCH_KERNEL_STACK + /* The child's own user stack pointer, for the signal dispatch path */ + + child->xcp.ustkptr = (uintptr_t *)newsp; +#endif + + /* And, finally, start the child task. On a failure, nxtask_start_fork() + * will discard the TCB by calling nxtask_abort_fork(). + */ + + return nxtask_start_fork(child, vfork); +} + +#endif /* CONFIG_LIB_SYSCALL */ + +/**************************************************************************** + * Public Functions + ****************************************************************************/ + +/**************************************************************************** + * Name: x86_64_fork + * + * Description: + * The common x86_64 worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. + * + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). The libc wrapper enters + * up_fork(), which collects context information and transfers control + * to x86_64_fork(). + * 2) x86_64_fork() calls nxtask_setup_fork(). + * 3) nxtask_setup_fork() allocates and configures the child task's TCB. + * This consists of: + * - Allocation of the child task's TCB. + * - Initialization of file descriptors and streams + * - Configuration of environment variables + * - Allocate and initialize the stack + * - Setup the input parameters for the task. + * - Initialization of the TCB (including call to up_initial_state()) + * 4) x86_64_fork() provides any additional operating context. It must: + * - Initialize special values in any CPU registers that were not + * already configured by up_initial_state() + * 5) x86_64_fork() then calls nxtask_start_fork() + * 6) nxtask_start_fork() then executes the child thread. + * + * nxtask_abort_fork() may be called if an error occurs between steps 3 and + * 6. + * + * Everything above is common to the two ways this can be reached, which + * differ only in where the caller's registers are to be found -- see + * x86_64_fork_direct() and x86_64_fork_syscall(). + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by fork.S + * + * Returned Value: + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. + * + ****************************************************************************/ + +pid_t x86_64_fork(bool vfork, const struct fork_s *context) +{ + struct tcb_s *parent = this_task(); + +#ifdef CONFIG_LIB_SYSCALL + /* A non-NULL xcp.sregs means a system call is in progress: + * x86_64_syscall() publishes the caller's frame there for the duration of + * the call and nowhere else. So this was reached from a kernel-side stub, + * and the caller to clone is the user task that trapped, not the code that + * called into fork.S. + * + * arm64 and RISC-V discriminate on TCB_FLAG_SYSCALL instead. x86_64 + * cannot: that flag also defers signal actions, which x86_64 has never + * done and which its kernel-build signal path does not currently survive + * -- see the note in x86_64_syscall(). xcp.sregs says exactly what is + * needed here and means nothing to anyone else. + */ + + if (parent->xcp.sregs != NULL) + { + return x86_64_fork_syscall(vfork, parent); + } +#endif + + return x86_64_fork_direct(vfork, parent, context); +} diff --git a/arch/x86_64/src/common/x86_64_syscall.c b/arch/x86_64/src/common/x86_64_syscall.c index 4e92a68641026..651d0d28a44f8 100644 --- a/arch/x86_64/src/common/x86_64_syscall.c +++ b/arch/x86_64/src/common/x86_64_syscall.c @@ -301,15 +301,42 @@ uint64_t *x86_64_syscall(uint64_t *regs) #ifdef CONFIG_LIB_SYSCALL int nbr = cmd - CONFIG_SYS_RESERVED; syscall_stub_t stub = (syscall_stub_t)g_stublookup[nbr]; - -#ifdef CONFIG_ARCH_KERNEL_STACK struct tcb_s *rtcb = nxsched_self(); + uint64_t *sregs; +#ifdef CONFIG_ARCH_KERNEL_STACK /* Store reference to user RSP for signals */ rtcb->xcp.saved_ursp = regs[REG_RSP]; #endif + /* Publish the caller's register context. up_fork() has to clone + * the caller rather than the stub that is about to invoke it, and + * this frame is the only description of it -- see + * x86_64_fork_syscall(), which also takes a non-NULL xcp.sregs as + * its "reached here through a system call" discriminator. + * + * It is saved and restored rather than simply set and cleared: + * x86_64_syscall_entry() has an explicit path for a nested system + * call, and when the inner one returns the outer one must still be + * described by its own frame. + * + * Note what is deliberately *not* done here. arm64 and RISC-V + * also raise TCB_FLAG_SYSCALL across the stub call, which defers + * any signal action until the system call returns. x86_64 has + * never set it, and making it do so is not free: the deferred + * action then has to be picked up by nxsig_unmask_pendingsignal() + * on the way out, and the signal dispatch path of an x86_64 kernel + * build does not survive that today -- it faults in + * x86_64_syscall_entry()'s return path with RSP == 0. That is a + * pre-existing bug in a configuration nothing has exercised, and + * fixing it does not belong to the fork/vfork work; so this + * records the frame and changes nothing else. + */ + + sregs = rtcb->xcp.sregs; + rtcb->xcp.sregs = regs; + /* Re-enable interrupts if enabled before. * Current task RFLAGS are stored in R11. */ @@ -322,6 +349,10 @@ uint64_t *x86_64_syscall(uint64_t *regs) /* Call syscall function and store return value in RAX register */ regs[REG_RAX] = stub(nbr, arg1, arg2, arg3, arg4, arg5, arg6); + + /* The system call is now done */ + + rtcb->xcp.sregs = sregs; #else svcerr("ERROR: Bad SYS call: %" PRId32 "\n", cmd); #endif