Skip to content

Add DotRecast-based navigation subsystem - #335

Open
Acissathar wants to merge 68 commits into
ProwlEngine:mainfrom
Acissathar:NavMesh-Single-Representation
Open

Add DotRecast-based navigation subsystem#335
Acissathar wants to merge 68 commits into
ProwlEngine:mainfrom
Acissathar:NavMesh-Single-Representation

Conversation

@Acissathar

Copy link
Copy Markdown
Contributor

Navigation: NavMesh, agents, and runtime rebuilding (DotRecast)

Disclaimer is that this was mostly a claude effort to get ported over and integrated. I've tested it with my local project (which I also used to help drive some minor performance tweaks like skipping empty tiles and the rasterizer to eek out a bit less allocations), but that project is mostly flat with runtime modified terrain and NavMesh rebakes so I didn't fully explore everything beyond what for the little gif snippets I created with the components.

Also included is a minor fix to fields rendering incorrectly. Some fields when serialized would overlap the borders and have the text be a lighter white while popping out of place with their data. This tweak ensures it renders as expected (so like all other files).


Adds a navigation subsystem to Prowl (navmesh baking, pathfinding queries, crowd-driven agents, obstacle carving, and runtime mesh mutation) built on DotRecast (zlib, from NuGet). Prowl owns the abstraction layer; DotRecast stays an implementation detail you can reach past when you need to, the same way PhysicsWorld exposes Jitter.

The API mirrors UnityEngine.AI closely enough that Unity navigation code and muscle memory transfer, with Prowl naming conventions and a few additions Unity doesn't offer (partial rebuilds, declared bake bounds, thread-safe queries).

Every surface is backed by a tile cache, so carving, links, and agents work everywhere with nothing to configure and no combination that silently doesn't work.

NavMeshOverview

Contents


Quick start

  1. Add a NavMesh Surface to any GameObject in the scene.
  2. Press Bake NavMesh in the inspector. The result is saved next to the scene as a .navmesh asset and assigned to the surface.
  3. Add a NavMesh Agent to a character and call agent.SetDestination(target).
Screenshot 2026-08-04 211902

Components

NavMesh Surface

Bakes a navmesh for one agent type and registers it with the scene. The baked result is a standalone .navmesh asset, so it survives scene reloads and ships with the project.

Field Meaning
AgentTypeId Which agent type this navmesh is built for (dropdown of names from project settings). Agents only walk navmeshes of their own type.
CollectObjects All (whole scene), Volume (a box you define), or Children (this object's hierarchy).
Center / Size The volume, in Volume mode. Also acts as a declared bake extent — see Beyond Unity.
Layers Only objects on these layers contribute geometry.
UseGeometry Voxelize RenderMeshes or PhysicsColliders. Terrain contributes in both modes.
NavMeshData The baked asset. Assigned by baking, or point it at an existing .navmesh.
AlwaysShowNavMesh Draw the walkable overlay even when the surface isn't selected — the only way to watch carving while playing, since entering play mode clears the selection.
AdvancedDefaultArea Area stamped on walkable geometry that nothing overrides.
AdvancedBuildOverrides Voxel size, tile size, min region area, edge max error, and the three span filters. Most bakes never touch these.

Anything on or under a NavMeshAgent is skipped by collection regardless of these settings — agents walk the navmesh rather than forming it, so baking an agent's collider or renderer would stamp a permanent hole wherever it happened to stand. The whole subtree is excluded because agent visuals and colliders normally hang off child objects, and it is the presence of the component that excludes rather than its enabled state, so a bake never depends on when an agent was last toggled.

Runtime methods: BuildNavMesh(), BuildNavMeshAsync(), ApplyNavMeshData(), RefreshRegistration(), RebuildTiles(AABB) and overloads, RebuildTilesAsync() / ApplyRebuiltTiles(), RebuildLinkTiles(), CollectSources() / CollectVolumes() / CollectLinks().

Screenshot 2026-08-04 211928

NavMesh Agent

Moves a character along the navmesh using DotRecast's crowd simulation. Give it a destination and it steers there, avoiding other agents.

Field Meaning
AgentTypeId Which navmesh this agent walks on.
Radius / Height / BaseOffset Physical envelope for avoidance, and the offset between the mesh surface and the Transform.
Speed / AngularSpeed / Acceleration Movement limits.
StoppingDistance / AutoBraking How it finishes a path.
ObstacleAvoidanceQuality Avoidance quality, None → High. Higher costs more CPU.
AvoidancePriority 0 = most important, 99 = least. Mapped to crowd separation weight.
Separation Push away from neighbours. Turn off for units that should pack tightly or walk single file.
CollisionQueryRange / PathOptimizationRange Steering scan distances. 0 derives them from the radius. On tile/corridor maps, tuning CollisionQueryRange down toward the corridor width stops avoidance oscillation.
AreaMask Which areas this agent may traverse (multi-select of area names).
AutoRepath Re-path automatically when the navmesh changes underneath.
UpdatePosition / UpdateRotation Whether the crowd drives the Transform. Turn UpdatePosition off to drive a Rigidbody or CharacterController from DesiredVelocity yourself.

Key API: SetDestination, Destination, IsStopped, RemainingDistance, PathPending, HasPath, PathStatus, Velocity, DesiredVelocity, NextPosition, SteeringTarget, Warp, Move, ResetPath, SetPath, SetAreaCost / GetAreaCost, CalculatePath, Raycast, FindClosestEdge, SamplePathPosition, IsOnOffMeshLink, CurrentOffMeshLinkData, RefreshParams.

The Unity arrival idiom works as-is:

if (!agent.PathPending && agent.RemainingDistance <= agent.StoppingDistance)
    // arrived
Screenshot 2026-08-04 211951

NavMesh Modifier

Changes how one GameObject (and by default its children) contributes to bakes.

Field Meaning
IgnoreFromBuild Exclude this object's geometry from bakes entirely — it still renders and collides.
OverrideArea + Area Stamp this object's geometry with a specific area instead of the surface default.
ApplyToChildren Also apply to descendants. A child's own modifier always wins.
AffectAllAgentTypes / AffectedAgentTypeIds Restrict the modifier to specific agent types.

Resolution rules: an object's own modifier always wins; otherwise the nearest ancestor whose modifier applies to children. Modifiers that are disabled or scoped to a different agent type are transparent — the search continues past them to higher ancestors.

Useful for decorative geometry that shouldn't shape the mesh, and for construction previews that become real (flip IgnoreFromBuild off and rebuild the affected tiles).

Screenshot 2026-08-04 212007

NavMesh Modifier Volume

Stamps an area over a world region, independent of which objects the geometry came from.

Field Meaning
Center / Size The box, local to the Transform (rotation and scale are honoured).
Area Area applied inside. Not Walkable erases walkability — punches a real hole.
AffectAllAgentTypes / AffectedAgentTypeIds Agent-type scoping.

A volume only re-marks surface that geometry produced; it never creates walkable surface.

Screenshot 2026-08-04 212019

NavMesh Link

Connects two navmesh positions that aren't walkably connected — a jump, a drop, a ladder.

Field Meaning
StartPoint / EndPoint Endpoints, local to the Transform.
Width 0 = a single crossing point. Wider links emit parallel connections across the span so an agent enters at the nearest one rather than queueing through the middle.
Bidirectional Traversable both ways.
Area The link's area (default Jump). Traversal cost comes from this area's cost, and agents whose mask excludes it won't use the link.
Activated Whether the link is usable.
AutoUpdatePosition Follow Transform movement at runtime (each move pays a partial rebuild).
AutoRebuild Rebuild the affected tiles automatically when the link changes. Turn off in games that drive rebuilds themselves with explicit geometry.

Agents traverse links automatically as part of pathing and expose IsOnOffMeshLink / CurrentOffMeshLinkData mid-hop.

Links ride on the asset rather than being baked into tile bytes, and are re-injected every time a tile is contoured — anything written into a tile once would be regenerated away the moment an obstacle carved near it. That also makes editing a link cheap: the compressed layers are untouched and only the affected tiles are re-contoured, with no re-voxelization at all (NavMeshSurface.RebuildLinkTiles).

NavMeshLink Screenshot 2026-08-04 212102

NavMesh Obstacle

Blocks agents while enabled — a parked vehicle, a dropped crate, a placed building.

Field Meaning
Shape Box (oriented, yaw) or Capsule (carved as a cylinder).
Center / Size / Radius / Height Shape dimensions, scaled by the Transform.
Carve On: cut a hole so paths route around it. Off: leave the mesh alone and have agents steer around it locally.
CarveOnlyStationary Lift the carve while moving, re-apply once settled.
CarvingMoveThreshold / CarvingTimeToStationary Movement tolerance and settle time.

Affected tiles rebuild incrementally over the following frames, so carve cost amortizes off the critical path rather than spiking one frame.

Carve picks between the same two modes Unity has, and both work on any surface. On, the obstacle cuts a hole and pathfinding itself routes around it. Off, the obstacle joins each crowd as an immovable neighbour: the mesh is untouched and paths still lead through it, but agents steer around it locally — that costs nothing when the obstacle moves (the blocker is simply repositioned), which makes it the right choice for something in motion, since a patrolling vehicle re-cutting tiles every frame is the case carving handles badly.

The hole is widened by the navmesh's agent radius. A navmesh records where an agent's centre may be, not where its body fits — a bake pulls the mesh back from every wall by that much — so a hole cut to the obstacle's exact footprint would let agents walk their centre onto its surface and stand half inside it.

Carving runs in the editor as well as in play, so positioning a building shows the hole it will cut without entering play mode — only the live navmesh changes, since obstacles never touch the baked asset. Velocity mode stays play-only, being crowd steering. Turn on the surface's AlwaysShowNavMesh to watch either one.

Obstacles never contribute bake geometry, in either mode: they block at runtime from wherever the object actually is, and voxelizing one would freeze a hole where it happened to be standing at bake time. Exclusion is by presence of the component rather than its enabled state — the same rule agents use — so bake output can never depend on when something was last toggled. An object meant to be permanent map geometry should not carry the component at all.

NavMeshObstacle image

Runtime API

NavMesh — static facade mirroring Unity's, forwarding to the current scene: CalculatePath, SamplePosition, Raycast, FindClosestEdge, CalculateTriangulation, AddNavMeshData / RemoveNavMeshData, GetAreaFromName, GetAreaCost / SetAreaCost.

Scene.Navigation (NavMeshWorld) — the per-scene owner, sitting beside Scene.Physics. Holds registered navmeshes, the query layer, and one crowd per agent type. Advanced access: GetNativeCrowd(agentTypeId), TryRentQuery(...), MutateTileCache, SetObstacleAvoidanceParams, CrowdMaxAgentRadius, TileCacheMaxObstacles, NavMeshChanged / PreUpdate events.

NavMeshData — the serializable baked asset (.navmesh), independent of any scene, so procedural games can build one at runtime and register it without an editor bake.

NavMeshQueryFilter — area mask + per-area cost overrides for a single query.

Queries are thread-safe. Each takes a pooled Detour query under a read lock, so gameplay code can path-find from worker threads; mesh mutations take the write lock.


Project settings

Project Settings → Navigation, with two tabs mirroring Unity's:

  • Agents — the agent-type table (name, radius, height, slope, climb). Humanoid (id 0) is built in and can't be removed. Ids are persistent, so renaming or removing a type never silently re-points existing surfaces and agents at a different one.
  • Areas — 32 named areas with per-area path costs, Tags-style rather than a fixed list. Walkable, Not Walkable, and Jump are built in. Removing an area clears its slot rather than shifting later areas, because masks and baked polygons reference areas by index.

Both tables are written to NavigationSettings.yaml for built players and restored on startup, and both have code-side defaults so headless and procedural use needs no settings file at all.


Editor integration

  • Surface inspector — basic fields up top, an Advanced foldout (collapsed by default) holding the default area and all Recast tuning, with a Reset Advanced To Defaults button. Bake NavMesh writes a .navmesh asset into a scene-named folder next to the scene (Assets/Scenes/<Scene>/<Scene> NavMesh.navmesh), mirroring the lightmapper's convention.
  • Scene overlay — selected surfaces draw the walkable surface coloured per area, plus the bake bounds. Cached and invalidated when the navmesh reports a change, so it's cheap to leave on and it follows runtime carving live.
  • Attributes[NavMeshArea], [NavMeshAreaMask], and [NavMeshAgentType] render int and List<int> fields as name dropdowns and multi-selects, so users pick "Walkable" and "Humanoid" rather than typing indices. Usable by game code too.
  • Asset importer.navmesh files import as NavMeshData.

Two general-purpose inspector attributes were added alongside, since navigation needed them and nothing equivalent existed: [EnableIf(member)] (greys a field out while a bool is false — the complement to the existing [ShowIf]) and [InspectorName(text)] (overrides the displayed name of a field or enum member without renaming the code symbol).


Design notes

One representation, always carve-capable

Recast offers two ways to store a navmesh: finished Detour tiles, and a tile cache of compressed voxel layers that can be re-contoured on demand. Only the second can be carved. Exposing that as a user-facing choice meant every feature had to answer "which one?", and the answers diverged — obstacles inert on one, links stored differently, different rebuild entry points, different tile-size limits.

So there is no choice: every surface bakes compressed layers. What the finished-tile path was for — not paying for a cache you never carve — is preserved as an invisible optimisation instead. An instance is only handed to the per-frame cache pump when something has actually queued work into it, so a surface nothing ever carves costs nothing per frame after load, and its tiles are finished Detour tiles that stay that way.

Why a custom rasterizer (NavMeshRasterizer)

This is the largest "why did you reimplement that?" in the PR, so: it's a ~300-line port of DotRecast's triangle rasterization (zlib, attributed in the file header) that exists for one reason — the upstream port has the span pool wired up but never uses it.

C++ Recast allocates heightfield spans from a pooled freelist. DotRecast faithfully ports RcSpanPool and the freelist machinery, but its public AddSpan path allocates every span with new instead, leaving the pool as dead code. Since Prowl bakes tiles at gameplay frequency (partial rebuilds on destructible maps), that allocation lands in the frame budget.

The port restores the pooled path, returns merge-removed spans to the freelist, and folds the walkable-slope test into rasterization so no per-chunk area arrays are allocated. Pool pages are then recycled across tiles per thread, so a bake allocates spans once and never again. Measured on the current path, a region rebuild costs 1.8 KB for a tile no geometry overlaps (the chunky-index check rejects it before any heightfield exists) and ~658 KB for a fully covered one. On bounded bakes of mostly-sealed worlds the great majority of tiles take the first path, which is what keeps full-bake allocation proportional to walkable area rather than world area. A test pins the empty-tile case under a hard 16 KB bound.

Why our own tile loop instead of RcBuilder.BuildTiles

Per-source areas. Geometry is grouped into one triangle soup per resolved area, and each group is rasterized with its own area value — that's how a NavMeshModifier's area survives into the baked polygons. The stock helper stamps a single area across everything. The loop also indexes results by tile so parallel builds produce byte-identical output regardless of scheduling.

Why no dependency on DotRecast.Recast.Toolset

It's demo scaffolding, and its UpdateAreaAndFlags hardcodes an area→flag mapping that conflicts with Prowl's area model. Only Core/Recast/Detour/Detour.Crowd/Detour.TileCache are referenced.

Area model

Prowl exposes 32 areas; Detour stores a 6-bit area id per polygon where 0 is reserved for "not part of the navmesh". So Prowl area i is stored as Detour area i+1, with Not Walkable mapping to the null area so that geometry becomes a genuine obstacle rather than traversable "area 1" polygons. One conversion seam handles this for sources, modifiers, and volumes alike.

Polygon flags are always 1. Filtering happens on the polygon's area via a custom IDtQueryFilter, not on flags, because Detour's stock filter only supports 16 flag bits and we want all 32 areas usable.

Area costs are clamped to ≥ 1

Detour's A* heuristic is straight-line distance, which is only admissible when no traversal is cheaper than distance itself. Costs below 1 silently produce suboptimal paths. To express "agents prefer this area", raise the other areas' costs. This clamp is load-bearing — please don't remove it.

Navmeshes are always tiled

Even single-tile bakes go through the tiled path, because that's what makes RebuildTiles possible. The tile grid is anchored in XZ to the original bake bounds so rebuilt tiles stay aligned with the live mesh; the vertical range unions with current geometry, because Recast clips rasterized spans to the heightfield's vertical extent and new tall geometry would otherwise vanish silently from a rebuild.

Tiles default to 64 voxels and are capped at 255, because a compressed layer header stores tile dimensions in a byte and a wider tile wraps to an empty layer — a navmesh with no polygons at all, from a bake that reported success. Small tiles are also what keep carving cheap: an obstacle change re-contours whole tiles, so tile size is the per-carve cost.

Contours and polygons are built at runtime, not at bake

A bake stores compressed voxel layers; the tile cache turns them into Detour polygons per tile, on demand. That is what makes carving cheap — no re-voxelization — and it means the bake-time settings that only affect contouring and polygonisation aren't exposed, because the cache uses its own fixed parameters for those stages. Heights come from polygon planes rather than a detail mesh, matching Unity's default (its HeightMesh is off).

Small-island culling (MinRegionArea, Unity's) is the exception: regions are built at bake time purely to find the spans to erase, so the islands never reach the layers at all.

Navigation ticks in the variable update

Crowd steering is frame-rate work, not fixed-step physics work, so it runs in Scene.Update before component Updates (gameplay sees fresh agent state), and agents write their Transforms in LateUpdate.

Crowds are per agent type, with per-agent steering filters

One DtCrowd per agent type, each bound to that type's navmesh. A crowd exposes 16 query-filter slots: slot 0 is the shared default, and agents with distinct AreaMask/cost configurations get their own refcounted slot, so per-agent masks and costs affect steering, not just explicit queries. Past 15 distinct configurations per type it warns once and falls back to the default filter.

Agents face where they steer, and skip avoidance with nothing to avoid

Facing follows the pre-avoidance steering vector, not the crowd's actual velocity: the latter carries avoidance and collision corrections that don't shrink with speed, so braking into a goal they come to dominate a small vector's direction and the agent visibly shivers along a dead straight path. Heading is also held once the target is closer than the agent is wide, where what remains of the vector to it is drift.

Separately, velocity-obstacle sampling is skipped entirely for an agent with no neighbours. It picks from a discrete candidate set, so running it against nothing still rounds the chosen velocity and walks the agent centimetres off a straight line — and it is the expensive half of a crowd step.


Beyond Unity

I know that Prowl tries to take inspiration from Unity to make it as close to seamless as possible for transferring, but since we're taking a library directly, we can expose a bit more and open the door to some more functionality:

  • RebuildTiles(AABB) — rebuild only the tiles overlapping a changed region, against scene geometry or caller-supplied geometry. Cost scales with the changed volume, not map size. This is the feature destructible/procedural worlds need, and it has async variants that keep voxelization off the main thread.
  • Declared bake bounds — tell the bake how big the world will become, so a map that grows (drilling, streaming) can add tiles anywhere inside that extent later without a full rebake.
  • Thread-safe queries — path-find from worker threads via a pooled, lock-guarded query API.
  • DotRecast escape hatchesNativeNavMesh, GetNativeCrowd, TryRentQuery, and the TileCache are all reachable, so anything the abstraction doesn't cover is still available rather than walled off.
  • Per-quality avoidance tuning — override DotRecast's avoidance presets per quality level; overrides survive crowd rebinds.

Known limitations and deliberate divergences

  • Off-mesh link traversal is automatic only. AutoTraverseOffMeshLink = false (manual traversal for scripted jump animations) needs to intercept the crowd's animation step — deferred.
  • Links crossing one tile boundary are capped. Detour sizes a tile's link pool when the tile is built and doesn't fully budget connections that leave it or arrive from a neighbour, so past four per tile the pool overflows. Connections are rationed breadth-first — every link keeps one crossing point before any link gets a second — and a link that ends up with none is reported. In practice this only bites when several links crowd the same tile edge.
  • Precision is bounded by voxel size. Navmesh edges and carved holes quantize to the voxel grid, which defaults to agentRadius / 3. The carve is exactly as accurate as the surface it cuts — both are offset identically — but a tighter voxel size is the knob if a hole needs to hug its obstacle more closely.
  • One navmesh per agent type per scene. Unity's additive multi-surface merging is out of scope; it needs tile-grid stitching or Detour-side multi-mesh queries.
  • Skinned meshes are not collected (matches Unity).
  • 3D only, and determinism is not guaranteed across platforms (same as Unity's).
  • AOT: trim analysis is clean; native linking is unverified (no MSVC C++ workload on the dev machine).
  • Area costs bias pathfinding, but Detour's corridor visibility optimization raycasts with the pass filter only, so in wide-open terrain it can re-straighten a route back through expensive-but-passable ground within PathOptimizationRange. Masked areas are immune. This is upstream Detour behaviour, and PathOptimizationRange is the exposed knob.

A Unity-shaped navigation stack built on DotRecast: baked navmesh assets,
runtime queries, crowd-driven agents, off-mesh links, obstacle carving, and
the editor tooling to author and inspect it.

Runtime
- NavMeshData assets and NavMeshBuilder, with region rebuilds that scale with
  the changed area rather than the map size, synchronous or off-thread.
- NavMeshWorld on Scene.Navigation: registered instances, a pooled thread-safe
  query layer, one crowd per agent type, and a demand-driven tile-cache pump
  that costs nothing per frame for surfaces with no queued work.
- Components: NavMeshSurface, NavMeshAgent, NavMeshObstacle (carve or
  velocity-block), NavMeshLink, NavMeshModifier and NavMeshModifierVolume.
- Project-level agent types and 32 Unity-style areas with per-agent costs.
- Our own area-aware rasterizer with pooled heightfield spans, so repeated
  tile bakes on destructible geometry do not churn the heap.

Every surface is tile-cache backed and can carve; there is no representation
for a user to choose. Links ride on the asset and are re-injected as tiles are
contoured, so carving and links coexist. Escape hatches to the underlying
DotRecast objects are exposed, in the spirit of PhysicsWorld exposing Jitter.

Editor
- Surface inspector with bake-to-asset, a .navmesh importer, Navigation project
  settings, area and agent-type drawers, and a scene-view overlay that follows
  runtime carving.

Covered by 127 runtime navigation tests.
@PaperPrototype

Copy link
Copy Markdown
Contributor

Bro is cooking

@PaperPrototype

PaperPrototype commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Would it be possible to reduce the surface area of this PR? Mixing multiple fixes in a single PR can make it harder to merge. Maybe things like the UI fixes could be their own PR and then this PR could be based on those? I did that at one point where one PR depended on another PR. Just an idea but will have to wait for the big man (Wulferis) to see what he says.

@Acissathar

Copy link
Copy Markdown
Contributor Author

Yeah I don't have a problem breaking it down if desired. I think the main core is Surface + Agent related, and then everything else (Link, Obstacles, Modifier, etc) builds off of that with that minor UI tweak throw in AttributeHandlers (but also has a few new attributes).

I just wanted to at least get some visibility out there since I needed it for my project anyways, so I could move past some things in my port but I'm not married to anything here (other than runtime baking/modification to the mesh :p)

@Acissathar

Copy link
Copy Markdown
Contributor Author

For what its worth, tying into the above, if there's a different desired route (previous work to port up instead, different library/implementation approach, etc.) I'm down to help out with that instead.

As mentioned I just needed something navmesh in so I could continue porting over my project, and then ended up spending the weekend to flesh it out so it wasn't just my use case.

@michaelsakharov

Copy link
Copy Markdown
Contributor

This looks great, It will take some time for me to properly review it, I'll try todo it today.

@michaelsakharov michaelsakharov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all thanks so much! this is 90% of the work needed to implement navmesh properly into prowl!

Mostly just some small quirks, bugs and issues to look over,

First up this PR could be split into 3 other ones

  1. ApplyGeneral being dead code is unrelated to Navigation
  2. RangeAttributeHandler is rewritten, its correct, but again would be nice as its own PR it deserves its own Commit
  3. Same with EnableIfAttribute and InspectorNameAttribute, they are new features that deserve their own PR and Commits.

NavMeshWorld.DefaultQueryExtents is a public mutable Float3 read from worker threads during queries. Not atomic, so a write during a query can tear. Typically i wouldnt care but it is on the documented thread safe surface.

This is a huge PR, so its expected to come across a ton of issues, But overall this is pretty good so far, its got all the features, its just cleanup and random quirks here and there. And making a DotRecast fork inside Anthology would simplify a ton because a few of the Workarounds can be fixed in DotRecast directly.

// Velocity-obstacle sampling is the expensive half of a crowd step, and it picks from a
// DISCRETE set of candidate velocities: with nothing in range to dodge, the winner is
// merely the sample nearest the velocity we asked for, and that rounding walks the agent
// a few centimetres sideways off a straight line by the time it arrives. An agent with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"An agent with no neighbours has nothing to avoid".

That I believe is wrong. In DtCrowd.cs:1136-1181 the DT_CROWD_OBSTACLE_AVOIDANCE branch is also the only place navmesh boundary segments get fed into the avoidance query, so agents avoid the edges of a navmesh still.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified so it takes neighbors and boundaries into account

// they are gathered from several metres out, which is many frames of approach.
if (ObstacleAvoidanceQuality != ObstacleAvoidanceType.NoObstacleAvoidance)
{
_agent.option.updateFlags = _agent.nneis > 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing _agent.option.updateFlags in place mutates the params object the crowd owns, bypassing UpdateAgentParameters so it can get reset.

}

/// <summary>Displace the agent by a world-space offset, constrained to the navmesh.</summary>
public void Move(Float3 offset) => Warp(NextPosition + offset);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NavMeshAgent.Move() tears down and rebuilds the crowd agent

public void Move(Float3 offset) => Warp(NextPosition + offset);
// Warp:  crowd.RemoveAgent(_agent);  _agent = crowd.AddAgent(...);  RequestPathTo(...);

In Unity, Move is the per frame API for driving an agent yourself. Here every call drops the path corridor, the local boundary cache, the neighbour set and the move target, then runs FindNearestPoly plus RequestMoveTarget again. Called per frame it is both very expensive and behaviorally broken.

Warp also never clears _arrived, so warping a parked agent leaves it parked, and it writes Transform.Position to the raw unsnapped position one frame before the crowd snaps the agent to the mesh.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move is now properly implemented to be similar to Unity to support nudging it without getting rid of the target.

AddNavMeshMove

NavMeshAgentTypes.ApplyTable(AgentTypes);
}

public override void ResetToDefaults()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesnt reset properly?

private static List<string> CreateDefaultNames() {
    for (int i = 0; i < MaxAreas; i++) names.Add(NavMeshAreas.GetAreaName(i));  // reads LIVE state

CreateDefaultNames and CreateDefaultCosts read the live NavMeshAreas statics, which by that point hold the user's edited table. ResetToDefaults then only overwrites indices 0, 1 and 2. Every user defined area name and every edited cost survives a "reset to defaults".

int tx1 = (int)Math.Floor((worldBounds.Max.X - data.Origin.X) / ts);
int tz0 = (int)Math.Floor((worldBounds.Min.Z - data.Origin.Z) / ts);
int tz1 = (int)Math.Floor((worldBounds.Max.Z - data.Origin.Z) / ts);
for (int tz = tz0; tz <= tz1; tz++)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unclamped tile loop. Measured at 1.9 seconds.
No clamp to the baked grid, no early out for a region that misses it.
NavMeshBuilder.TryPrepareRebuild does this correctly, so the pattern exists.

Measured on a 20x20 floor, tile world size 16, calling the public RebuildLinkTiles:
| 100 | 1 ms |
| 10 000 | 49 ms |
| 100 000 | 1919 ms |
All of it on the calling thread, inside MutateTileCache, holding the instance write lock, so every worker thread query blocks for the duration.

/// physics engine uses (capsules included); mesh colliders reuse the shared physics bake so
/// the triangle extraction cost is paid once per mesh, not per bake.
/// </summary>
public static void CollectCollider(Collider collider, int area, List<NavMeshGeometrySource> results, AABB? bounds = null)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc claims an optimization that is not there,
"mesh colliders reuse the shared physics bake so the triangle extraction cost is paid once per mesh, not per bake."
But the code uses the Mesh.Res.Vertices directly despite the comment.

/// <see cref="NotWalkable"/> to the null area (calling <see cref="ToDetourArea"/> directly on
/// a source/volume area silently resurrects traversable "Not Walkable" polys).
/// </summary>
public static class NavMeshAreas

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a mutable process wide static read from worker threads (IDtQueryFilter.GetCost calls NavMeshAreas.GetAreaCost) with no synchronization, while the settings UI writes them via ApplyTable on every keystroke.

/// default (the built-in Humanoid, id 0) so headless and procedural use needs no settings
/// file. Unity's Agents tab equivalent.
/// </summary>
public static class NavMeshAgentTypes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as NavMeshArea.cs

This is a mutable process wide static read from worker threads (IDtQueryFilter.GetCost calls NavMeshAreas.GetAreaCost) with no synchronization, while the settings UI writes them via ApplyTable on every keystroke.

/// <summary>Snapshot copy, so a bake isn't mutated by later inspector edits. Note this is
/// a MemberwiseClone — valid only while every field is a value type; a future reference
/// field must be cloned explicitly here.</summary>
public NavMeshBuildSettings Clone() => (NavMeshBuildSettings)MemberwiseClone();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MemberwiseClone means every field MUST be a Value type.
A hand written copy is a few lines and cannot silently alias later.

People can already access a MemberwiseClone themselves so this API is redundant, when someone sees a Clone() method, they expect it to be a hand written method that copies it all over.

This same thing applies to NavMeshAgentType.Clone()

string fileRel = folderRel + "/" + Sanitize(sceneName + " NavMesh") + ".navmesh";
string fileAbs = Path.Combine(Project.Current.AssetsPath, fileRel);
data!.Name = Path.GetFileNameWithoutExtension(fileRel);
File.WriteAllText(fileAbs, Serializer.Serialize(typeof(object), data).WriteToString());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.navmesh assets are written as Echo text (Serializer.Serialize(...).WriteToString()).

Every compressed tile layer blob goes into a text file. For a real level that is a large, un diffable, slow to parse asset.
You should use the Binary format instead (also we avoid string format in general, YAML is more Git Friendly which Echo supports)

This was self introduced and not part of main
@Acissathar

Copy link
Copy Markdown
Contributor Author

I appreciate you taking a look! I will spin #2 and #3 off into their own PRs (#1 is dead code this PR introduced, so just gonna delete it in place).

And making a DotRecast fork inside Anthology would simplify a ton because a few of the Workarounds can be fixed in DotRecast directly.

My only concern with this was that then it'd have to be managed more directly rather than just bumping up a version, but I suppose that's really only a problem when it's a problem? It would let us do a little more specific to Prowl, so I'm not opposed to it if that's the desired route.

From reading the Anthology repo, just to be sure I'm understanding it right, this is the process:

  1. Create a separate repo/fork of DotRecast. Make the changes needed in there.
  2. Fork Anthology and make a branch.
  3. In that branch, run the MergeLibrary ps in the anthology repo, pointing to the DotRecast fork.
  4. Create a PR based on the changes of that Anthology branch.

Is that correct? Also is there a cool name for it, or just going with DotRecast?

Everything else looks pretty straight forward with the review though!

@michaelsakharov

Copy link
Copy Markdown
Contributor

That sounds about right, We can just name is DotRecast. I'll actually get DotRecast into Anthology for you, so you just have to modify it, give me a bit

@michaelsakharov

Copy link
Copy Markdown
Contributor

Okay, added DotRecast to Anthology

A Unity-shaped navigation stack built on DotRecast: baked navmesh assets,
runtime queries, crowd-driven agents, off-mesh links, obstacle carving, and
the editor tooling to author and inspect it.

Runtime
- NavMeshData assets and NavMeshBuilder, with region rebuilds that scale with
  the changed area rather than the map size, synchronous or off-thread.
- NavMeshWorld on Scene.Navigation: registered instances, a pooled thread-safe
  query layer, one crowd per agent type, and a demand-driven tile-cache pump
  that costs nothing per frame for surfaces with no queued work.
- Components: NavMeshSurface, NavMeshAgent, NavMeshObstacle (carve or
  velocity-block), NavMeshLink, NavMeshModifier and NavMeshModifierVolume.
- Project-level agent types and 32 Unity-style areas with per-agent costs.
- Our own area-aware rasterizer with pooled heightfield spans, so repeated
  tile bakes on destructible geometry do not churn the heap.

Every surface is tile-cache backed and can carve; there is no representation
for a user to choose. Links ride on the asset and are re-injected as tiles are
contoured, so carving and links coexist. Escape hatches to the underlying
DotRecast objects are exposed, in the spirit of PhysicsWorld exposing Jitter.

Editor
- Surface inspector with bake-to-asset, a .navmesh importer, Navigation project
  settings, area and agent-type drawers, and a scene-view overlay that follows
  runtime carving.

Covered by 127 runtime navigation tests.
@Acissathar
Acissathar marked this pull request as draft August 7, 2026 16:22
…rowl.Recast

Swap the five DotRecast.* packages for the Prowl.Recast fork and delete the
Prowl-side workarounds it makes unnecessary.

Off-mesh link rationing: Detour sizes a tile's link pool from the connections
stored in the tile and budgets nothing for those arriving from neighbours, then
indexed the failed allocation unchecked. Prowl capped crossings at four per tile
and dropped the rest with a warning. Prowl.Recast restores upstream C++ Detour's
DT_NULL_LINK guard at all five unguarded AllocLink sites, so the cap, the
per-tile budget, the two-pass rationing and the warning are gone. Links crowding
a boundary now all cross.

Rasterizer: NavMeshRasterizer.cs existed only because the C# port's AddSpan
ignores the RcSpanPool free list it ships and news up every span. Prowl.Recast
wires the pool up and adds a slope-cosine RasterizeTriangles overload plus
AdoptSpanPools, so the vendored copy is deleted (-363 lines). Steady-state tile
rebuilds now allocate nothing.

Bake scratch lifetime: a [ThreadStatic] scratch set on a Parallel.For worker
outlived the bake by the life of that thread, pinning span pages per worker.
Bakes now pass a loop-local scratch; runtime rebuilds keep sharing the main
thread's.

Also: reset the navmesh asset format version to 1, and drop comments describing
development history rather than current behaviour.
Note - doesn't yet fix editor carving update
NavMeshSurface and NavMeshLink are [ExecuteAlways]

NavMeshLink found its surfaces by walking every GameObject in the scene, then collected links by walking it again, on every edit.
Both read registries on the world now.

The scene view draws off-mesh connections as part of the walkable overlay: line, endpoints, the width the connection covers, an arrowhead when it is one-way.

Reporting a connection requires it to be attached at both ends. Detour keeps a stub for either end failing, so ContainsLinkId used to answer "a stub exists" rather than "an agent can cross": a link that failed was taken for success and never retried by the catch-up.
Note this deliberately does not repoint the NavMeshData reference at the copy, which would be smaller: links run outside play mode, so moving one in the editor would swap the reference to a runtime object and detach the surface from its asset at the next scene save.

Changed clone to be field by field

.navmesh asset now binary for Echo

Poly and Straight path are now settable fields rather than private constants
Also rename Capsule to Cylinder (the shape actually used) and update the wireframe to not be a sphere

Reorganize new attribute handler to be with the navmesh ones
This is something specific to my project where we generate and bake the navmesh at runtime (no editor generation). Copying it would mean the next registration rebuilt from the original bake since it had no owner.
@Acissathar

Copy link
Copy Markdown
Contributor Author

Believe I hit everything - note requires the changes on the Anthology PR (ProwlEngine/Anthology#8) and I didn't bump the Build Props version as that seemed to be something you did manually as a separate commit, but I can include it.

@Acissathar
Acissathar marked this pull request as ready for review August 8, 2026 22:12
@michaelsakharov

Copy link
Copy Markdown
Contributor

Alright i pushed an update to recast.

Let me know when this is ready to review again!

@Acissathar

Copy link
Copy Markdown
Contributor Author

Should be good to go! Conflict resolved and everything appears to be working

@michaelsakharov

Copy link
Copy Markdown
Contributor

I'll review this once i finish my work on refactoring the Audio and Prefab systems, I've got my plate full with those at the moment, sorry. Shouldn't take more then a couple more days.

@Acissathar

Copy link
Copy Markdown
Contributor Author

No worries at all, no rush!

@PaperPrototype

PaperPrototype commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

I just thought of something. Has this been tested in-editor with the Terrain system? If not, then might be worth someone switching to this PR locally and attempting to make navmesh on terrain and see if anything breaks.

@Acissathar

Acissathar commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

It's supposed to gather terrain colliders and meshes, but in my test project it looks like it is not. I'll check it out and see what's going on with it.

Edit: I have a local fix for using the actual terrain data rather than the collider (since the collider isn't populated at edit time). However, it exposed an issue with some of the generation where the generated polys aren't as accurate to sloped/curved ground as they are flat. Working on getting that fixed as well.

Terrain now properly contributes to NavMesh generation, and also added vertices and edges to the NavMesh gizmo.

Also wire in Tile-cache parameters from Prowl.Recast changes to help close up seams from being generated
@Acissathar

Copy link
Copy Markdown
Contributor Author

@PaperPrototype Good catch to check Terrain, it revealed a few issues with the navmesh generation on a few different slopes!

NavMeshTerrain

Note this does require the Anthology PR (ProwlEngine/Anthology#10) to be merged in first.

Additionally, I added in the vertex points on the NavMesh gizmo as I saw Unity also had that and it was surprisingly useful for debugging the original issue.

image

Should be good to go now I believe!

@PaperPrototype

Copy link
Copy Markdown
Contributor

Nice! Yeah I've found so many bugs by just trying to use the features in an actual project haha

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants