Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 87 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

AgentExperience.NET captures what an AI agent actually tried, verifies whether it worked, and turns the result into an auditable lesson that future runs can reuse safely. It sits between [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) (MAF) execution and durable storage, without replacing either.

> **Status: early development.** Epic 1 (capture and explain agent experience) is implemented and tested. Epic 2 has started: Experience Records can be stored in PostgreSQL and moved through their lifecycle with atomic, audited commits. Retrieval, injection, and governance are planned (see [Roadmap](#roadmap)). Nothing is published to NuGet yet, and APIs may change.
> **Status: early development.** Epic 1 (capture and explain agent experience) is implemented and tested. Epic 2 has started: a completed run can now be finalized into a durable Experience Record in PostgreSQL in one call, and moved through its lifecycle with atomic, audited commits. Retrieval, injection, and governance are planned (see [Roadmap](#roadmap)). Nothing is published to NuGet yet, and APIs may change.

## Why

Expand All @@ -33,6 +33,8 @@ AgentExperience.NET records observable evidence (tool calls, results, errors, ve
| PostgreSQL Experience Record store: create, get, and scoped query; host authorization checked before database access; exact scope matching in SQL | `AgentExperience.Storage.Postgres` |
| Atomic audited lifecycle commits: the event and the record's projection in one transaction, idempotent by event ID, revision-checked, with append-only history | `AgentExperience.Core`, `AgentExperience.Storage.Postgres` |
| Journaled schema migrations: embedded scripts applied once, one transaction per script, serialized across processes by an advisory lock | `AgentExperience.Storage.Postgres` |
| One finalization call: evaluate, gate on authorization and the host's storage decision, reflect, create the record as a `Candidate`, commit the initial event that promotes it — replay-safe and structured at every stage | `AgentExperience.Core` |
| Dependency-injection registration for each package, so a host wires capture, finalization, and storage without knowing concrete types | `AgentExperience.Core`, `AgentExperience.Storage.Postgres` |

## Quick look

Expand All @@ -54,9 +56,88 @@ await agent.RunAsync("Triage ticket #4812", session);

See the [adapter README](src/AgentExperience.MicrosoftAgentFramework/README.md) for options, supported agent types, and caveats. See the [PostgreSQL store README](src/AgentExperience.Storage.Postgres/README.md) for the trust boundary, the `ExperienceSchemaMigrator.MigrateAsync` startup call, and data semantics.

## Turning a run into a durable record

A captured run becomes a durable, reusable Experience Record through one Core call.
`ExperienceFinalizationService.FinalizeAsync` runs six stages in order — load the captured snapshot, evaluate it,
check authorization and the host's storage decision, reflect on it, create the record, commit its initial lifecycle
event — and stops at the first stage that ends the call, always returning a structured result rather than throwing.
The two gates precede reflection on purpose: the reflector is the seam a host would plug a model into, so a run that
is about to be refused is never handed to it.

```csharp
using AgentExperience.Core.Finalization;
using AgentExperience.Core.Verification;

var result = await finalization.FinalizeAsync(
new FinalizeExperienceRequest(
RunId: runId,
Authorization: authorization, // host-established; the run's scope must lie inside it
ClosedRound: new ClosedVerificationRound(roundId, "rev-7"),
RequiredChecks: [new RequiredCheck("unit-tests-pass", ExpectedKind: "TestResult")],
Evidence: evidence, // finalization filters and aggregates it itself
CurrentArtifactRevision: "rev-7",
StorageDecision: StorageDecision.Permit, // or StorageDecision.Deny("retention policy")
FinalizedAt: DateTimeOffset.UtcNow),
cancellationToken);

if (result.IsDurable)
{
logger.LogInformation("Experience {Id} is {Status} at revision {Revision}",
result.ExperienceId, result.Status, result.Revision);
}
else
{
logger.LogWarning("Finalization ended at {Stage}: {Outcome} — {Reason}",
result.Stage, result.Outcome, result.Failure?.Reason);
}
```

| Outcome | When | What was written |
| --- | --- | --- |
| `Validated` | Verified, reflection succeeded, storage permitted | The record (reuse confidence 2/3, one supporting validation, no contradictions), created as `Candidate`, plus the initial event that moved it to `Validated` |
| `Quarantined` | Storage permitted, but verification did not pass or the reflector threw | The record, with **no** reflection, created as `Candidate`, plus the initial event that moved it to `Quarantined`. `Failure` names the stage that decided it |
| `AlreadyFinalized` | This run's record already exists *and* is already confirmed | Nothing. The result reports the stored record, status, and revision. (A record left unconfirmed by an earlier call is resumed instead: the retry commits its initial event and returns `Validated`/`Quarantined`.) |
| `StorageDenied` | The host's `StorageDecision` denied | Nothing at all, and no record ID is issued |
| `NotAuthorized` | The run's scope lies outside the authorization | Nothing; denied before any store call |
| `RunNotFound` / `RunNotFinished` | No such captured run, or it has no execution status | Nothing |
| `Failed` | A stage failed (for example the database was unavailable) | Never reported as durable. Any record already created stays a `Candidate`, which is never reusable, and the captured run stays available for a retry |

Three properties make retrying safe. The record is *created* as a `Candidate` and its initial lifecycle event
performs the real transition, so a commit that never lands leaves nothing reusable behind. The record ID, the
reflection ID, and the initial event ID are all derived from the run ID, so a second call cannot create a second
record or a second initial confirmation. And the initial event's fields are a pure function of the stored record, so
a retry re-derives exactly the event the store already deduplicates on.

Finalization never sanitizes — capture already rejected anything unsafe — and never decides storage or risk policy on
the host's behalf: `StorageDecision` travels in the request and Core simply obeys it.

### Wiring it

Each package registers its own services, so a host never names a concrete type:

```csharp
using AgentExperience.Core.DependencyInjection;
using AgentExperience.Storage.Postgres.DependencyInjection;

services.AddSingleton(NpgsqlDataSource.Create(connectionString));
services.AddAgentExperiencePostgresStore(); // IExperienceRecordStore
services.AddAgentExperienceCore(sanitizationOptions, captureLimits);
// -> ISanitizer, IExperienceCaptureService, IExperienceReflector,
// ExperienceLifecycleService, ExperienceFinalizationService
```

`AgentExperience.Abstractions` stays BCL-only; only `Core` and the storage adapter take
`Microsoft.Extensions.DependencyInjection.Abstractions`, and every registration uses `TryAdd`, so a host's own
implementation wins. Call `ExperienceSchemaMigrator.MigrateAsync` once at startup before the store is used.

The MAF adapter can drive finalization for you: set `FinalizationService` and `ResolveFinalization` on
`ExperienceCaptureOptions` and every successfully captured invocation is finalized right after it is completed. See
the [adapter README](src/AgentExperience.MicrosoftAgentFramework/README.md#finalizing-captured-runs).

## Design principles

- **Hexagonal core.** `Abstractions` and `Core` depend only on the BCL and a redaction primitive. MAF, databases, models, and telemetry stay in adapters. Dependency-boundary tests enforce this in CI.
- **Hexagonal core.** `Abstractions` depends only on the BCL; `Core` adds a redaction primitive and the dependency-injection *abstractions* it needs to register its own services. MAF, databases, models, and telemetry stay in adapters. Dependency-boundary tests enforce this in CI.
- **Failure-preserving capture.** Failed and cancelled runs are recorded through an outer lifecycle path, never only a success callback.
- **Evidence before trust.** Verification is deterministic and bound to a host-closed round and artifact revision. A completion score is never mistaken for reuse confidence.
- **Sanitize before anything is stored.** Unknown payload fields are dropped by default, and secrets are redacted from nested values.
Expand All @@ -67,7 +148,7 @@ See the [adapter README](src/AgentExperience.MicrosoftAgentFramework/README.md)
```
src/
AgentExperience.Abstractions/ domain contracts and ports (BCL only)
AgentExperience.Core/ sanitization, capture, verification, reflection, lifecycle transitions
AgentExperience.Core/ sanitization, capture, verification, reflection, lifecycle transitions, finalization
AgentExperience.MicrosoftAgentFramework/ MAF adapter (pinned Microsoft.Agents.AI 1.20.0)
AgentExperience.Storage.Postgres/ PostgreSQL Experience Record store and schema migrator (pinned Npgsql 10.0.3, dbup-postgresql 7.0.1, dbup-core 6.1.1)
tests/
Expand All @@ -90,16 +171,16 @@ dotnet build
dotnet test
```

Unit and MAF adapter tests run in memory, with no network, database, or model credentials. `AgentExperience.CompatibilityProof` and the `PostgresExperienceRecordStoreTests`, `PostgresLifecycleCommitTests`, and `ExperienceSchemaMigratorTests` in `AgentExperience.Storage.Postgres.Tests` start a PostgreSQL/pgvector container through Testcontainers, so they need Docker. If Testcontainers' Ryuk container fails to start under your local Docker setup, set `TESTCONTAINERS_RYUK_DISABLED=true`. To skip the container-backed tests:
Unit and MAF adapter tests run in memory, with no network, database, or model credentials. `AgentExperience.CompatibilityProof` and the `PostgresExperienceRecordStoreTests`, `PostgresLifecycleCommitTests`, `PostgresFinalizationTests`, and `ExperienceSchemaMigratorTests` in `AgentExperience.Storage.Postgres.Tests` start a PostgreSQL/pgvector container through Testcontainers, so they need Docker. If Testcontainers' Ryuk container fails to start under your local Docker setup, set `TESTCONTAINERS_RYUK_DISABLED=true`. To skip the container-backed tests:

```bash
dotnet test --filter "FullyQualifiedName!~CompatibilityProof&FullyQualifiedName!~PostgresExperienceRecordStoreTests&FullyQualifiedName!~PostgresLifecycleCommitTests&FullyQualifiedName!~ExperienceSchemaMigratorTests"
dotnet test --filter "FullyQualifiedName!~CompatibilityProof&FullyQualifiedName!~PostgresExperienceRecordStoreTests&FullyQualifiedName!~PostgresLifecycleCommitTests&FullyQualifiedName!~PostgresFinalizationTests&FullyQualifiedName!~ExperienceSchemaMigratorTests"
```

## Roadmap

1. **Capture and explain agent experience** ✅ contracts, sanitization, capture, verification, reflection, MAF adapter
2. **Reuse relevant experience:** PostgreSQL persistence and atomic audited lifecycle commits (in place), hybrid text and vector retrieval, historical-reference injection into MAF
2. **Reuse relevant experience:** PostgreSQL persistence, atomic audited lifecycle commits, and one-call finalization of captured runs (in place), hybrid text and vector retrieval, historical-reference injection into MAF
3. **Govern experience safely:** sharing grants, the remaining lifecycle transitions, evidence-based confidence updates
4. **Operate and measure the learning loop:** OpenTelemetry instrumentation, an end-to-end demo, measured reuse against a baseline, data deletion and expiry

Expand Down
5 changes: 4 additions & 1 deletion src/AgentExperience.Core/AgentExperience.Core.csproj
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<Description>AgentExperience.NET's first production Core package: the default sanitization pipeline (per-Kind allowlists, secret-field classification, recursive traversal, fail-closed rejection) built on AgentExperience.Abstractions' ISanitizer port. No dependency on MAF, EF Core, PostgreSQL, model providers, or OpenTelemetry -- only the BCL, Abstractions, and Microsoft.Extensions.Compliance.Redaction (AD-1).</Description>
<Description>AgentExperience.NET's first production Core package: the default sanitization pipeline (per-Kind allowlists, secret-field classification, recursive traversal, fail-closed rejection) built on AgentExperience.Abstractions' ISanitizer port. No dependency on MAF, EF Core, PostgreSQL, model providers, or OpenTelemetry -- only the BCL, Abstractions, Microsoft.Extensions.Compliance.Redaction, and Microsoft.Extensions.DependencyInjection.Abstractions (AD-1).</Description>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Compliance.Redaction" Version="10.9.0" />
<!-- DI abstractions only (no container, no hosting): lets Core ship its own AddAgentExperienceCore
registration extension. Abstractions stays BCL-only. -->
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="[10.0.11]" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
using AgentExperience.Abstractions;
using AgentExperience.Core.Capture;
using AgentExperience.Core.Finalization;
using AgentExperience.Core.Lifecycle;
using AgentExperience.Core.Reflections;
using AgentExperience.Core.Sanitization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;

namespace AgentExperience.Core.DependencyInjection;

/// <summary>
/// Registers AgentExperience.NET's Core services in a <see cref="IServiceCollection"/>. Core owns
/// its own registration so a host never has to know which concrete types implement which port; the
/// storage adapter registers its own in the same way (see
/// <c>AddAgentExperiencePostgresStore</c>), and <c>AgentExperience.Abstractions</c> stays BCL-only.
/// </summary>
public static class AgentExperienceCoreServiceCollectionExtensions
{
/// <summary>
/// Registers the sanitizer, the in-memory capture service, the default reflector, the lifecycle
/// service, and the finalization service as singletons.
/// </summary>
/// <remarks>
/// <para>
/// Every registration uses <c>TryAdd</c>, so a host that has already registered its own
/// <see cref="ISanitizer"/>, <see cref="IExperienceCaptureService"/>, or
/// <see cref="IExperienceReflector"/> keeps it.
/// </para>
/// <para>
/// <see cref="ExperienceLifecycleService"/> and <see cref="ExperienceFinalizationService"/> both
/// need an <see cref="IExperienceRecordStore"/>, which Core does not implement: register a
/// storage adapter (for example <c>AddAgentExperiencePostgresStore</c>) as well, or resolving
/// them fails.
/// </para>
/// <para>
/// No sanitization policy or capture limit is invented here: both are host decisions with real
/// security and memory consequences, so both are required arguments.
/// </para>
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="sanitizationOptions">The per-<c>Kind</c> sanitization policy the default sanitizer applies.</param>
/// <param name="captureLimits">The limits in-memory capture enforces.</param>
/// <returns><paramref name="services"/>, for chaining.</returns>
/// <exception cref="ArgumentNullException">Any argument is <see langword="null"/>.</exception>
public static IServiceCollection AddAgentExperienceCore(
this IServiceCollection services,
SanitizationOptions sanitizationOptions,
CaptureLimits captureLimits)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(sanitizationOptions);
ArgumentNullException.ThrowIfNull(captureLimits);

// The arguments are captured by the factories rather than resolved back out of the container.
// Re-resolving them would let a SanitizationOptions or CaptureLimits the host registered earlier
// silently replace the caller's, so the sanitizer would run a policy nobody passed to it.
services.TryAddSingleton(sanitizationOptions);
services.TryAddSingleton(captureLimits);
services.TryAddSingleton<ISanitizer>(_ => new DefaultSanitizer(sanitizationOptions));
services.TryAddSingleton<IExperienceCaptureService>(provider => new InMemoryExperienceCaptureService(
provider.GetRequiredService<ISanitizer>(),
captureLimits));
services.TryAddSingleton<IExperienceReflector, DefaultExperienceReflector>();
services.TryAddSingleton<ExperienceLifecycleService>();
services.TryAddSingleton<ExperienceFinalizationService>();

return services;
}
}
Loading
Loading