Add Insight diagnostic capture policy

CHANGELOG.md

@@ -4,10 +4,11 @@
 
 ### Insight
 
-- Added `evolvephp/insight` as a public experimental diagnostic-batch collection foundation. The package consumes Core execution observations through the existing observation sink boundary, collects immutable bounded per-execution batches and keeps redaction, dashboards, watchers, OpenTelemetry, Evolve Observe, runtime wiring and independent package release deferred.
+- Added `evolvephp/insight` as a public experimental diagnostic-batch collection foundation. The package consumes Core execution observations through the existing observation sink boundary, collects immutable bounded per-execution batches and keeps dashboards, watchers, OpenTelemetry, Evolve Observe, runtime wiring and independent package release deferred.
 - Added storage-neutral Insight batch snapshots, projection from finalized diagnostic batches, a minimal diagnostic batch store contract, an in-memory store for tests and short-lived local development, and a sink adapter that projects and stores accepted batches while keeping runtime wiring deferred.
 - Added a versioned primitive diagnostic batch snapshot codec and an optional SQLite diagnostic batch store for explicit local-development persistence through caller-supplied SQLite `PDO` connections, while keeping application database integration deferred.
 - Added explicit count-bounded retention to the first-party Insight diagnostic batch stores, with deterministic oldest-first pruning for in-memory and SQLite storage.
+- Added a diagnostic capture-policy foundation with primitive-only entries, explicit data classifications, deterministic redaction, exact category/name filtering and deterministic execution-level sampling, while keeping rich watchers, runtime wiring and storage integration deferred.
 
 ### Repository
 

packages/insight/README.md

@@ -10,6 +10,8 @@ Evolve Insight consumes safe Core execution observations and collects them into
 
 Insight also provides a storage-neutral projection boundary for finalized diagnostic batches. `DiagnosticBatchProjector` detaches a `DiagnosticBatch` into a primitive-only `DiagnosticBatchSnapshot` made of string-backed execution identity, execution kind, ordered `DiagnosticObservationSnapshot` values and the dropped observation count. Snapshots do not retain Core `Observation`, execution identifier, request, response, container, throwable or execution-scope objects.
 
+Insight includes a detached diagnostic capture-policy foundation for future rich diagnostic sources. `DiagnosticEntry` and `DiagnosticAttribute` represent bounded primitive-only diagnostic data: execution identifier value, diagnostic category, diagnostic name and ordered attributes whose values are limited to `string`, `int`, `float`, `bool` or `null`. Attribute names, entry identifiers, categories, names and string values are bounded, and non-finite floats, arrays, objects, resources and callables are not accepted.
+
 Current bounded behavior:
 
 - collection starts only after Core reports an execution start
@@ -36,6 +38,19 @@ The SQLite store creates its diagnostic table only when explicitly constructed w
 
 `pdo_sqlite` is a runtime requirement only for applications that explicitly use `SqliteDiagnosticBatchStore`; it is not required for installing or using the non-SQLite Insight functionality.
 
+Current capture-policy behavior:
+
+- `DiagnosticDataClassification` provides explicit machine-readable classifications for public operational metadata, internal operational metadata, personal data, authentication data, secret data, business-sensitive payloads and regulated data
+- `DiagnosticCapturePolicy` accepts public and internal operational metadata by default; personal, business-sensitive and regulated data are excluded unless explicitly enabled by application code
+- secret and authentication data are never deliberately accepted raw
+- `DefaultDiagnosticRedactor` deterministically suppresses secret and authentication attributes, and replaces common sensitive operational machine names such as authorization, password, cookies, tokens, API keys, secrets and session identifiers with `[REDACTED]`
+- `DiagnosticCaptureFilter` supports exact category and diagnostic-name disabling for volume control only
+- `DeterministicDiagnosticSampler` supports integer percentage sampling from 0 to 100 using a stable hash of the execution identifier
+- sampling controls diagnostic volume only; it is not authorization, authentication, security enforcement, redaction, legal retention or an error-retention guarantee
+- accepted attributes keep their original order, use first-accepted-wins duplicate-name handling and are capped by a deterministic retained-attribute limit
+
+The capture-policy foundation does not add rich HTTP, database, cache, log, event, queue or other watchers. Capture entries are not integrated into diagnostic batches, storage snapshots, SQLite payloads or automatic runtime wiring.
+
 ## Requirements
 
 PHP `^8.4`
@@ -52,7 +67,7 @@ https://github.com/josiahking/evolvephp
 
 ## Current Limitations
 
-This package does not provide time-based retention, redaction, rich diagnostic capture, filtering, sampling, dashboards, watchers, OpenTelemetry, Evolve Observe, trace propagation, runtime composition, automatic registration, application database integration or production-ready diagnostics.
+This package does not provide time-based retention, rich HTTP/database/cache/log/event/queue diagnostic watchers, dashboards, OpenTelemetry, Evolve Observe, trace propagation, runtime composition, automatic registration, application database integration, production telemetry export or production-ready diagnostics.
 
 ## Licence
 

packages/insight/src/Capture/DefaultDiagnosticRedactor.php

@@ -0,0 +1,69 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+final class DefaultDiagnosticRedactor implements DiagnosticRedactor
+{
+    public const string REDACTION_MARKER = '[REDACTED]';
+
+    public function redact(DiagnosticAttribute $attribute): ?DiagnosticAttribute
+    {
+        if (
+            $attribute->classification() === DiagnosticDataClassification::SecretData
+            || $attribute->classification() === DiagnosticDataClassification::AuthenticationData
+        ) {
+            return null;
+        }
+
+        if ($this->isOperational($attribute) && $this->isSensitiveMachineName($attribute->name())) {
+            return new DiagnosticAttribute(
+                $attribute->name(),
+                $attribute->classification(),
+                self::REDACTION_MARKER,
+            );
+        }
+
+        return $attribute;
+    }
+
+    private function isOperational(DiagnosticAttribute $attribute): bool
+    {
+        return $attribute->classification() === DiagnosticDataClassification::PublicOperationalMetadata
+            || $attribute->classification() === DiagnosticDataClassification::InternalOperationalMetadata;
+    }
+
+    private function isSensitiveMachineName(string $name): bool
+    {
+        $segments = array_values(array_filter(
+            preg_split('/[._\\-\\s\\/]+/', strtolower($name)) ?: array(),
+            static fn (string $segment): bool => $segment !== '',
+        ));
+
+        foreach ($segments as $segment) {
+            if (in_array($segment, array('authorization', 'authentication', 'password', 'passwd', 'cookie', 'secret', 'session'), true)) {
+                return true;
+            }
+        }
+
+        $pairs = array(
+            array('set', 'cookie'),
+            array('access', 'token'),
+            array('refresh', 'token'),
+            array('api', 'key'),
+            array('session', 'id'),
+            array('session', 'identifier'),
+        );
+
+        foreach ($pairs as $pair) {
+            for ($index = 0, $count = count($segments) - 1; $index < $count; $index++) {
+                if ($segments[$index] === $pair[0] && $segments[$index + 1] === $pair[1]) {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+}

packages/insight/src/Capture/DeterministicDiagnosticSampler.php

@@ -0,0 +1,34 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+final class DeterministicDiagnosticSampler
+{
+    public function __construct(private int $percentage)
+    {
+        if ($this->percentage < 0 || $this->percentage > 100) {
+            throw new \InvalidArgumentException('Diagnostic sampler percentage must be between 0 and 100.');
+        }
+    }
+
+    public function accepts(string $executionIdentifier): bool
+    {
+        DiagnosticAttribute::assertBoundedNonEmptyString(
+            $executionIdentifier,
+            'Diagnostic execution identifier',
+            DiagnosticEntry::MAX_EXECUTION_IDENTIFIER_LENGTH,
+        );
+
+        if ($this->percentage === 0) {
+            return false;
+        }
+
+        if ($this->percentage === 100) {
+            return true;
+        }
+
+        return ((int) sprintf('%u', crc32($executionIdentifier)) % 100) < $this->percentage;
+    }
+}

packages/insight/src/Capture/DiagnosticAttribute.php

@@ -0,0 +1,53 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+final class DiagnosticAttribute
+{
+    public const int MAX_NAME_LENGTH = 128;
+    public const int MAX_STRING_VALUE_LENGTH = 2048;
+
+    public function __construct(
+        private string $name,
+        private DiagnosticDataClassification $classification,
+        private string|int|float|bool|null $value,
+    ) {
+        self::assertBoundedNonEmptyString($this->name, 'Diagnostic attribute name', self::MAX_NAME_LENGTH);
+
+        if (is_float($this->value) && !is_finite($this->value)) {
+            throw new \InvalidArgumentException('Diagnostic attribute float value must be finite.');
+        }
+
+        if (is_string($this->value) && strlen($this->value) > self::MAX_STRING_VALUE_LENGTH) {
+            throw new \InvalidArgumentException('Diagnostic attribute string value is too long.');
+        }
+    }
+
+    public function name(): string
+    {
+        return $this->name;
+    }
+
+    public function classification(): DiagnosticDataClassification
+    {
+        return $this->classification;
+    }
+
+    public function value(): string|int|float|bool|null
+    {
+        return $this->value;
+    }
+
+    public static function assertBoundedNonEmptyString(string $value, string $field, int $maximumLength): void
+    {
+        if ($value === '') {
+            throw new \InvalidArgumentException($field . ' must not be empty.');
+        }
+
+        if (strlen($value) > $maximumLength) {
+            throw new \InvalidArgumentException($field . ' is too long.');
+        }
+    }
+}

packages/insight/src/Capture/DiagnosticCaptureFilter.php

@@ -0,0 +1,41 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+final class DiagnosticCaptureFilter
+{
+    /**
+     * @var array<string, true>
+     */
+    private array $disabledCategories = array();
+
+    /**
+     * @var array<string, true>
+     */
+    private array $disabledNames = array();
+
+    /**
+     * @param list<string> $disabledCategories
+     * @param list<string> $disabledNames
+     */
+    public function __construct(array $disabledCategories = array(), array $disabledNames = array())
+    {
+        foreach ($disabledCategories as $category) {
+            DiagnosticAttribute::assertBoundedNonEmptyString($category, 'Disabled diagnostic category', DiagnosticEntry::MAX_CATEGORY_LENGTH);
+            $this->disabledCategories[$category] = true;
+        }
+
+        foreach ($disabledNames as $name) {
+            DiagnosticAttribute::assertBoundedNonEmptyString($name, 'Disabled diagnostic name', DiagnosticEntry::MAX_NAME_LENGTH);
+            $this->disabledNames[$name] = true;
+        }
+    }
+
+    public function allows(DiagnosticEntry $entry): bool
+    {
+        return !isset($this->disabledCategories[$entry->category()])
+            && !isset($this->disabledNames[$entry->name()]);
+    }
+}

packages/insight/src/Capture/DiagnosticCapturePolicy.php

@@ -0,0 +1,116 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+final class DiagnosticCapturePolicy
+{
+    public const int DEFAULT_MAXIMUM_ACCEPTED_ATTRIBUTE_COUNT = 16;
+
+    private DiagnosticRedactor $redactor;
+
+    private DiagnosticCaptureFilter $filter;
+
+    private DeterministicDiagnosticSampler $sampler;
+
+    /**
+     * @var array<string, true>
+     */
+    private array $acceptedClassifications = array();
+
+    /**
+     * @param list<DiagnosticDataClassification>|null $acceptedClassifications
+     */
+    public function __construct(
+        ?DiagnosticRedactor $redactor = null,
+        ?DiagnosticCaptureFilter $filter = null,
+        ?DeterministicDiagnosticSampler $sampler = null,
+        ?array $acceptedClassifications = null,
+        private int $maximumAcceptedAttributeCount = self::DEFAULT_MAXIMUM_ACCEPTED_ATTRIBUTE_COUNT,
+    ) {
+        if ($this->maximumAcceptedAttributeCount <= 0 || $this->maximumAcceptedAttributeCount > DiagnosticEntry::MAX_ATTRIBUTE_COUNT) {
+            throw new \InvalidArgumentException('Maximum accepted diagnostic attribute count must be positive and bounded.');
+        }
+
+        $this->redactor = $redactor ?? new DefaultDiagnosticRedactor();
+        $this->filter = $filter ?? new DiagnosticCaptureFilter();
+        $this->sampler = $sampler ?? new DeterministicDiagnosticSampler(100);
+
+        foreach ($acceptedClassifications ?? $this->defaultAcceptedClassifications() as $classification) {
+            if (
+                $classification === DiagnosticDataClassification::SecretData
+                || $classification === DiagnosticDataClassification::AuthenticationData
+            ) {
+                continue;
+            }
+
+            $this->acceptedClassifications[$classification->value] = true;
+        }
+    }
+
+    public function apply(DiagnosticEntry $candidate): ?DiagnosticEntry
+    {
+        $attributes = array();
+        $acceptedNames = array();
+
+        foreach ($candidate->attributes() as $attribute) {
+            if (!isset($this->acceptedClassifications[$attribute->classification()->value])) {
+                continue;
+            }
+
+            try {
+                $redacted = $this->redactor->redact($attribute);
+            } catch (\Throwable) {
+                continue;
+            }
+
+            if ($redacted === null || !isset($this->acceptedClassifications[$redacted->classification()->value])) {
+                continue;
+            }
+
+            if (isset($acceptedNames[$redacted->name()])) {
+                continue;
+            }
+
+            $attributes[] = $redacted;
+            $acceptedNames[$redacted->name()] = true;
+
+            if (count($attributes) >= $this->maximumAcceptedAttributeCount) {
+                break;
+            }
+        }
+
+        if ($attributes === array()) {
+            return null;
+        }
+
+        $accepted = new DiagnosticEntry(
+            $candidate->executionIdentifier(),
+            $candidate->category(),
+            $candidate->name(),
+            $attributes,
+        );
+
+        if (!$this->filter->allows($accepted)) {
+            return null;
+        }
+
+        if (!$this->sampler->accepts($accepted->executionIdentifier())) {
+            return null;
+        }
+
+        return $accepted;
+    }
+
+    /**
+     * @return list<DiagnosticDataClassification>
+     */
+    private function defaultAcceptedClassifications(): array
+    {
+        return array(
+            DiagnosticDataClassification::PublicOperationalMetadata,
+            DiagnosticDataClassification::InternalOperationalMetadata,
+        );
+    }
+}

packages/insight/src/Capture/DiagnosticDataClassification.php

@@ -0,0 +1,16 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Evolve\Insight\Capture;
+
+enum DiagnosticDataClassification: string
+{
+    case PublicOperationalMetadata = 'public-operational-metadata';
+    case InternalOperationalMetadata = 'internal-operational-metadata';
+    case PersonalData = 'personal-data';
+    case AuthenticationData = 'authentication-data';
+    case SecretData = 'secret-data';
+    case BusinessSensitivePayload = 'business-sensitive-payload';
+    case RegulatedData = 'regulated-data';
+}