diff --git a/.gitignore b/.gitignore index d1f62c2742..06c59331ca 100644 --- a/.gitignore +++ b/.gitignore @@ -86,3 +86,4 @@ docs/_cache/ .devcontainer/devcontainer-lock.json *.lscache +.gstack/ diff --git a/openspec/changes/add-usage-budget-controls/design.md b/openspec/changes/add-usage-budget-controls/design.md new file mode 100644 index 0000000000..c073d24c1c --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/design.md @@ -0,0 +1,893 @@ +# Design: Add Usage Budget Controls + +## Overview + +Add three complementary usage budget controls: + +1. **Organization budget alert emails** — warnings sent when organization accepted event usage crosses configured percentage thresholds. +2. **Automatic smart project throttling** — automatic project-isolated throttling/sampling when a project spikes enough to threaten organization usage. +3. **Optional project event budgets** — user-configured project-level accepted event caps. + +These are event-usage controls, not generic API request throttling controls. + +The implementation should extend existing usage and notification paths: + +- `UsageService` remains responsible for usage totals, budget threshold detection, smart throughput calculations, and limit calculations. +- Existing organization overage notification behavior remains unchanged. +- `OverageMiddleware` remains the event-post gate for coarse request-level enforcement. +- `EventPostsJob` / event post processing is where project-level sampled acceptance should happen after events are parsed. +- Existing organization usage enforcement remains the outer hard limit. +- Existing `ThrottlingMiddleware` remains unchanged for this feature. +- Budget alert and smart throttling emails are asynchronous side effects and must not block event ingestion. + +## Existing code paths + +### Event post enforcement + +`OverageMiddleware` currently: + +1. Skips non-event posts. +2. Gets the default organization id from the authenticated request. +3. Rejects when event submission is globally disabled. +4. Rejects invalid/oversized posts. +5. Calls `UsageService.GetEventsLeftAsync(organizationId)`. +6. Rejects organization overage. +7. Allows the request to continue. + +This change keeps that coarse gate but does not rely on it as the only project-level enforcement layer. + +### Usage tracking + +`UsageService.IncrementTotalAsync(organizationId, projectId, eventCount)` already increments both: + +- organization bucket total +- project bucket total + +Existing cache key helpers accept an optional `projectId`. + +Use these existing counters for organization budget alerts, smart project throttling, and project event budgets. Do not introduce a parallel accepted-event counter. + +### Event post processing + +`EventPostsJob` already: + +- loads project and organization +- parses the post into events +- calculates how many events can be processed +- processes only allowed events +- increments blocked usage for events over plan limit +- increments accepted total usage for processed events +- increments discarded usage for discarded events + +Extend this path for sampled project-level acceptance. + +### Existing organization notice email path + +Existing flow for monthly/hourly overage notices: + +1. `UsageService` publishes `PlanOverage`. +2. `EnqueueOrganizationNotificationOnPlanOverage` subscribes to `PlanOverage`. +3. It enqueues `OrganizationNotificationWorkItem`. +4. `OrganizationNotificationWorkItemHandler` loads organization and users. +5. The handler sends organization emails only to users with verified email addresses and email notifications enabled. +6. `Mailer.SendOrganizationNoticeAsync` renders the organization notice email. + +Budget alerts and project smart throttling notifications should follow this pattern with new message/work item/mailer/template types. + +--- + +## Automatic Smart Project Throttling + +### Overview + +Automatic smart project throttling is the default guardrail for sudden event spikes. + +It addresses historical feedback that: + +- organization-level throttling can punish all projects when one project is noisy +- throttling should notify users when applied +- throttling should still allow a small percentage of events through for troubleshooting +- users should not need to configure many options before this protection works + +This is separate from optional project event budgets: + +- Smart throttling is automatic and adaptive. +- Project event budgets are explicit user-configured caps. + +### Design principle: minimal configuration + +Smart project throttling should not require users to configure project percentages, per-project caps, or per-stack settings. + +The UI may expose status and explanation, but not a large set of tuning options. + +### Throughput calculation + +Smart throttling should use remaining monthly allowance and remaining time in the monthly usage period. + +Preferred formula shape: + +```text +maxThroughput(window) = eventsLeftInMonth / windowsLeftInMonth * burstMultiplier +``` + +Where: + +* `eventsLeftInMonth` is the organization's current effective monthly allowance minus accepted usage. +* `windowsLeftInMonth` is the number of five-minute usage buckets remaining in the period. +* `burstMultiplier` is 10 to preserve existing Exceptionless burst tolerance. +* Accepted totals include the current and previous unsaved buckets. + +This is intentionally different from a static `monthlyPlanLimit / timeInMonth` calculation. Customers should not be throttled harshly late in the month when they still have substantial allowance remaining. + +### Project isolation + +When organization-level smart throttling would apply, the system should identify the project or projects contributing to the spike and throttle those projects first. + +Observable behavior: + +* One noisy project should not cause all other projects in the organization to stop ingesting events. +* Other projects should continue under organization-level usage enforcement unless they also exceed smart-throttling criteria. +* If the organization hard monthly limit is exhausted, all projects remain subject to existing organization overage behavior. + +### Sampling while throttled + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless should accept a small sample of events from that project. + +Recommended v1 behavior: + +```text +sampleRate = 5% +``` + +V1 uses stable hash selection across the whole post. Sampling is a long-run probability, so a single-event post is genuinely accepted about 5% of the time rather than being forced through. + +Sampling goals: + +* Preserve visibility into whether the problem is still occurring. +* Preserve visibility into whether a deployed fix reduced occurrences. +* Preserve a representative long-run sample without a per-post minimum that biases single-event traffic. +* Avoid allowing a noisy project to consume the entire organization allowance. + +### Event processing location + +Smart throttling and sampled acceptance should be implemented in the event processing path after event posts are parsed, not solely in `OverageMiddleware`. + +Reason: + +* `OverageMiddleware` can reject the entire request before the event count is known. +* It cannot preserve a 1–5% sample of events from a batch. +* `EventPostsJob` already parses the post, calculates `eventsToProcess`, processes only allowed events, and increments blocked usage for the rest. +* Extending this job/path allows project-level partial acceptance while preserving existing usage accounting. + +### Event selection + +When only part of a batch can be accepted, the implementation should avoid always taking only the first events if doing so would bias troubleshooting data. + +Acceptable v1 approaches: + +* deterministic sampling by event id/hash +* random sampling with stable seed per post +* first-N selection only if no safe sampling utility exists, but this should be documented as a known limitation + +### Smart throttling state + +The system may store project throttling state in cache to avoid recalculating expensive decisions on every event. + +Suggested key shape: + +```text +usage:{fiveMinuteBucket}:{organizationId}:{projectId}:smart-throttle +``` + +Suggested value: + +```json +{ + "is_throttled": true, + "sample_rate": 0.05, + "effective_until_utc": "..." +} +``` + +TTL: short window, aligned with usage bucket/window duration. + +If cached throttle state expires, the system can re-evaluate based on current usage counters. + +`EnableSmartProjectThrottling` defaults to enabled and acts only as an operational kill switch for automatic smart throttling. Project-list status enrichment bulk-reads these short-lived keys once. + +### Smart throttling notification + +When a project enters smart-throttled state, eligible organization users should receive an email notification. + +Notification behavior: + +* Send at most once per project per throttling period or cooldown window. +* Reuse the existing organization notification eligibility rules. +* Do not send repeated emails for every event post while the project remains throttled. +* Include project name, organization name, current project accepted usage, current fair-share limit, and a link to project usage. +* Re-check current plan, usage, kill-switch, and throttle state in the delayed handler before sending. +* Emit the dedicated throttled-event metric for sampled-out events and structured logs only when state transitions to active. + +Suggested message/work item: + +```csharp +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Suggested mailer method: + +```csharp +Task SendProjectThrottledNoticeAsync( + User user, + Organization organization, + Project project, + double sampleRate, + int currentEventCount, + int eventLimit); +``` + +### Relationship to organization overage + +Smart project throttling is not a substitute for organization hard overage enforcement. + +Behavior: + +* If organization monthly allowance remains, smart-throttled projects may still have sampled events accepted. +* If organization monthly allowance is exhausted, existing organization overage behavior applies. +* Project sampling must not allow accepted event totals to exceed the organization hard monthly allowance. + +### Relationship to optional project event budgets + +If a project has an explicit project event budget, that budget participates in the same allowed-event calculation. + +Recommended order: + +```text +organization hard allowance +↓ +explicit project budget, if configured +↓ +automatic smart throttling sample allowance, if active +``` + +The final number of events accepted from a batch should be the minimum allowed by all applicable controls. + +### Relationship to budget alert emails + +Smart throttling email notification is distinct from organization budget alert emails. + +* Budget alerts notify when organization accepted usage crosses configured percentages. +* Smart throttling notifications notify when the system starts sampling/throttling a noisy project. +* A project can be smart-throttled before any budget alert threshold is crossed. +* A budget alert can fire without any project being smart-throttled. + +--- + +## Organization Budget Alert Emails + +### Overview + +Add configurable organization-level event budget alert emails. These alerts warn users before monthly plan overage, based on accepted event usage crossing configured percentage thresholds. + +Budget alerts are informational only. They do not block ingestion and do not replace organization overage enforcement, smart throttling, or project event budgets. + +### Domain model + +Add budget alert settings to Organization. + +```csharp +public class Organization +{ + // existing properties... + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Add a new model: + +```csharp +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} +``` + +### Defaults + +Existing organizations: + +```text +budget_alert_settings = null +``` + +Null means disabled. + +New organizations: + +```text +budget_alert_settings = null +``` + +The UI may suggest default thresholds [50, 80], but alerts must not be enabled until a user explicitly saves settings. + +### Validation + +Validation rules: + +* `BudgetAlertSettings == null` is valid. +* `Enabled == false` with empty thresholds is valid. +* If enabled, thresholds must contain at least one value. +* Each threshold must be greater than 0. +* Each threshold must be less than 100. +* Threshold 100 is not allowed because existing monthly overage notification already handles reaching/exceeding the plan limit. +* Duplicate thresholds must be removed. +* Thresholds must be stored sorted ascending. +* Percentage budget alerts must be rejected or disabled for unlimited organizations. + +### API contract + +Add `BudgetAlertSettings` to organization update and view DTOs. + +Preferred cleanup: + +```csharp +public record UpdateOrganization +{ + public string Name { get; set; } = null!; + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Then update `OrganizationController` generic usage from: + +```csharp +RepositoryApiController +``` + +to: + +```csharp +RepositoryApiController +``` + +If minimizing backend churn is preferred, `BudgetAlertSettings` can be added to `NewOrganization` because the current controller uses `NewOrganization` for both create and update. However, adding `UpdateOrganization` is cleaner because `NewOrganization` currently only contains the required organization name. + +Add to `ViewOrganization`: + +```csharp +public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +``` + +Both `budget_alert_settings: null` and `ingest_limit: null` clear their settings. The canonical Delta OpenAPI transformer must generate the referenced complex component schemas and represent these PATCH properties as `oneOf: [null, $ref]`; snapshots and generated clients are regenerated from that output. + +Serialized JSON uses the existing snake_case policy. + +### Threshold calculation + +Use the same effective organization allowance as existing usage enforcement: + +```csharp +effectiveOrganizationAllowance = organization.GetMaxEventsPerMonthWithBonus(timeProvider) +``` + +If effective allowance is negative/unlimited: + +```text +budget alert thresholds are inactive +``` + +Threshold event count: + +```csharp +thresholdEventCount = ceil(effectiveOrganizationAllowance * thresholdPercent / 100) +``` + +### Triggering alerts + +Budget alert checks should run in the accepted-event usage increment path. + +Recommended location: + +```text +UsageService.IncrementTotalAsync +``` + +Observable behavior must be: + +* Alert fires when usage crosses threshold. +* Alert does not fire before threshold. +* Alert does not fire repeatedly after threshold. +* Multiple crossed thresholds can fire if a large batch jumps over more than one threshold. +* Crossing compares bucket-inclusive previous and current accepted totals. +* Enabling alerts after usage already exceeds a threshold evaluates the newly active thresholds once immediately after save. + +### Deduplication + +Each organization threshold must email at most once per monthly usage period. + +Recommended cache key: + +```text +usage-budget-alert:{yyyyMM}:{organizationId}:{threshold} +``` + +TTL: until end of current monthly usage period + safety buffer. + +The cache key is claimed atomically with `AddAsync` before publishing to prevent duplicate sends under concurrency. + +### Message and work item + +Add a new message: + +```csharp +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a new work item: + +```csharp +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a startup subscriber: + +```text +EnqueueOrganizationBudgetAlertOnUsageThreshold +``` + +### Work item handler + +Add: + +```text +OrganizationBudgetAlertWorkItemHandler +``` + +Behavior: + +* Load organization by id. +* Load users in organization. +* Send only to users with verified email addresses. +* Send only to users with `EmailNotificationsEnabled == true`. +* Use the same eligibility rules as existing organization notices. +* Do not send if organization no longer exists. +* Re-check current organization budget alert settings before sending. +* If budget alerts are disabled or the threshold was removed, skip sending. + +### Mailer + +Add to `IMailer`: + +```csharp +Task SendOrganizationBudgetAlertAsync( + User user, + Organization organization, + int threshold, + int thresholdEventCount, + int currentEventCount, + int eventLimit); +``` + +Add implementation in `Mailer`. + +Add template: + +```text +src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html +``` + +Template must include: + +* organization name +* threshold percentage +* threshold event count +* current accepted event count +* organization event limit +* remaining event count +* link to organization usage +* link to billing/plan management when applicable +* link to notification settings + +Do not include event payload data. + +--- + +## Optional Project Event Budgets + +### Overview + +Add optional project-level event budgets enforced during event processing. This is an event-usage control, not generic API request throttling. + +### Domain model + +Add a nullable event budget configuration to Project. + +```csharp +public class Project +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } +} +``` + +Add a new model in `src/Exceptionless.Core/Models/ProjectIngestLimit.cs`: + +```csharp +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} +``` + +### Naming + +Use `IngestLimit`, not `RateLimit`, because the feature limits accepted event volume for a project, not generic request rate. + +Use `ProjectIngestLimit`, not `ProjectQuota`, because this is a cap, not a reservation. + +### API contract + +Add `IngestLimit` to: + +```text +src/Exceptionless.Web/Models/Project/UpdateProject.cs +src/Exceptionless.Web/Models/Project/ViewProject.cs +``` + +Recommended DTO shape: + +```csharp +public record UpdateProject +{ + public string Name { get; set; } = null!; + public bool DeleteBotDataEnabled { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } +} + +public class ViewProject +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } + public int? EffectiveIngestLimit { get; set; } + public bool IsSmartThrottled { get; set; } + public double? SmartThrottleSampleRate { get; set; } +} +``` + +### Project update behavior + +Use existing `PATCH /api/v2/projects/{id}`. Do not add a new endpoint. + +Clear project budget: + +```json +{ + "ingest_limit": null +} +``` + +Fixed budget: + +```json +{ + "ingest_limit": { + "type": "fixed", + "fixed_limit": 20000 + } +} +``` + +Percentage budget: + +```json +{ + "ingest_limit": { + "type": "percent_of_organization_limit", + "percent_of_organization_limit": 20 + } +} +``` + +### Validation + +General validation: + +* `ingest_limit == null` is valid and means Off. +* Unknown limit type is invalid. +* Limit values must not be negative. +* Ingest limit validation must not change existing project name validation. + +For Fixed: + +* `fixed_limit` is required. +* `fixed_limit` must be greater than 0. +* `percent_of_organization_limit` should be ignored or cleared server-side. +* Backend should not reject a fixed limit greater than the current organization limit. The effective cap is clamped during evaluation. + +For PercentOfOrganizationLimit: + +* `percent_of_organization_limit` is required. +* Percentage must be greater than 0. +* Percentage must be less than or equal to 100. +* `fixed_limit` should be ignored or cleared server-side. +* If the organization currently has an unlimited event allowance, the API should reject setting a percentage cap because a percentage of unlimited is not meaningful. +* If an existing percentage cap later encounters an unlimited organization due to a plan change, enforcement should treat the percentage cap as inactive until the organization returns to a finite allowance. + +### Effective project limit calculation + +Use the same effective organization max that current usage enforcement uses. + +```csharp +private static int? GetEffectiveProjectIngestLimit(ProjectIngestLimit? limit, int organizationMaxEvents) +{ + if (limit is null) + return null; + + return limit.Type switch + { + ProjectIngestLimitType.Fixed when limit.FixedLimit is > 0 => + organizationMaxEvents < 0 + ? limit.FixedLimit.Value + : Math.Min(limit.FixedLimit.Value, organizationMaxEvents), + ProjectIngestLimitType.PercentOfOrganizationLimit + when organizationMaxEvents > 0 && limit.PercentOfOrganizationLimit is > 0 => + Math.Max(1, (int)Math.Ceiling(organizationMaxEvents * (limit.PercentOfOrganizationLimit.Value / 100m))), + _ => null + }; +} +``` + +### UsageService API + +Add project-aware/budget-aware event allowance calculation. + +The final event count accepted from a parsed batch should be the minimum allowed by: + +1. organization hard allowance +2. explicit project event budget, if configured +3. automatic smart throttling sampled allowance, if active + +Avoid a design that returns only a Boolean. The event processing path needs a count. + +Suggested result: + +```csharp +public record EventIngestAllowanceResult +{ + public int EventsAllowed { get; init; } + public int OrganizationEventsLeft { get; init; } + public int? ProjectEventsLeft { get; init; } + public int? EffectiveProjectLimit { get; init; } + public bool IsSmartThrottled { get; init; } + public double? SmartThrottleSampleRate { get; init; } + public string? Reason { get; init; } +} +``` + +Suggested method: + +```csharp +public Task GetEventIngestAllowanceAsync( + Organization organization, + Project project); +``` + +Behavior: + +* Compute organization and project totals from one bulk read of persisted-total/current-bucket/previous-bucket cache keys. +* If organization events left is <= 0, allow 0 events. +* Compute explicit project budget events left if configured. +* Compute smart throttling state/sample allowance if project is noisy. +* Return the minimum allowed count. +* Never return an allowed count greater than submitted event count. +* Never return an allowed count greater than organization events left. + +### Refactor current events-left logic + +The event job's project-aware path uses the organization and project it already loaded. It must not issue repository rereads in normal traffic. Existing middleware may retain the organization-id-only helper for its coarse gate. + +```csharp +public Task GetEventIngestAllowanceAsync( + Organization organization, + Project project) +``` + +Use existing total and bucket cache keys with optional project id. Query the repository's canonically invalidated organization project-count cache only when spike and remaining-allowance checks require fair-share evaluation. + +### OverageMiddleware behavior + +`OverageMiddleware` should continue handling request-level event-post gates: + +* non-event posts are skipped +* missing organization context is rejected +* globally disabled event submission is rejected +* missing content length is rejected +* oversized posts are rejected +* organization hard overage can still be rejected early when there are no events left + +Project-level smart throttling and sampled project-budget behavior should not be implemented solely as early middleware rejection because early rejection drops the entire post and cannot preserve a sample. + +If `OverageMiddleware` detects that the organization has no remaining monthly allowance, it may preserve the existing organization overage rejection behavior. + +If the organization has remaining allowance, project-level controls should be evaluated later in `EventPostsJob` / `UsageService` after events are parsed. + +### Status code behavior + +* Organization monthly/bucket hard overage remains `402 PaymentRequired` where existing behavior already returns that status. +* Oversized submissions continue to return `413 RequestEntityTooLarge`. +* Missing content length continues to return `411 LengthRequired`. +* Missing organization context continues to return `401 Unauthorized`. +* Disabled event submission continues to return `503 ServiceUnavailable`. +* Project smart throttling should not require changing the HTTP response for accepted queued posts; the job may accept a sample and block/discard the rest asynchronously. +* Project hard rejection may return `429 TooManyRequests` only in code paths where the server can make an immediate project-level decision without sacrificing sampled acceptance. For the normal queued event-post path, project throttling should be represented by partial processing and blocked/discarded usage accounting rather than a full-request 429. + +### Headers + +Do not repurpose existing generic API throttle headers for project monthly ingest caps or smart project throttling in this change. + +### API throttling middleware + +Do not change `ThrottlingMiddleware` for this feature. + +### Storage and indexing + +Persisted data: + +* Existing organizations have `BudgetAlertSettings = null`. +* Existing projects have `IngestLimit = null`. +* No migration/backfill is required. +* Null/default behavior must preserve current behavior. + +Elasticsearch: + +* Do not add Elasticsearch mappings for `Organization.BudgetAlertSettings` or `Project.IngestLimit` unless filtering/sorting/searching by those fields is required. +* Avoid reindexing. + +### Organization Usage UI + +Add budget alert settings to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte +``` + +Place above the usage chart. + +Suggested card: + +```text +Budget Alerts +Receive an email when your organization reaches selected percentages of its monthly event allowance. +[ ] Enable budget alerts +Thresholds +[50] [80] [90] +[+ Add threshold] +Current plan allowance: 100,000 events +50% = 50,000 events +80% = 80,000 events +Alerts are sent once per threshold per monthly usage period to organization users who have email notifications enabled. +``` + +### Project Usage UI + +Add project event budget and smart throttling status to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte +``` + +Copy: + +```text +Project Event Budget +Protect your organization's monthly event allowance by limiting how many accepted events this project can use. +This is a cap, not a reservation. Other projects are not guaranteed unused capacity. +``` + +Smart throttling status copy when active: + +```text +Smart throttling is currently active for this project. Exceptionless is accepting a small sample of events so you can continue troubleshooting while protecting your organization's monthly event allowance. +``` + +Do not expose many tuning knobs. + +### Project usage chart behavior + +If `effective_ingest_limit` is not null, display the dashed limit line as the project limit. + +If `effective_ingest_limit` is null, keep showing the organization limit. + +Show smart throttling status as text or badge if active. + +### Accessibility + +* Slider controls must have accessible labels. +* Numeric inputs must have associated labels. +* Computed threshold/effective limit text should update in screen-reader-friendly text. +* Do not rely on color only. +* Save success/error must use existing toast/error patterns. + +### Security + +* Only users authorized to update the organization may update budget alert settings. +* Only users authorized to update the project may update project event budgets. +* Budget alert and smart throttling emails are sent only to verified users with email notifications enabled. +* Project budgets do not grant additional access. +* Users cannot increase organization allowance through project budgets. +* Smart throttling must not allow accepted usage beyond organization hard allowance. + +### Privacy + +No new personal event data is collected. + +Budget alert and smart throttling emails include organization/project names and aggregate usage numbers only. They must not include event payload data, stack data, user-identifying event details, or project-specific event contents. + +### Failure modes + +**Budget alert email queue failure** + +If mail queue enqueue fails, do not block event ingestion. + +**Smart throttling email queue failure** + +If smart throttling notification enqueue fails, do not block event ingestion. + +**Dedupe cache failure** + +If dedupe cache fails, prefer avoiding ingestion failure. The implementation may skip alert sending rather than risk duplicate emails or ingestion failure. + +**Organization/project settings changed after message publication** + +Re-load settings in the work item handler. If the relevant notification is disabled or no longer applicable, skip sending. + +**Project lookup fails** + +If the project cannot be loaded during project-level evaluation, do not block solely due to project-level settings. Continue organization-level enforcement. + +**Invalid persisted settings** + +Treat invalid persisted settings as inactive for enforcement/notification. Do not crash event ingestion. diff --git a/openspec/changes/add-usage-budget-controls/proposal.md b/openspec/changes/add-usage-budget-controls/proposal.md new file mode 100644 index 0000000000..e4cee16752 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/proposal.md @@ -0,0 +1,224 @@ +# Proposal: Add Usage Budget Controls + +## Summary + +Add usage budget controls that help organizations both **warn before overage** and **prevent noisy projects from consuming the full organization event allowance**. + +This change includes three related capabilities: + +1. **Organization budget alert emails** — configurable percentage thresholds such as 50% and 80% of the organization's monthly event allowance. When accepted event usage crosses a configured threshold, Exceptionless sends a budget alert email to eligible organization users. +2. **Automatic smart project throttling** — when a project/environment starts sending enough events to threaten the organization's budget, Exceptionless should automatically isolate throttling to the noisy project where possible, instead of stopping ingestion for the entire organization. +3. **Optional project event budgets** — project-level budget caps that allow customers to explicitly limit how many accepted events a project can consume in the monthly usage period. + +Budget alerts are warning controls. Smart project throttling is the default automatic guardrail. Project event budgets are optional prevention controls for customers who want explicit per-project limits. + +This change addresses customer feedback asking for budget alerts before plan overage: + +> Hi support team +> +> Is there any budget alerts we can configure in terms of the number of events? For example I want to receive an email when i have reached 50% of my plan, and then when 80% of my plan. +> +> This month we have exceeded twice our plan, and we should have a mechanism to prevent this situation. + +It also addresses historical customer feedback from issue #112 requesting smarter throttling: + +- project-level throttling +- email notification when throttling is applied +- 1–5% of errors delivered even under throttling + +## User-visible behavior + +### Organization budget alerts + +Users managing an organization can configure event budget alert thresholds from Organization Settings → Usage. + +Supported behavior: + +- Alerts are disabled by default for existing organizations. +- Users can enable budget alerts and configure one or more percentage thresholds. +- Example thresholds: 50%, 80%, 90%. +- A threshold represents the organization's current monthly event allowance, including active bonus events if those are included by existing usage enforcement. +- When accepted usage crosses a configured threshold for the first time in a monthly usage period, eligible organization users receive an email. +- Each threshold is emailed at most once per monthly usage period. +- Alerts are not sent for unlimited organizations because a percentage of unlimited usage is undefined. +- Alerts do not block event ingestion. +- Existing monthly overage emails remain unchanged. + +### Automatic smart project throttling + +Exceptionless should automatically protect organization usage when one project starts sending a large event spike. + +Supported behavior: + +- Users should not be required to configure smart throttling. +- The system should avoid exposing a large number of throttling options. +- When throttling is needed, it should be scoped to the noisy project where possible rather than blocking all projects in the organization. +- While a project is throttled, Exceptionless should still accept a small sample of events from that project when the organization has remaining monthly allowance. +- The accepted sample should preserve troubleshooting visibility so users can tell whether the issue is still happening and whether a deployed fix helped. +- When smart throttling is applied, eligible organization users should receive an email notification. +- Smart throttling should use remaining monthly allowance and remaining time in the period, not only the static plan size, to avoid throttling customers who still have substantial monthly allowance left. + +### Optional project event budgets + +Users managing a project can configure a Project Event Budget from Project Settings → Usage. + +Supported modes: + +- **Off**: no project-specific cap; current behavior is preserved. +- **Fixed limit**: cap this project at a configured number of accepted events per monthly usage period. +- **Percentage of organization limit**: cap this project at a percentage of the organization's current monthly event allowance. + +Examples: + +- A project with no configured ingest limit can consume organization events as it does today. +- A project with a fixed limit of `20,000` can accept up to `20,000` events in the current monthly usage period, subject to the organization still having events available. +- A project with a `20%` limit in an organization with `100,000` monthly events has an effective project cap of `20,000`. +- If the organization plan changes, percentage-based project caps are recalculated from the organization's current event allowance. + +When a project limit is reached: + +- Event submissions for that project are limited or sampled. +- Event submissions for other projects in the organization are not limited solely because this project reached its cap. +- Blocked event usage is recorded for the project and organization. +- Existing organization overage behavior remains unchanged. + +## Classification + +- **Type:** Feature +- **Affected areas:** Backend/API, billing/usage enforcement, organization model, project model, Redis/cache usage counters, mail/work items, Svelte UI, generated API types, tests +- **OpenSpec justification:** This change affects event ingestion behavior, organization/project usage limits, smart throttling behavior, email notification behavior, API response contracts, persisted organization/project data, Redis/cache usage state, SDK/client expectations, jobs, and Svelte UI/API contracts. + +## Current implementation context + +Exceptionless currently enforces organization event overage in `OverageMiddleware` for event posts. The middleware checks the authenticated organization id, rejects disabled event submission, rejects oversized submissions, and calls `UsageService.GetEventsLeftAsync(organizationId)` before allowing the event post to continue. + +`UsageService` already tracks both organization and project usage counters. `IncrementTotalAsync` increments organization totals and project totals using existing cache keys that accept an optional `projectId`. + +`EventPostsJob` already parses queued event posts, calculates how many events may be processed, processes only allowed events, increments blocked usage for events over the limit, and increments total/discarded usage for processed events. This is the correct place to implement project-level sampled acceptance. + +Exceptionless already has an organization overage email path. Existing usage overage publishes `PlanOverage`, which enqueues `OrganizationNotificationWorkItem`, and the work item handler sends organization notice emails to verified users with email notifications enabled. Budget alerts and project throttling notifications should reuse that mail/work-item pattern with distinct alert types rather than overloading monthly/hourly overage booleans. + +The existing generic API throttling middleware remains separate. It limits API requests over a short period and is not the same as event-plan usage enforcement. + +## Affected areas + +### Backend/API + +- Add organization usage budget alert settings to organization model/API responses. +- Add project event budget fields to project model/API responses. +- Extend usage tracking to publish budget-alert notifications when configured thresholds are crossed. +- Extend usage/event post processing to compute smart project throttling and sampled acceptance. +- Extend usage/event post processing to compute optional project budget limits. +- Preserve existing organization-level overage behavior and status codes. +- Avoid implementing project sampled throttling solely in middleware. + +### Redis/cache + +- Reuse existing organization/project usage counter keys. +- Add budget alert dedupe state keyed by organization, threshold, and monthly usage period. +- Add smart project throttling state or notification dedupe state keyed by organization/project/window where useful. +- Preserve existing usage counter TTL behavior. +- Do not add new project usage counter families unless required for efficient limit evaluation. + +### Mail/work items + +- Reuse the existing organization notification pattern. +- Add budget-alert-specific message/work item/template. +- Add project-smart-throttling-specific message/work item/template. +- Send emails only to verified users with email notifications enabled, matching existing organization notice behavior. + +### Svelte UI + +- Add organization budget alert controls to the Svelte Organization Settings → Usage page. +- Add project event budget controls to the Svelte Project Settings → Usage page. +- Surface smart throttling status where feasible, especially on Project Usage. +- Show computed thresholds/caps and current usage. +- Update the project usage chart to display the project limit when configured. + +### Legacy Angular UI + +- Legacy Angular parity is out of scope for this change unless the release path requires these settings to be available in the legacy UI. +- This change must not break existing legacy Angular organization or project management behavior. + +### SDK/client compatibility + +- Existing event submission authentication mechanisms must continue to work. +- Existing project API keys and user tokens must continue to work. +- Organization overage behavior must remain compatible. +- Smart project throttling for queued event posts should prefer asynchronous sampled processing over changing the HTTP submission response. +- Budget alert emails must be asynchronous side effects and must not change normal event submission success responses. + +### Tests + +- Add backend unit/integration tests for budget threshold calculation, smart throughput calculation, sampled acceptance, and project budget computation. +- Add usage service tests for threshold crossing, dedupe, organization cap, project cap, and smart throttling behavior. +- Add event post job tests for sampled acceptance and blocked usage accounting. +- Add API tests for organization budget alert and project event budget update/validation. +- Add mail/work-item tests for budget alert and smart throttling email eligibility/dedupe behavior. +- Add Svelte UI tests or targeted manual QA for the Usage page controls. +- Update HTTP samples if request/response contracts are changed. + +## Compatibility risks + +| Risk | Mitigation | +|------|------------| +| Existing event submissions could be rejected unexpectedly | Project event budgets default to null/Off for all existing projects. Smart throttling should preserve sampled acceptance when organization allowance remains. | +| Existing organization overage behavior could change | Keep organization-level monthly/bucket enforcement intact. Organization hard overage remains the outer limit. | +| Middleware-level project rejection would drop 100% of events | Project smart throttling and sampled acceptance must be implemented in the event processing path after events are parsed, not only in `OverageMiddleware`. | +| Too much configuration conflicts with product direction | Smart project throttling must work automatically with minimal or no required user configuration. | +| Existing customers expect some data while throttled | When the organization still has allowance, throttled projects should retain a small accepted sample rather than being fully blocked. | +| Static monthly plan throttling can feel unfair late in the month | Throughput calculations should consider events remaining in the monthly period and time remaining in the monthly period. | +| Project cap could be confused with reserved quota | UI copy must state that this is a cap, not a reservation. Other projects are not guaranteed unused capacity. | +| Percentage caps could behave unexpectedly when organization plans change | Effective percentage caps are recalculated from the current organization event allowance each time limits are evaluated. | +| Unlimited organization plans make percentage caps ambiguous | Percentage mode must be disabled or invalid for unlimited organizations. Fixed limits remain supported for project event budgets. Budget alert percentages are unavailable for unlimited organizations. | +| Budget alerts could send unexpected emails | Default budget alert settings to disabled for existing organizations. Users must opt in. | +| Duplicate threshold emails could annoy users | Each configured threshold is sent at most once per organization per monthly usage period. | +| Existing overage emails could change | Keep existing `PlanOverage` monthly/hourly emails unchanged and add budget alerts and smart throttling emails as separate notification types. | +| Persisted project or organization model changes could require Elasticsearch mapping/reindex | Store new settings source-only unless filtering/searching is added. No Elasticsearch mapping or reindex is required for enforcement. | + +## Non-goals + +- Do not implement API-key/token-level ingest limits in this change. +- Do not implement rate-based billing. +- Do not implement reserved project quotas. +- Do not change plan billing, invoices, Stripe integration, or organization plan limits. +- Do not change generic API request throttling behavior in `ThrottlingMiddleware`. +- Do not add new public event-submission endpoints. +- Do not migrate historical usage data. +- Do not require Elasticsearch reindexing. +- Do not require existing organizations or projects to be backfilled. +- Do not implement per-recipient custom alert thresholds in this change. +- Do not implement Slack/webhook budget alerts in this change. +- Do not implement stack-level adaptive throttling in v1 unless it can be done with low risk; project-level isolation is the v1 scope. +- Do not expose a large set of smart throttling tuning options in the UI. + +## Future considerations + +This design should leave room for future layers: + +1. Organization plan/event allowance. +2. Organization budget alert thresholds. +3. Automatic smart project throttling. +4. Optional project event budget. +5. API key/token ingest cap. +6. Future rate-based or usage-based billing. + +Budget alerts should leave room for future channels and scopes: + +- fixed event-count alert thresholds +- project-specific budget alerts +- token/API-key-specific budget alerts +- Slack/webhook alert delivery +- role-based recipients +- rate-based billing alerts + +The original issue suggested stack-level adaptive throttling with approximate occurrence counts. This change should leave room for that future direction, but v1 should focus on project-level isolation because it is simpler, aligns with project usage counters already tracked by Exceptionless, and directly addresses the customer problem of one product/environment affecting all others. + +## Rollback plan + +- Because project event budgets are opt-in, disabling or removing the UI leaves existing unset project budgets unchanged. +- Because budget alerts are opt-in, disabling or removing the UI leaves existing unset organization alert settings unchanged. +- If smart project throttling causes issues, smart throttling can be disabled while leaving budget alerts and optional project budgets in place. +- If budget alert emails cause issues, budget alert publishing/subscribers can be disabled while leaving stored settings in place. +- Existing organizations and projects default to null/Off, so rolling back model/API exposure has no effect on existing ingestion behavior unless users have already configured settings. +- If stored settings must be disabled quickly, project budget and budget alert evaluation can be short-circuited to treat all settings as unset. diff --git a/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md new file mode 100644 index 0000000000..518366e482 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md @@ -0,0 +1,166 @@ +# Spec: API Compatibility + +## MODIFIED Requirements + +### Requirement: Event submission MUST preserve existing organization overage behavior + +Exceptionless event submission MUST preserve existing organization overage and API authentication behavior while adding automatic smart project throttling and optional project event budgets. + +#### Scenario: Organization overage status remains unchanged + +Given an organization has no remaining event allowance +When an event is submitted for a project in that organization +Then the response status code must remain the existing organization overage status code. + +#### Scenario: Project smart throttling preserves queued event post compatibility + +Given an organization has remaining event allowance and a project is under smart throttling +When an event post is submitted through the normal queued event submission path +Then the HTTP response should not be changed solely to report project throttling. + +#### Scenario: Generic API throttle behavior is unchanged + +Given API request throttling is enabled +When non-event API requests are made +Then existing generic API request throttle behavior must remain unchanged. + +#### Scenario: Existing client API keys continue to authenticate + +Given an existing project API key is used for event submission and the project has no configured event budget +When an event is submitted +Then authentication and authorization behavior must match existing behavior. + +### Requirement: Rate-limit headers MUST NOT be repurposed for project monthly controls + +Existing generic API throttle response headers MUST NOT be redefined to mean project monthly event budgets or smart project throttling. + +#### Scenario: Generic API throttle headers remain request-rate headers + +Given generic API throttling emits request-rate headers +When usage budget controls are added +Then those headers must continue to represent generic API request throttling behavior. + +## ADDED Requirements + +### Requirement: Organization API exposes budget alert settings + +The organization API may expose and update budget alert settings as an additive organization field. + +#### Scenario: Organization response includes budget alert settings + +Given an organization has budget alert settings +When the organization is returned from the API +Then the response must include budget_alert_settings. + +#### Scenario: Existing organization response remains compatible + +Given an existing organization has no budget alert settings +When the organization is returned from the API +Then budget_alert_settings may be null and existing fields must remain unchanged. + +#### Scenario: Authorized user updates budget alert settings + +Given a user is authorized to update an organization +When the user updates budget alert settings with valid thresholds +Then the API must persist the settings and include them in the response. + +#### Scenario: Invalid budget alert settings rejected + +Given a user is authorized to update an organization +When the user submits invalid budget alert settings +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Budget alert emails are asynchronous side effects + +Budget alert emails are side effects of accepted usage threshold crossing and must not change event submission success responses. + +#### Scenario: Event crosses budget alert threshold + +Given budget alerts are enabled and an accepted event causes usage to cross a threshold +When the event submission completes +Then the event submission response must remain the normal accepted response. + +#### Scenario: Budget alert failure does not reject event + +Given budget alerts are enabled and budget alert queuing fails +When an event is submitted +Then the event must not be rejected solely because budget alert queuing failed. + +## ADDED Requirements + +### Requirement: Project API accepts event budget configuration + +The project update API may accept an optional project event budget configuration as an additive project field. + +#### Scenario: Clear project event budget + +Given an authorized user can update a project +When the user patches the project with ingest_limit set to null +Then the project event budget must be cleared. + +#### Scenario: Set fixed project event budget + +Given an authorized user can update a project +When the user patches the project with a fixed positive ingest_limit +Then the project must persist the fixed event budget. + +#### Scenario: Set percentage project event budget + +Given an authorized user can update a project and the organization has finite allowance +When the user patches the project with a valid percentage event budget +Then the project must persist the percentage event budget. + +#### Scenario: Invalid fixed limit is rejected + +Given an authorized user can update a project +When the user patches the project with a fixed event budget less than or equal to 0 +Then the API must reject the request with a validation error. + +#### Scenario: Invalid percentage limit is rejected + +Given an authorized user can update a project +When the user patches the project with a percentage event budget less than or equal to 0 or greater than 100 +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Project response MUST include budget and throttling state + +Project responses used by the UI MUST expose the configured project event budget, currently effective cap, and smart throttling state where available. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget +When the project is returned from the API +Then ingest_limit must be null and effective_ingest_limit must be null. + +#### Scenario: Project has fixed event budget + +Given a project has a fixed event budget +When the project is returned from the API +Then effective_ingest_limit must contain the currently effective cap. + +#### Scenario: Project is smart-throttled + +Given a project is currently under smart throttling +When the project is returned from the API +Then the response should indicate smart throttling state where feasible. + +### Requirement: Nullable complex PATCH contracts MUST be generated canonically + +Organization and project settings MUST be clearable with JSON null, and their OpenAPI contracts MUST reference the canonical complex component schemas. + +#### Scenario: Clear organization budget settings + +Given an authorized user can update an organization +When the user patches budget_alert_settings with null +Then the settings must be cleared. + +#### Scenario: Delta OpenAPI schema + +Given a nullable complex property belongs to a Delta PATCH model +When OpenAPI is generated +Then the property must be represented as null or a component schema reference +And generated clients must come from that document rather than manual snapshot edits. diff --git a/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md new file mode 100644 index 0000000000..53833b64cc --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md @@ -0,0 +1,364 @@ +# Spec: Event Ingestion + +## ADDED Requirements + +### Requirement: Smart throttling works without required user configuration + +Exceptionless must provide automatic smart project throttling without requiring users to configure per-project or per-stack throttling settings. + +#### Scenario: No smart throttling configuration exists + +Given an organization has projects +And no smart throttling settings have been configured +When one project starts sending a large event spike +Then Exceptionless should still be able to apply smart throttling automatically. + +#### Scenario: UI does not require many throttling options + +Given smart throttling is available +When a user views organization or project usage settings +Then the UI must not require the user to configure many low-level throttling parameters before smart throttling can protect the organization. + +### Requirement: Smart throttling uses remaining allowance and remaining time + +Smart throttling must calculate allowed throughput using the organization's remaining monthly event allowance and the time remaining in the monthly usage period. + +#### Scenario: Organization has substantial allowance remaining late in the month + +Given an organization is late in the monthly usage period +And the organization still has substantial event allowance remaining +When smart throttling calculates allowed throughput +Then the calculation must account for remaining event allowance +And must not rely only on the static monthly plan size divided by the original month duration. + +#### Scenario: Organization has little allowance remaining + +Given an organization has little event allowance remaining +When smart throttling calculates allowed throughput +Then the allowed throughput should decrease to protect the remaining monthly allowance. + +### Requirement: Smart throttling isolates noisy projects where possible + +When one project is responsible for a usage spike, smart throttling must apply to that project where possible rather than fully throttling the entire organization. + +#### Scenario: One project spikes + +Given an organization has Project A and Project B +And Project A sends a large event spike +And Project B remains within normal usage +When smart throttling is applied +Then Project A may be throttled or sampled +And Project B must not be blocked solely because Project A spiked. + +#### Scenario: Multiple projects spike + +Given an organization has multiple projects +And more than one project exceeds smart throttling criteria +When smart throttling is applied +Then each noisy project may be evaluated and throttled independently. + +### Requirement: Smart throttling preserves a sample of events + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless must continue accepting a small sample of events for that project. + +#### Scenario: Project is throttled but organization has allowance + +Given an organization has remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then Exceptionless must accept a small sample of events +And must block or discard the remainder according to usage accounting rules. + +#### Scenario: Sample rate remains small + +Given a project is under smart throttling +When the accepted sample is calculated +Then the accepted sample should be within the intended 1 to 5 percent range +Unless a lower organization or project hard limit leaves fewer events available. + +#### Scenario: Organization allowance exhausted + +Given an organization has no remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then existing organization overage behavior must apply +And smart throttling must not allow sampled events beyond the organization hard allowance. + +### Requirement: Smart throttling operates after event posts are parsed + +Smart project throttling must be evaluated in a processing path that knows the number of events in the post so it can accept a sample and block the remainder. + +#### Scenario: Batch contains more events than allowed + +Given a queued event post contains many events +And smart throttling allows only a sample +When the event post is processed +Then the processor must process only the sampled/allowed events +And must record blocked usage for the rest. + +#### Scenario: Full request rejection would prevent sampling + +Given an event post would otherwise be rejected entirely by project-level throttling +When the organization still has remaining allowance +Then the system should prefer sampled processing over full request rejection. + +### Requirement: Smart throttling sends project throttling notification + +When smart throttling is applied to a project, Exceptionless must send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled +And smart throttling is applied to the project +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed while the project remains throttled +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization +And the user does not have a verified email address or has email notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. + +### Requirement: Smart throttling usage accounting remains accurate + +Smart throttling must record accepted, blocked, and discarded event counts consistently with existing usage accounting. + +#### Scenario: Sampled events accepted + +Given a project is smart-throttled +When a sample of events is accepted +Then accepted organization and project usage must increase by the number of processed events. + +#### Scenario: Non-sampled events blocked + +Given a project is smart-throttled +When some events in a post are not accepted due to throttling +Then blocked usage must increase by the number of non-accepted events. + +#### Scenario: Budget alerts use accepted events only + +Given budget alerts are enabled and a project is smart-throttled +When non-sampled events are blocked +Then blocked events must not count as accepted usage for organization budget alert thresholds. + +### Requirement: Smart throttling does not replace explicit project budgets + +Automatic smart throttling must coexist with optional project event budgets. + +#### Scenario: Project has no explicit budget + +Given a project has no configured project event budget +When the project spikes +Then automatic smart throttling may still apply. + +#### Scenario: Project has explicit budget + +Given a project has an explicit project event budget and automatic smart throttling also applies +When events are processed +Then the number of events accepted must not exceed the lowest applicable allowance from organization hard limit, project budget, and smart throttling sample. + +## ADDED Requirements + +### Requirement: Projects MUST be able to define optional event budgets + +Exceptionless projects MUST be able to define an optional project-level event budget that caps or limits the number of accepted events for that project within the current monthly usage period. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget and the organization has remaining event allowance +When an event is submitted for the project +Then the explicit project event budget must not block the event +And existing organization-level usage enforcement and automatic smart throttling must apply. + +#### Scenario: Existing projects remain uncapped + +Given a project existed before project event budgets were introduced +When the project is loaded +Then its project event budget must be treated as unset. + +### Requirement: Fixed project event budgets MUST cap accepted project events + +A fixed project event budget MUST cap accepted event volume for a project to a configured positive integer for the current monthly usage period. + +#### Scenario: Fixed project budget allows events below cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted fewer than 20000 events in the current monthly usage period +When an event is submitted for the project +Then the event must not be blocked by the explicit project event budget. + +#### Scenario: Fixed project budget limits events at cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted 20000 events in the current monthly usage period +When events are submitted for the project +Then accepted events for that project must be limited by the project event budget +And blocked usage must be recorded for non-accepted events. + +#### Scenario: Fixed project budget above organization allowance is clamped + +Given an organization has a finite monthly event allowance of 10000 events +And a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 10000 events. + +#### Scenario: Fixed project budget on unlimited organization + +Given an organization has unlimited event allowance and a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +### Requirement: Percentage project event budgets MUST derive from organization allowance + +A percentage project event budget MUST cap accepted event volume for a project to a percentage of the organization's current finite monthly event allowance. + +#### Scenario: Percentage project budget computes effective cap + +Given an organization has a finite monthly event allowance of 100000 events +And a project has a percentage event budget of 20 percent +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +#### Scenario: Percentage project budget on unlimited organization is inactive + +Given an organization has unlimited event allowance and a project has a percentage event budget +When the effective project event budget is evaluated +Then the percentage project event budget must not produce an effective project cap. + +### Requirement: Project event budgets MUST be caps, not reservations + +Project event budgets MUST prevent a project from exceeding its cap but must not reserve organization event allowance for other projects. + +#### Scenario: Unused project budget does not reserve events + +Given an organization has a monthly event allowance of 100000 events +And Project A has a percentage event budget of 20 percent and uses 0 events +When Project B submits events +Then Project B must not be blocked solely because Project A has unused project budget. + +#### Scenario: Project budget does not increase organization allowance + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project event budget exhaustion MUST limit only the capped project + +When one project reaches its project event budget, other projects in the same organization MUST continue to be evaluated independently. + +#### Scenario: One capped project does not block another project + +Given an organization has remaining event allowance +And Project A has reached its project event budget and Project B has no project event budget +When an event is submitted for Project B +Then the event must not be blocked because Project A reached its budget. + +#### Scenario: Multiple projects have independent budgets + +Given an organization has remaining event allowance +And Project A has a fixed event budget of 1000 events and Project B has a fixed event budget of 2000 events +When Project A reaches 1000 accepted events +Then Project A must be limited by its project event budget +And Project B must continue to be evaluated against its own 2000-event budget. + +### Requirement: Organization overage MUST remain the hard outer limit + +Organization-level event allowance MUST remain the hard billing/usage boundary. + +#### Scenario: Organization overage takes precedence + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project ingest controls MUST reuse existing usage accounting + +Project budget and smart throttling enforcement MUST use existing project usage counters. + +#### Scenario: Accepted event increments project usage + +Given an organization has remaining event allowance and a project has remaining project event budget +When an event is accepted for the project +Then existing project accepted event usage must be incremented. + +#### Scenario: Blocked event does not count as accepted project usage + +Given a project has reached its project event budget +When an event is submitted for the project +Then accepted project usage must not increase for non-accepted events +And blocked usage must be recorded. + +### Requirement: Project controls MUST support safe defaults on failure + +Project budget and smart throttling evaluation MUST avoid introducing new ingestion failures when project data is missing or invalid. + +#### Scenario: Missing project does not cause project-budget rejection + +Given an event submission has an organization id and the project cannot be loaded +When project controls are evaluated +Then the system must not reject the event solely due to project budget evaluation. + +#### Scenario: Invalid persisted project budget is treated as inactive + +Given a project has an invalid persisted event budget configuration +When event ingestion evaluates project controls +Then the invalid project budget must be treated as inactive. + +### Requirement: Project ingest controls MUST prefer sampled processing over full request rejection + +Project-level ingest controls MUST preserve sampled visibility where possible. + +#### Scenario: Queued event post contains many project events + +Given an event post has been queued and the project is over its smart throttling allowance +And the organization still has remaining allowance +When the event post is processed +Then the job should process an allowed sample and record blocked usage for non-sampled events. + +#### Scenario: Middleware cannot sample project events + +Given a project-level condition would require preserving only 1 to 5 percent of events +When the request is still in middleware before parsing +Then middleware must not be the only enforcement layer. + +### Requirement: Smart throttling MUST use stable five-percent sampling + +Automatic project throttling MUST select events with a stable five-percent hash sample across the complete parsed post. + +#### Scenario: Single-event post is genuinely sampled + +Given a project is smart-throttled and the organization has remaining allowance +When repeated single-event posts with stable distinct identities are processed +Then approximately five percent must be accepted over time +And no per-post minimum may force every single event through. + +#### Scenario: Nonaccepted events are counted once + +Given organization allowance, explicit project budget, and smart sampling are evaluated for one parsed post +When the final accepted set is selected +Then every event outside that set must increment blocked usage exactly once +And those events must not also increment discarded usage. + +### Requirement: Smart throttling MUST be cheap and operationally disableable + +Automatic project throttling MUST use bucket-inclusive totals, five-minute remaining windows, the existing ten-times burst tolerance, and current noisy-project fair share. + +#### Scenario: Normal unlimited traffic avoids extra work + +Given an organization has unlimited allowance and the project has no explicit cap +When ingestion evaluates the already-loaded organization and project +Then it must not reload either model or query the organization project count. + +#### Scenario: Kill switch is disabled + +Given EnableSmartProjectThrottling is false +When a noisy project submits events +Then automatic sampling must be bypassed +And explicit project and organization limits must remain enforced. diff --git a/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md new file mode 100644 index 0000000000..12eb011936 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md @@ -0,0 +1,218 @@ +# Spec: Jobs, Notifications & Queues + +## ADDED Requirements + +### Requirement: Organizations may configure budget alert thresholds + +Organizations may configure event budget alert thresholds as percentages of their effective monthly event allowance. Budget alerts are disabled by default. + +#### Scenario: Existing organization has no budget alerts + +Given an organization existed before budget alerts were introduced +When the organization is loaded +Then budget alerts must be treated as disabled. + +#### Scenario: Enable budget alerts with thresholds + +Given an authorized user can update an organization +When the user enables budget alerts with thresholds 50 and 80 +Then the organization must store budget alert settings as enabled with thresholds 50 and 80. + +#### Scenario: Disable budget alerts + +Given an organization has enabled budget alerts +When an authorized user disables budget alerts +Then budget alert emails must no longer be sent for that organization. + +### Requirement: Budget alert thresholds validate percentage values + +Budget alert threshold percentages must be valid pre-overage warning thresholds. + +#### Scenario: Reject zero threshold + +Given an authorized user can update budget alert settings +When the user configures threshold 0 +Then the API must reject the request with a validation error. + +#### Scenario: Reject threshold at or above one hundred percent + +Given an authorized user can update budget alert settings +When the user configures threshold 100 or above +Then the API must reject the request with a validation error. + +#### Scenario: Enabled alerts require at least one threshold + +Given an authorized user can update budget alert settings +When the user enables budget alerts with no thresholds +Then the API must reject the request with a validation error. + +### Requirement: Budget alerts use effective organization allowance + +Budget alert thresholds must be evaluated against the same effective organization event allowance used by existing usage enforcement. + +#### Scenario: Threshold uses plan allowance + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 +When the threshold event count is calculated +Then the threshold event count must be 50000. + +#### Scenario: Unlimited organization has no percentage budget alert + +Given an organization has unlimited event allowance +When budget alert thresholds are evaluated +Then percentage budget alerts must be inactive and no budget alert email must be sent. + +### Requirement: Budget alerts send when accepted usage crosses thresholds + +Budget alert emails must be triggered when accepted organization event usage crosses a configured threshold for the first time in a monthly usage period. + +#### Scenario: Usage below threshold does not send alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49998 +When one event is accepted +Then no budget alert email must be sent. + +#### Scenario: Usage crossing threshold sends alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49999 +When one event is accepted +Then a 50 percent budget alert email must be queued. + +#### Scenario: Usage already above threshold does not resend alert + +Given the 50 percent alert has already been sent in the current monthly usage period +When another event is accepted +Then another 50 percent budget alert email must not be queued. + +#### Scenario: Large batch crosses multiple thresholds + +Given budget alerts are enabled for thresholds 50 and 80 and accepted event usage is 45000 +When a batch of 40000 events is accepted +Then both the 50 percent and 80 percent budget alert emails must be queued. + +### Requirement: Budget alert emails are sent once per threshold per monthly usage period + +Each configured threshold may generate at most one budget alert email per organization per monthly usage period. + +#### Scenario: Threshold already sent in period + +Given the 80 percent alert was sent earlier in the current monthly usage period +When usage remains above 80 percent +Then no additional 80 percent budget alert email must be sent. + +#### Scenario: New monthly period resets sent threshold state + +Given an organization received an 80 percent alert in the previous monthly usage period +When usage crosses 80 percent in the new period +Then a new 80 percent budget alert email may be sent. + +### Requirement: Budget alert emails use organization email notification eligibility + +Budget alert emails must be sent only to organization users who are eligible for existing organization email notices. + +#### Scenario: Verified user with email notifications enabled receives alert + +Given a user belongs to an organization with a verified email and email notifications enabled +When a budget alert is sent for the organization +Then the user must receive the budget alert email. + +#### Scenario: Unverified user does not receive alert + +Given a user belongs to an organization and does not have a verified email address +When a budget alert is sent for the organization +Then the user must not receive the budget alert email. + +### Requirement: Budget alerts do not block ingestion + +Budget alert processing must not block accepted event ingestion. + +#### Scenario: Alert email enqueue fails + +Given budget alerts are enabled and usage crosses a configured threshold +When the budget alert email cannot be queued +Then the accepted event must not be rejected solely because the alert email failed. + +### Requirement: Budget alerts preserve existing overage notifications + +Budget alerts must not replace or suppress existing monthly/hourly overage notifications. + +#### Scenario: Existing monthly overage still sends + +Given an organization reaches or exceeds its monthly event allowance +When existing monthly overage notification behavior is triggered +Then the existing monthly overage email must still be sent. + +### Requirement: Budget alert emails are separate from smart throttling emails + +Organization budget alerts and project smart throttling notifications must be distinct notifications. + +#### Scenario: Project throttling below budget threshold + +Given organization budget alerts are enabled and accepted usage has not crossed a threshold +When a project enters smart-throttled state +Then no organization budget threshold email must be sent solely because smart throttling was applied. + +## ADDED Requirements + +### Requirement: Smart throttling MUST send project throttling notification email + +When smart throttling is applied to a project, Exceptionless MUST send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled and smart throttling is applied +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization and does not have a verified email or has notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. + +### Requirement: Budget alert crossings MUST be current and atomically deduplicated + +Budget alert publication MUST compare bucket-inclusive previous/current accepted totals and atomically claim each organization-threshold-period key. + +#### Scenario: Concurrent crossing + +Given concurrent accepted batches cross the same configured threshold +When each publisher attempts AddAsync for the dedupe key +Then only the successful claimant may publish the alert. + +#### Scenario: Alerts enabled above threshold + +Given accepted usage already exceeds a threshold while alerts are disabled +When an authorized user enables that threshold +Then it must be evaluated once immediately after settings are saved. + +#### Scenario: Invalid disabled threshold + +Given budget alert settings exist but are disabled +When a threshold is outside 1 through 99 +Then validation must reject the settings. + +### Requirement: Delayed usage notifications MUST suppress stale mail + +Notification handlers MUST re-check current persisted settings, plan, bucket-inclusive usage, and throttle state before sending. + +#### Scenario: Budget alert becomes stale + +Given a budget alert work item is queued +When the threshold is removed, alerts are disabled, usage is below the current threshold count, or the plan becomes unlimited before handling +Then the handler must not send the email. + +#### Scenario: Smart throttle becomes stale + +Given a smart-throttle work item is queued +When the operational kill switch is disabled or the project's current throttle state expires before handling +Then the handler must not send the email. diff --git a/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md new file mode 100644 index 0000000000..18c006aa47 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md @@ -0,0 +1,115 @@ +# Spec: Organizations, Projects, Users & Auth + +## ADDED Requirements + +### Requirement: Budget alert settings belong to an organization + +Budget alert settings belong to an organization and may be changed only by users authorized to update that organization. + +#### Scenario: Authorized organization user updates budget alerts + +Given a user is authorized to update an organization +When the user updates budget alert settings +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update budget alerts + +Given a user is not authorized to update an organization +When the user attempts to update budget alert settings +Then the API must reject the operation according to existing organization authorization behavior. + +## ADDED Requirements + +### Requirement: Budget alert email recipients respect user preferences + +Budget alert email recipients must respect existing user email verification and email notification preferences. + +#### Scenario: User email notifications disabled + +Given a user belongs to an organization and has email notifications disabled +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: User email unverified + +Given a user belongs to an organization and the user's email address is not verified +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: Verified user with email notifications enabled + +Given a user belongs to an organization with a verified email and notifications enabled +When a budget alert is sent +Then the user may receive the budget alert email. + +## ADDED Requirements + +### Requirement: Project event budget authorization follows project authorization + +A project event budget belongs to a project and is scoped by the project's owning organization. Only users authorized to update the project may configure its event budget. + +#### Scenario: Authorized project user updates project event budget + +Given a user is authorized to update a project +When the user updates the project's event budget +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update project event budget + +Given a user is not authorized to update a project +When the user attempts to update the project's event budget +Then the API must reject the operation according to existing project authorization behavior. + +#### Scenario: Project event budget cannot grant organization capacity + +Given a user configures a project event budget +When events are submitted for the project +Then the project event budget must not allow accepted event usage beyond the organization's effective allowance. + +## ADDED Requirements + +### Requirement: Budget controls preserve existing token and auth behavior + +Budget alert settings, smart project throttling, and project event budget configuration must not change API key authentication, token scopes, or user authorization roles. + +#### Scenario: Existing project token behavior is preserved + +Given an existing project API key has client scope and the project has no configured event budget +When the key is used for event submission +Then token authentication and scope behavior must match existing behavior. + +#### Scenario: Project smart throttling does not disable API key + +Given a project is under smart throttling +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project is smart-throttled. + +#### Scenario: Project budget does not disable API key + +Given a project reaches its project event budget +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project reached its event budget. + +#### Scenario: Disabled or suspended token remains rejected + +Given a project has remaining project event budget and the API key is disabled or suspended +When an event is submitted with that API key +Then the event must still be rejected due to token authentication state. + +## ADDED Requirements + +### Requirement: Project event budget MUST NOT preclude future token-level caps + +Project event budget and smart throttling behavior MUST NOT preclude future API-key/token-level ingest caps. + +#### Scenario: Future token cap is absent + +Given token-level ingest caps are not implemented +When a project event budget or smart throttling is evaluated +Then enforcement must consider organization and project scopes only. + +#### Scenario: Project budget remains independent of token count + +Given a project has multiple API keys and a configured project event budget +When events are submitted using different API keys for that project +Then all accepted events for the project must count against the same project event budget. diff --git a/openspec/changes/add-usage-budget-controls/tasks.md b/openspec/changes/add-usage-budget-controls/tasks.md new file mode 100644 index 0000000000..d63e599416 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/tasks.md @@ -0,0 +1,368 @@ +# Tasks: Add Usage Budget Controls + +## OpenSpec + +- [x] 1. Create OpenSpec change + * Create proposal, design, tasks, and spec deltas for: + * organization budget alerts + * automatic smart project throttling + * project event budgets + * API compatibility + * organization/project auth behavior + * Verification: + * openspec validate add-usage-budget-controls --strict --no-interactive + +## Backend model and API contract + +- [x] 2. Add organization budget alert settings model + * Add OrganizationBudgetAlertSettings. + * Add nullable BudgetAlertSettings to Organization. + * Validate every present threshold as 1-99 even while disabled; enabled settings require at least one threshold. + * Verification: + * dotnet build + * Add model/validation tests if existing model validation tests are available. +- [x] 3. Add budget alert settings to organization API contract + * Add BudgetAlertSettings to ViewOrganization. + * Prefer adding UpdateOrganization with Name and BudgetAlertSettings. + * Update OrganizationController generic update DTO if using UpdateOrganization. + * Regenerate generated API and Svelte schemas. + * Verification: + * dotnet build + * dotnet test -- --filter-class OrganizationControllerTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check +- [x] 4. Add project event budget domain model + * Add ProjectIngestLimit and ProjectIngestLimitType under src/Exceptionless.Core/Models/. + * Add nullable Project.IngestLimit. + * Add validation for fixed and percentage modes. + * Verification: + * dotnet build + * Add/adjust serializer tests if project model serialization coverage exists. +- [x] 5. Add project event budget to project DTOs + * Add ProjectIngestLimit? IngestLimit to UpdateProject. + * Add ProjectIngestLimit? IngestLimit to ViewProject. + * Add int? EffectiveIngestLimit to ViewProject. + * Add smart throttling state fields to ViewProject if feasible: + * bool IsSmartThrottled + * double? SmartThrottleSampleRate + * Ensure Mapperly maps the new fields or add explicit mapping if required. + * Verification: + * dotnet build + * dotnet test -- --filter-class ProjectControllerTests +- [x] 6. Regenerate OpenAPI and Svelte generated types + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts. + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts. + * Confirm UpdateOrganization, ViewOrganization, UpdateProject, ViewProject, and validation schemas include new fields. + * Generate nullable complex Delta properties as `null | $ref` from the canonical transformer and add regression coverage. + * Verification: + * dotnet build + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + +## Organization budget alert emails + +- [x] 7. Add budget alert threshold calculation + * Use the same effective organization allowance as existing usage enforcement. + * Support configured percentage thresholds from 1 through 99. + * Treat thresholds as inactive for unlimited organizations. + * Verification: + * Add tests to UsageServiceTests. + * dotnet test -- --filter-class UsageServiceTests +- [x] 8. Publish budget alert messages when thresholds are crossed + * Add OrganizationBudgetAlert message. + * In UsageService.IncrementTotalAsync, detect threshold crossings after accepted event usage increments. + * Compare bucket-inclusive previous/current totals and immediately evaluate newly enabled thresholds after settings save. + * Publish one message per crossed threshold. + * Do not publish repeatedly after threshold was already sent in the current monthly usage period. + * Verification: + * Tests: + * below threshold does not publish + * crossing threshold publishes + * already-sent threshold does not publish + * batch crossing multiple thresholds publishes each crossed threshold + * dotnet test -- --filter-class UsageServiceTests +- [x] 9. Add budget alert dedupe + * Add dedupe key per organization, threshold, and monthly usage period. + * Atomically claim each key with `AddAsync` and set TTL through the end of the monthly usage period plus a safety buffer. + * Prefer non-blocking behavior if dedupe cache fails. + * Verification: + * Add tests for one-email-per-threshold-per-period behavior. + * dotnet test -- --filter-class UsageServiceTests +- [x] 10. Add budget alert work item and subscriber + * Add OrganizationBudgetAlertWorkItem. + * Add startup subscriber that turns OrganizationBudgetAlert messages into work items. + * Add work item handler that loads organization/users and sends emails. + * Re-check current organization budget alert settings before sending. + * Re-check current usage and effective plan allowance before sending stale queued work. + * Verification: + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * If no dedicated handler test class exists, add one. +- [x] 11. Add budget alert mailer method and template + * Add IMailer.SendOrganizationBudgetAlertAsync. + * Implement the method in Mailer. + * Add organization-budget-alert.html template. + * Include current usage, threshold percentage, threshold event count, event limit, remaining events, and links to organization usage/billing. + * Verification: + * Mailer/template rendering test if existing template tests exist. + * dotnet build + +## Automatic Smart Project Throttling + +- [x] 12. Add smart throughput calculation + * Calculate allowed throughput from events left in the monthly period and time/windows left in the period. + * Use five-minute windows and preserve the existing 10x burst tolerance. + * Avoid static plan-size-only calculations. + * Verification: + * Tests for early-month, late-month, high-remaining, and low-remaining allowance cases. + * dotnet test -- --filter-class UsageServiceTests +- [x] 13. Add project-level smart throttling decision + * Detect projects contributing to usage spikes. + * Apply smart throttling to noisy projects where possible instead of organization-wide full blocking. + * Use the canonically invalidated repository project-count cache only after spike criteria require fair-share evaluation. + * Add `EnableSmartProjectThrottling`, default enabled, as an operational kill switch. + * Verification: + * Tests where Project A spikes and Project B remains unaffected. + * Tests where multiple projects are evaluated independently. + * dotnet test -- --filter-class UsageServiceTests +- [x] 14. Add sampled acceptance for smart-throttled projects + * Implement fixed 5% stable hash sampling across the whole batch, including true sampling of single-event posts. + * Count every nonaccepted event exactly once as blocked, never again as discarded. + * Verification: + * Tests for sampled acceptance count. + * Tests that blocked usage increments for non-sampled events. + * Tests that accepted usage increments only for sampled events. + * dotnet test -- --filter-class EventPostsJobTests +- [x] 15. Move project-level sampled enforcement into event post processing + * Extend EventPostsJob / UsageService after event post parsing to calculate allowed events. + * Avoid using only OverageMiddleware for project-level sampled enforcement. + * Preserve existing organization hard overage middleware behavior. + * Verification: + * Tests for batch event posts where only a sample is processed. + * Tests for organization hard overage still blocking. + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class OverageMiddlewareTests +- [x] 16. Add smart throttling notification message/work item/email + * Add project throttling notification message. + * Add work item and handler. + * Add mailer method and template. + * Send to verified users with email notifications enabled. + * Deduplicate notifications during a throttling cooldown window. + * Re-check current plan, usage, and throttle state before sending; email fields describe project usage and fair-share limits. + * Add a dedicated smart-throttled event metric and transition-only structured logging. + * Verification: + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet build + +## Optional project event budgets + +- [x] 17. Refactor UsageService events-left calculation + * Make one allowance decision from the organization/project models already loaded by EventPostsJob. + * Bulk-read bucket-inclusive organization/project totals using existing project-aware cache key helpers. + * Preserve current organization behavior. + * Verification: + * Add tests for organization-only events-left behavior to ensure no regression. + * dotnet test -- --filter-class UsageServiceTests +- [x] 18. Add project effective budget calculation + * Implement effective fixed budget calculation. + * Implement effective percentage budget calculation. + * Clamp fixed budgets to finite organization allowance during enforcement. + * Treat percentage budgets as inactive when organization allowance is unlimited. + * Verification: + * dotnet test -- --filter-class UsageServiceTests +- [x] 19. Add parsed-event ingest allowance result + * Add EventIngestAllowanceResult. + * Add UsageService.GetEventIngestAllowanceAsync(Organization organization, Project project) or equivalent without repository reloads. + * Return event count allowed, organization remaining, project remaining, effective project budget, and smart throttling state. + * Verification: + * Tests: + * no project id or missing project falls back safely + * no project budget uses organization + smart throttling + * fixed budget limits allowed event count + * percentage budget computes effective cap from organization allowance + * organization overage takes precedence over project controls + * smart throttling sample and project budget combine by taking the minimum allowed count + * dotnet test -- --filter-class UsageServiceTests + +## Middleware and event processing + +- [x] 20. Keep OverageMiddleware as coarse gate + * Preserve existing status codes for organization overage, disabled submission, missing content length, oversized posts, and unauthorized organization context. + * Do not implement sampled project enforcement only in middleware. + * Verification: + * dotnet test -- --filter-class OverageMiddlewareTests +- [x] 21. Update EventPostsJob for sampled/project-budget allowance + * After parsing events, ask UsageService for allowed event count. + * Process only allowed events. + * Use deterministic sampling/selection when allowed count is lower than submitted count due to smart throttling. + * Increment blocked usage for non-accepted events. + * Increment blocked usage once after all applicable controls have selected the final accepted set. + * Increment accepted usage only for processed events. + * Verification: + * dotnet test -- --filter-class EventPostsJobTests + +## Backend API tests + +- [x] 22. Add OrganizationController tests for budget alert settings + * Test enabling budget alerts with thresholds. + * Test disabling budget alerts. + * Test threshold normalization/deduplication. + * Test invalid thresholds reject. + * Test unauthorized organization update remains rejected. + * Verification: + * dotnet test -- --filter-class OrganizationControllerTests +- [x] 23. Add ProjectController tests for event budget update behavior + * Test setting fixed event budget. + * Test setting percentage event budget. + * Test clearing event budget. + * Test invalid fixed budget rejects. + * Test invalid percentage rejects. + * Test percentage budget rejects for unlimited organization or is marked inactive according to final implementation decision. + * Verification: + * dotnet test -- --filter-class ProjectControllerTests +- [x] 24. Add event submission integration tests + * Verify event submission processing accepts samples under smart throttling. + * Verify organization overage behavior remains unchanged. + * Verify uncapped projects continue under the organization limit. + * Verify crossing budget alert thresholds does not change accepted event response. + * Verification: + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests +- [x] 25. Update HTTP samples if contracts are changed + * Update tests/http/*.http samples for organization update/get if organization request/response examples exist. + * Update tests/http/*.http samples for project update/get if project request/response examples exist. + * Verification: + * Manual review of tests/http/*.http. + * dotnet build + +## Svelte UI + +- [x] 26. Add budget alerts UI to Organization Usage page + * Update src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte. + * Add card for enabling/disabling budget alerts. + * Add threshold editor with default suggestions of 50 and 80. + * Use TanStack Form, Zod, existing shadcn components, accessible labels/live errors, and explicit loading/saving states. + * Show computed event counts for each threshold. + * Disable percentage alerts for unlimited organizations. + * Use existing organization API query/mutation patterns. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA on Organization Settings → Usage. +- [x] 27. Add project event budget UI component + * Create src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte. + * Support Off, Fixed, and Percentage modes. + * Show computed effective cap. + * Show current project usage against cap when active. + * Show fixed-cap warning when fixed cap exceeds current organization allowance. + * Disable percentage mode when organization allowance is unlimited. + * Reject truncated fixed-limit decimals and enforce backend-identical fixed/percentage ranges. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint +- [x] 28. Add smart throttling UI status + * Surface project smart-throttled status on Project Usage where feasible. + * Explain that a sample of events is still accepted. + * Link to organization/project usage. + * Avoid exposing many tuning options. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA with a smart-throttled project. +- [x] 29. Integrate project card into Project Usage page + * Add the card to src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte. + * Use existing getProjectQuery, getOrganizationQuery, and updateProject mutation. + * Invalidate/update project query data after saving. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * Manual localhost QA on /project/{projectId}/usage +- [x] 30. Update project usage chart limit/status display + * Use effective_ingest_limit as the chart limit when present. + * Otherwise preserve existing organization limit behavior. + * Label the limit line as Project Limit or Organization Limit. + * Show smart throttling status text/badge when active. + * Verification: + * Manual localhost QA with: + * no project budget + * fixed project budget + * percentage project budget + * smart-throttled project + * Confirm chart remains readable and accessible. +- [x] 31. Add frontend tests for budget control helpers + * Test organization budget alert threshold normalization. + * Test organization budget alert threshold validation. + * Test budget alert computed event counts. + * Test project budget mode-to-payload conversion. + * Test project percentage cap calculation display. + * Test fixed cap warning. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run test:unit + +## Compatibility and validation + +- [x] 32. Verify existing request throttling remains unchanged + * Confirm ThrottlingMiddleware behavior and tests are unchanged unless incidental generated code changes require updates. + * Verification: + * dotnet test -- --filter-class ThrottlingMiddlewareTests +- [x] 33. Verify existing overage notification remains unchanged + * Confirm existing PlanOverage monthly/hourly paths continue to enqueue and send existing organization notices. + * Confirm budget alerts and smart throttling use separate message/work item/template. + * Verification: + * Existing organization notification tests if present. + * New budget alert and smart throttling work item tests. +- [x] 34. Verify no Elasticsearch mapping/reindex is required + * Confirm budget alert settings and project event budget are not used for organization/project search/filter/sort. + * Confirm no organization/project index version bump is required. + * Verification: + * Code review checklist item. +- [x] 35. Local dogfood budget alerts + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Enable budget alerts for a low threshold. + * Submit events until threshold is crossed. + * Confirm budget alert work item/email is queued. + * Confirm alert does not send twice for same threshold in same monthly period. + * Confirm disabling alerts prevents future sends. + * Verification: + * Manual QA notes in PR. +- [x] 36. Local dogfood project budgets and smart throttling + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Configure a low fixed budget on a project. + * Submit events until budget is reached. + * Confirm: + * project is limited/sampled + * blocked usage increases + * another project can still ingest events + * clearing the budget restores behavior + * Create or simulate a noisy project. + * Confirm smart throttling applies without manual configuration. + * Confirm a sample is still accepted. + * Verification: + * Manual QA notes in PR. + +## Final validation + +- [x] 37. Run targeted validation + * Keep budget-specific tests out of UsageServiceTests.cs so the existing file does not grow past 1,000 lines. + * Commands: + * dotnet build + * dotnet test -- --filter-class UsageServiceTests + * dotnet test -- --filter-class OverageMiddlewareTests + * dotnet test -- --filter-class OrganizationControllerTests + * dotnet test -- --filter-class ProjectControllerTests + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class ThrottlingMiddlewareTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * cd src/Exceptionless.Web/ClientApp && npm run test:unit +- [x] 38. Run OpenSpec validation + * Command: + * openspec validate add-usage-budget-controls --strict --no-interactive +- [x] 39. Run full validation before merge + * Commands: + * dotnet test + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check && npm run lint && npm run test:unit && npm run build + * openspec validate --all --strict --no-interactive + * git diff --check + * Re-run the thermo-nuclear review against `origin/main...HEAD` and require no structural, hot-path, file-size, or generated-artifact drift blockers. diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index 9cfb19de39..08e966f5f1 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -89,6 +89,8 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); diff --git a/src/Exceptionless.Core/Configuration/AppOptions.cs b/src/Exceptionless.Core/Configuration/AppOptions.cs index ff6217829a..e6f47712c7 100644 --- a/src/Exceptionless.Core/Configuration/AppOptions.cs +++ b/src/Exceptionless.Core/Configuration/AppOptions.cs @@ -43,6 +43,11 @@ public class AppOptions public bool EventSubmissionDisabled { get; internal set; } + /// + /// Operational kill switch for automatic project-level sampling. Explicit project limits are unaffected. + /// + public bool EnableSmartProjectThrottling { get; internal set; } + internal List DisabledPipelineActions { get; set; } = null!; internal List DisabledPlugins { get; set; } = null!; @@ -99,6 +104,7 @@ public static AppOptions ReadFromConfiguration(IConfiguration config) options.EnableArchive = config.GetValue(nameof(options.EnableArchive), false); options.EnableSampleData = config.GetValue(nameof(options.EnableSampleData), options.AppMode == AppMode.Development); options.EventSubmissionDisabled = config.GetValue(nameof(options.EventSubmissionDisabled), false); + options.EnableSmartProjectThrottling = config.GetValue(nameof(options.EnableSmartProjectThrottling), true); options.DisabledPipelineActions = config.GetValueList(nameof(options.DisabledPipelineActions)); options.DisabledPlugins = config.GetValueList(nameof(options.DisabledPlugins)); options.MaximumEventPostSize = config.GetValue(nameof(options.MaximumEventPostSize), 200000).NormalizeValue(); diff --git a/src/Exceptionless.Core/Exceptionless.Core.csproj b/src/Exceptionless.Core/Exceptionless.Core.csproj index 64605e0c70..5e461edabd 100644 --- a/src/Exceptionless.Core/Exceptionless.Core.csproj +++ b/src/Exceptionless.Core/Exceptionless.Core.csproj @@ -3,10 +3,12 @@ + + @@ -14,10 +16,12 @@ + + diff --git a/src/Exceptionless.Core/Jobs/EventPostsJob.cs b/src/Exceptionless.Core/Jobs/EventPostsJob.cs index 0b7aaffec3..6c75893bc3 100644 --- a/src/Exceptionless.Core/Jobs/EventPostsJob.cs +++ b/src/Exceptionless.Core/Jobs/EventPostsJob.cs @@ -162,29 +162,46 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex return JobResult.Success; } - // Don't process all the events if it will put the account over its limits. - int eventsToProcess = await _usageService.GetEventsLeftAsync(organization.Id); - if (eventsToProcess < 1) + int submittedEventCount = events.Count; + bool isSingleEvent = submittedEventCount == 1; + var ingestAllowance = await _usageService.GetEventIngestAllowanceAsync(organization, project); + var candidates = events + .Select((persistentEvent, index) => new EventCandidate(persistentEvent, index, GetStableEventHash(entry.Id, persistentEvent, index))) + .ToList(); + + int smartThrottleBlocked = 0; + if (ingestAllowance.SmartThrottle.IsThrottled) { - if (!isInternalProject) - _logger.LogDebug("Unable to process EventPost {FilePath}: Over plan limits", payloadPath); - - await _usageService.IncrementBlockedAsync(organization.Id, project.Id, events.Count); + int sampleThreshold = (int)(ingestAllowance.SmartThrottle.SampleRate * 10_000); + candidates = candidates.Where(candidate => IsHashSelected(candidate.Hash, sampleThreshold)).ToList(); + smartThrottleBlocked = submittedEventCount - candidates.Count; + _usageService.RecordSmartThrottle(smartThrottleBlocked); + } - await CompleteEntryAsync(entry, ep, _timeProvider.GetUtcNow().UtcDateTime); - return JobResult.Success; + if (candidates.Count > ingestAllowance.EventsLeft) + { + candidates = candidates + .OrderBy(candidate => candidate.Hash) + .Take(ingestAllowance.EventsLeft) + .ToList(); } - // Keep track of the original event payload size, we can save some processing for retries in the case it was a massive batch. - bool isSingleEvent = events.Count == 1; + events = candidates + .OrderBy(candidate => candidate.Index) + .Select(candidate => candidate.Event) + .ToList(); - // Discard any events over the plan limit. - if (eventsToProcess < events.Count) + int blockedEventCount = submittedEventCount - events.Count; + if (blockedEventCount > 0) + await _usageService.IncrementBlockedAsync(organization.Id, project.Id, blockedEventCount); + + if (events.Count == 0) { - int discarded = events.Count - eventsToProcess; - events = events.Take(eventsToProcess).ToList(); + if (!isInternalProject) + _logger.LogDebug("Unable to process EventPost {FilePath}: accepted 0/{SubmittedEventCount} events after usage budget evaluation", payloadPath, submittedEventCount); - await _usageService.IncrementBlockedAsync(organization.Id, project.Id, discarded); + await CompleteEntryAsync(entry, ep, _timeProvider.GetUtcNow().UtcDateTime); + return JobResult.Success; } int errorCount = 0; @@ -200,7 +217,7 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex // increment the plan usage counters (note: OverageHandler already incremented usage by 1) int processedEvents = contexts.Count(c => c.IsProcessed); - await _usageService.IncrementTotalAsync(organization.Id, project.Id, processedEvents); + await _usageService.IncrementTotalAsync(organization, project.Id, processedEvents); int discardedEvents = contexts.Count(c => c.IsDiscarded); await _usageService.IncrementDiscardedAsync(organization.Id, project.Id, discardedEvents); @@ -250,6 +267,34 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex return JobResult.Success; } + private static ulong GetStableEventHash(string queueEntryId, PersistentEvent persistentEvent, int index) + { + string value = $"{queueEntryId}:{index}:{persistentEvent.ReferenceId}"; + const ulong offsetBasis = 14695981039346656037; + const ulong prime = 1099511628211; + ulong hash = offsetBasis; + foreach (char character in value) + { + hash ^= character; + hash *= prime; + } + + return hash; + } + + internal static bool IsSelectedForSmartThrottle(string queueEntryId, PersistentEvent persistentEvent, int index, double sampleRate = SmartThrottleResult.DefaultSampleRate) + { + int sampleThreshold = (int)(sampleRate * 10_000); + return IsHashSelected(GetStableEventHash(queueEntryId, persistentEvent, index), sampleThreshold); + } + + private static bool IsHashSelected(ulong hash, int sampleThreshold) + { + return hash % 10_000 < (ulong)sampleThreshold; + } + + private sealed record EventCandidate(PersistentEvent Event, int Index, ulong Hash); + private List? ParseEventPost(EventPostInfo ep, DateTime createdUtc, byte[] uncompressedData, string queueEntryId, bool isInternalProject) { using (_logger.BeginScope(new ExceptionlessState().Tag("parsing"))) diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs new file mode 100644 index 0000000000..e86148ae55 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs @@ -0,0 +1,117 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueOrganizationBudgetAlertOnUsageThreshold : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly ILogger _logger; + + public EnqueueOrganizationBudgetAlertOnUsageThreshold(IQueue workItemQueue, IMessageSubscriber subscriber, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async alert => + { + _logger.LogInformation("Enqueueing budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", alert.OrganizationId, alert.Threshold); + + await _workItemQueue.EnqueueAsync(new OrganizationBudgetAlertWorkItem + { + OrganizationId = alert.OrganizationId, + Threshold = alert.Threshold, + ThresholdEventCount = alert.ThresholdEventCount, + CurrentEventCount = alert.CurrentEventCount, + EventLimit = alert.EventLimit + }); + }, shutdownToken); + } +} + +public class OrganizationBudgetAlertWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + private readonly UsageService _usageService; + + public OrganizationBudgetAlertWorkItemHandler(IOrganizationRepository organizationRepository, IUserRepository userRepository, IMailer mailer, UsageService usageService, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _userRepository = userRepository; + _mailer = mailer; + _usageService = usageService; + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", wi.OrganizationId, wi.Threshold); + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping budget alert", wi.OrganizationId); + return; + } + + // Re-check that budget alerts are still enabled and the threshold is still configured + if (organization.BudgetAlertSettings is not { Enabled: true, Thresholds: not null } || !organization.BudgetAlertSettings.Thresholds.Contains(wi.Threshold)) + { + Log.LogInformation("Budget alerts disabled or threshold {Threshold}% removed for organization: {OrganizationId}, skipping", wi.Threshold, wi.OrganizationId); + return; + } + + int eventLimit = await _usageService.GetMaxEventsPerMonthAsync(organization.Id); + if (eventLimit <= 0) + { + Log.LogInformation("Organization {OrganizationId} no longer has a finite event allowance, skipping budget alert", wi.OrganizationId); + return; + } + + int thresholdEventCount = (int)Math.Ceiling(eventLimit * wi.Threshold / 100d); + int currentEventCount = (await _usageService.GetUsageAsync(organization.Id)).CurrentUsage.Total; + if (currentEventCount < thresholdEventCount) + { + Log.LogInformation("Organization {OrganizationId} usage is now below budget threshold {Threshold}%, skipping", wi.OrganizationId, wi.Threshold); + return; + } + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + Log.LogTrace("Sending budget alert email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendOrganizationBudgetAlertAsync(user, organization, wi.Threshold, thresholdEventCount, currentEventCount, eventLimit); + } + + Log.LogTrace("Done sending budget alert emails"); + } +} diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs new file mode 100644 index 0000000000..232a246a07 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs @@ -0,0 +1,122 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueProjectSmartThrottleOnThrottleApplied : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly ILogger _logger; + + public EnqueueProjectSmartThrottleOnThrottleApplied(IQueue workItemQueue, IMessageSubscriber subscriber, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async throttle => + { + _logger.LogInformation("Enqueueing smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", throttle.ProjectId, throttle.OrganizationId); + + await _workItemQueue.EnqueueAsync(new ProjectSmartThrottleWorkItem + { + OrganizationId = throttle.OrganizationId, + ProjectId = throttle.ProjectId, + SampleRate = throttle.SampleRate, + CurrentEventCount = throttle.CurrentEventCount, + EventLimit = throttle.EventLimit + }); + }, shutdownToken); + } +} + +public class ProjectSmartThrottleWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IProjectRepository _projectRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + private readonly UsageService _usageService; + + public ProjectSmartThrottleWorkItemHandler(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, IUserRepository userRepository, IMailer mailer, UsageService usageService, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _projectRepository = projectRepository; + _userRepository = userRepository; + _mailer = mailer; + _usageService = usageService; + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", wi.ProjectId, wi.OrganizationId); + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping smart throttle notification", wi.OrganizationId); + return; + } + + var project = await _projectRepository.GetByIdAsync(wi.ProjectId, o => o.Cache()); + if (project is null) + { + Log.LogWarning("Project {ProjectId} not found, skipping smart throttle notification", wi.ProjectId); + return; + } + + if (!String.Equals(project.OrganizationId, organization.Id, StringComparison.Ordinal) || + !await _usageService.IsProjectSmartThrottledAsync(organization.Id, project.Id)) + { + Log.LogInformation("Project {ProjectId} is no longer smart throttled, skipping notification", wi.ProjectId); + return; + } + + int organizationLimit = await _usageService.GetMaxEventsPerMonthAsync(organization.Id); + if (organizationLimit <= 0) + { + Log.LogInformation("Organization {OrganizationId} no longer has a finite event allowance, skipping smart throttle notification", wi.OrganizationId); + return; + } + + int projectCount = (int)Math.Max(1, (await _projectRepository.GetCountByOrganizationIdAsync(organization.Id)).Total); + int fairShareLimit = organizationLimit / projectCount; + int currentProjectUsage = (await _usageService.GetUsageAsync(organization.Id, project.Id)).CurrentUsage.Total; + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + Log.LogTrace("Sending smart throttle email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendProjectThrottledNoticeAsync(user, organization, project, wi.SampleRate, currentProjectUsage, fairShareLimit); + } + + Log.LogTrace("Done sending smart throttle emails"); + } +} diff --git a/src/Exceptionless.Core/Mail/IMailer.cs b/src/Exceptionless.Core/Mail/IMailer.cs index 6e8905cab5..7a121864ed 100644 --- a/src/Exceptionless.Core/Mail/IMailer.cs +++ b/src/Exceptionless.Core/Mail/IMailer.cs @@ -11,6 +11,8 @@ public interface IMailer Task SendOrganizationNoticeAsync(User user, Organization organization, bool isOverMonthlyLimit, bool isOverHourlyLimit); Task SendOrganizationPaymentFailedAsync(User owner, Organization organization); Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan); + Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit); + Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit); Task SendUserEmailVerifyAsync(User user); Task SendUserPasswordResetAsync(User user); } diff --git a/src/Exceptionless.Core/Mail/Mailer.cs b/src/Exceptionless.Core/Mail/Mailer.cs index 16394cca4d..c24e739983 100644 --- a/src/Exceptionless.Core/Mail/Mailer.cs +++ b/src/Exceptionless.Core/Mail/Mailer.cs @@ -231,6 +231,57 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza }, template); } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + const string template = "organization-budget-alert"; + string subject = $"[{organization.Name}] Budget Alert: {threshold}% of monthly event allowance used"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "Threshold", threshold }, + { "ThresholdEventCount", thresholdEventCount }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit }, + { "RemainingEventCount", Math.Max(0, eventLimit - currentEventCount) } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + const string template = "project-smart-throttle"; + string subject = $"[{organization.Name}] Smart Throttling Active: {project.Name}"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "ProjectId", project.Id }, + { "ProjectName", project.Name }, + { "SampleRate", sampleRate }, + { "SamplePercent", (int)(sampleRate * 100) }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { const string template = "project-daily-summary"; diff --git a/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html new file mode 100644 index 0000000000..c85c5cdc31 --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

{{OrganizationName}} has reached {{Threshold}}% of its monthly event allowance.

The organization has accepted {{CurrentEventCount}} events. The {{Threshold}}% alert is {{ThresholdEventCount}} events, the current allowance is {{EventLimit}}, and {{RemainingEventCount}} events remain.

View Usage
                                                           
\ No newline at end of file diff --git a/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html new file mode 100644 index 0000000000..a8891b00dd --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

Smart throttling is active for {{ProjectName}} in {{OrganizationName}}.

This project has accepted {{CurrentEventCount}} events this month, above its current fair-share limit of {{EventLimit}} events. To protect the organization's remaining allowance, Exceptionless is accepting a stable {{SamplePercent}}% sample from this project. Events outside that sample are counted as blocked.

Other projects are unaffected unless they independently meet the same smart-throttling criteria.

View Project Usage
                                                           
\ No newline at end of file diff --git a/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs new file mode 100644 index 0000000000..1d6bf4219b --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs @@ -0,0 +1,10 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs new file mode 100644 index 0000000000..b3e9ca7c2c --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs @@ -0,0 +1,10 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Organization.cs b/src/Exceptionless.Core/Models/Organization.cs index 0a92a03660..0c13e6ccf7 100644 --- a/src/Exceptionless.Core/Models/Organization.cs +++ b/src/Exceptionless.Core/Models/Organization.cs @@ -174,6 +174,11 @@ public Organization() /// public DataDictionary? Data { get; set; } + /// + /// Budget alert settings for this organization. Null means disabled. + /// + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } + public DateTime CreatedUtc { get; set; } public DateTime UpdatedUtc { get; set; } public bool IsDeleted { get; set; } @@ -268,6 +273,19 @@ public IEnumerable Validate(ValidationContext validationContex [nameof(SuspendedByUserId)]); } } + + if (BudgetAlertSettings is not null) + { + if (BudgetAlertSettings.Enabled && BudgetAlertSettings.Thresholds is not { Count: > 0 }) + { + yield return new ValidationResult("At least one threshold is required when budget alerts are enabled.", + [nameof(BudgetAlertSettings)]); + } + + if (BudgetAlertSettings.Thresholds?.Any(t => t <= 0 || t >= 100) is true) + yield return new ValidationResult("Budget alert thresholds must be between 1 and 99.", + [nameof(BudgetAlertSettings)]); + } } } @@ -279,5 +297,3 @@ public enum BillingStatus Canceled = 3, Unpaid = 4 } - - diff --git a/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs new file mode 100644 index 0000000000..e8d55eccb2 --- /dev/null +++ b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs @@ -0,0 +1,12 @@ +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} diff --git a/src/Exceptionless.Core/Models/Project.cs b/src/Exceptionless.Core/Models/Project.cs index 4ac783e893..c7f86d32fd 100644 --- a/src/Exceptionless.Core/Models/Project.cs +++ b/src/Exceptionless.Core/Models/Project.cs @@ -63,6 +63,11 @@ public Project() public bool DeleteBotDataEnabled { get; set; } + /// + /// Optional project-level event budget configuration. Null means no project-specific cap. + /// + public ProjectIngestLimit? IngestLimit { get; set; } + /// /// The tick count that represents the next time the daily summary job should run. This time is set to midnight of the /// projects local time. @@ -90,5 +95,20 @@ public IEnumerable Validate(ValidationContext validationContex yield return new ValidationResult("Please specify a valid next summary end of day ticks.", [nameof(NextSummaryEndOfDayTicks)]); } + + if (IngestLimit is null) + yield break; + + if (!Enum.IsDefined(IngestLimit.Type)) + { + yield return new ValidationResult("Please specify a valid project ingest limit type.", [nameof(IngestLimit)]); + yield break; + } + + if (IngestLimit.Type is ProjectIngestLimitType.Fixed && IngestLimit.FixedLimit is null or <= 0) + yield return new ValidationResult("The fixed project ingest limit must be greater than 0.", [nameof(IngestLimit)]); + + if (IngestLimit.Type is ProjectIngestLimitType.PercentOfOrganizationLimit && IngestLimit.PercentOfOrganizationLimit is null or <= 0 or > 100) + yield return new ValidationResult("The project ingest percentage must be greater than 0 and no more than 100.", [nameof(IngestLimit)]); } } diff --git a/src/Exceptionless.Core/Models/ProjectIngestLimit.cs b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs new file mode 100644 index 0000000000..67704e9de6 --- /dev/null +++ b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs @@ -0,0 +1,14 @@ +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} diff --git a/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs new file mode 100644 index 0000000000..6a11ee2b45 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs @@ -0,0 +1,14 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + + public string UniqueIdentifier => $"BudgetAlert:{OrganizationId}:{Threshold}"; +} diff --git a/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs new file mode 100644 index 0000000000..d9a6014603 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs @@ -0,0 +1,14 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record ProjectSmartThrottleWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + + public string UniqueIdentifier => $"SmartThrottle:{OrganizationId}:{ProjectId}"; +} diff --git a/src/Exceptionless.Core/Services/UsageService.Budgets.cs b/src/Exceptionless.Core/Services/UsageService.Budgets.cs new file mode 100644 index 0000000000..fe8072593d --- /dev/null +++ b/src/Exceptionless.Core/Services/UsageService.Budgets.cs @@ -0,0 +1,378 @@ +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.DateTimeExtensions; +using Foundatio.Caching; +using Foundatio.Messaging; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services; + +public partial class UsageService +{ + /// + /// Calculates the complete organization, explicit project, and automatic smart-throttle allowance + /// from the models already loaded by the ingestion job. + /// + public async Task GetEventIngestAllowanceAsync(Organization organization, Project project) + { + ArgumentNullException.ThrowIfNull(organization); + ArgumentNullException.ThrowIfNull(project); + if (!String.Equals(project.OrganizationId, organization.Id, StringComparison.Ordinal)) + throw new ArgumentException("The project does not belong to the organization.", nameof(project)); + + int maxEventsPerMonth = organization.GetMaxEventsPerMonthWithBonus(_timeProvider); + int effectiveProjectLimit = GetEffectiveProjectLimit(project, maxEventsPerMonth); + + // This is the common self-hosted/unlimited path: no cache or repository reads are needed. + if (maxEventsPerMonth < 0 && effectiveProjectLimit < 0) + return EventIngestAllowance.Unlimited; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + var totals = await GetAcceptedUsageTotalsAsync(utcNow, organization, project); + + int organizationEventsLeft = GetOrganizationEventsLeft(maxEventsPerMonth, totals.OrganizationTotal, totals.OrganizationCurrentBucket); + if (organizationEventsLeft <= 0) + { + return new EventIngestAllowance + { + EventsLeft = 0, + IsOverOrgLimit = true, + EffectiveProjectLimit = effectiveProjectLimit + }; + } + + int projectEventsLeft = effectiveProjectLimit < 0 + ? Int32.MaxValue + : Math.Max(0, effectiveProjectLimit - totals.ProjectTotal); + if (projectEventsLeft <= 0) + { + return new EventIngestAllowance + { + EventsLeft = 0, + IsOverProjectLimit = true, + EffectiveProjectLimit = effectiveProjectLimit + }; + } + + var smartThrottle = await GetSmartThrottleResultAsync(utcNow, organization, project, maxEventsPerMonth, totals); + return new EventIngestAllowance + { + EventsLeft = Math.Min(organizationEventsLeft, projectEventsLeft), + EffectiveProjectLimit = effectiveProjectLimit, + SmartThrottle = smartThrottle + }; + } + + public async Task GetEventIngestAllowanceAsync(string organizationId, string projectId) + { + var organizationTask = _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + var projectTask = _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + await Task.WhenAll(organizationTask, projectTask); + + var organization = await organizationTask ?? throw new UsageServiceException($"Organization '{organizationId}' not found."); + var project = await projectTask ?? throw new UsageServiceException($"Project '{projectId}' not found."); + return await GetEventIngestAllowanceAsync(organization, project); + } + + public async Task GetSmartThrottleRateAsync(string organizationId, string projectId) + { + var organizationTask = _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + var projectTask = _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + await Task.WhenAll(organizationTask, projectTask); + + var organization = await organizationTask; + var project = await projectTask; + if (organization is null || project is null) + return SmartThrottleResult.NoThrottle; + + return (await GetEventIngestAllowanceAsync(organization, project)).SmartThrottle; + } + + public int GetEffectiveProjectLimit(Project project, Organization organization) + { + ArgumentNullException.ThrowIfNull(project); + ArgumentNullException.ThrowIfNull(organization); + return GetEffectiveProjectLimit(project.IngestLimit, organization.GetMaxEventsPerMonthWithBonus(_timeProvider)); + } + + public int GetEffectiveProjectLimit(ProjectIngestLimit? ingestLimit, Organization organization) + { + ArgumentNullException.ThrowIfNull(organization); + return GetEffectiveProjectLimit(ingestLimit, organization.GetMaxEventsPerMonthWithBonus(_timeProvider)); + } + + private static int GetEffectiveProjectLimit(Project project, int maxEventsPerMonth) + { + return GetEffectiveProjectLimit(project.IngestLimit, maxEventsPerMonth); + } + + private static int GetEffectiveProjectLimit(ProjectIngestLimit? ingestLimit, int maxEventsPerMonth) + { + if (ingestLimit is null) + return -1; + + if (ingestLimit.Type is ProjectIngestLimitType.Fixed) + { + if (ingestLimit.FixedLimit is not > 0) + return -1; + + return maxEventsPerMonth < 0 + ? ingestLimit.FixedLimit.Value + : Math.Min(ingestLimit.FixedLimit.Value, maxEventsPerMonth); + } + + if (ingestLimit.Type is not ProjectIngestLimitType.PercentOfOrganizationLimit || + ingestLimit.PercentOfOrganizationLimit is not > 0 or > 100 || + maxEventsPerMonth < 0) + return -1; + + return Math.Min(maxEventsPerMonth, (int)Math.Ceiling(maxEventsPerMonth * (double)ingestLimit.PercentOfOrganizationLimit.Value / 100)); + } + + private async Task GetAcceptedUsageTotalsAsync(DateTime utcNow, Organization organization, Project project) + { + string organizationTotalKey = GetTotalCacheKey(utcNow, organization.Id); + string projectTotalKey = GetTotalCacheKey(utcNow, organization.Id, project.Id); + string organizationCurrentKey = GetBucketTotalCacheKey(utcNow, organization.Id); + string projectCurrentKey = GetBucketTotalCacheKey(utcNow, organization.Id, project.Id); + string organizationPreviousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id); + string projectPreviousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id, project.Id); + + var values = await _cache.GetAllAsync([ + organizationTotalKey, + projectTotalKey, + organizationCurrentKey, + projectCurrentKey, + organizationPreviousKey, + projectPreviousKey + ]); + + int organizationCurrent = GetCachedValue(values, organizationCurrentKey); + int projectCurrent = GetCachedValue(values, projectCurrentKey); + int organizationTotal = GetCachedValue(values, organizationTotalKey, organization.GetCurrentUsage(_timeProvider).Total) + + organizationCurrent + + GetCachedValue(values, organizationPreviousKey); + int projectTotal = GetCachedValue(values, projectTotalKey, project.GetCurrentUsage(_timeProvider).Total) + + projectCurrent + + GetCachedValue(values, projectPreviousKey); + + return new AcceptedUsageTotals(organizationTotal, projectTotal, organizationCurrent, projectCurrent); + } + + private static int GetCachedValue(IDictionary> values, string key, int defaultValue = 0) + { + return values.TryGetValue(key, out var value) && value.HasValue ? value.Value : defaultValue; + } + + private int GetOrganizationEventsLeft(int maxEventsPerMonth, int currentTotal, int currentBucketTotal) + { + if (maxEventsPerMonth < 0) + return Int32.MaxValue; + + int monthlyEventsLeft = Math.Max(0, maxEventsPerMonth - currentTotal); + int bucketEventsLeft = Math.Max(0, GetBucketEventLimit(maxEventsPerMonth) - currentBucketTotal); + return Math.Min(monthlyEventsLeft, bucketEventsLeft); + } + + private async Task GetSmartThrottleResultAsync(DateTime utcNow, Organization organization, Project project, int maxEventsPerMonth, AcceptedUsageTotals totals) + { + if (!_appOptions.EnableSmartProjectThrottling || maxEventsPerMonth <= 0 || totals.OrganizationCurrentBucket <= 0 || totals.ProjectCurrentBucket <= 0) + return SmartThrottleResult.NoThrottle; + + int remainingAllowance = Math.Max(0, maxEventsPerMonth - totals.OrganizationTotal); + if (remainingAllowance <= 0) + return SmartThrottleResult.NoThrottle; + + double windowsLeft = Math.Max(1, Math.Ceiling((utcNow.EndOfMonth() - utcNow).TotalMinutes / _bucketSize.TotalMinutes)); + double sustainableWindowAllowance = remainingAllowance / windowsLeft * 10; + if (totals.OrganizationCurrentBucket < sustainableWindowAllowance * 0.8) + return SmartThrottleResult.NoThrottle; + + int projectCount = (int)Math.Max(1, (await _projectRepository.GetCountByOrganizationIdAsync(organization.Id)).Total); + if (projectCount <= 1) + return SmartThrottleResult.NoThrottle; + + double fairShareWindowAllowance = sustainableWindowAllowance / projectCount; + double fairShareRatio = fairShareWindowAllowance > 0 ? totals.ProjectCurrentBucket / fairShareWindowAllowance : Double.PositiveInfinity; + if (fairShareRatio <= 2) + return SmartThrottleResult.NoThrottle; + + var result = new SmartThrottleResult + { + IsThrottled = true, + SampleRate = SmartThrottleResult.DefaultSampleRate, + ProjectShare = (double)totals.ProjectTotal / Math.Max(1, totals.OrganizationTotal), + FairShareRatio = fairShareRatio, + CurrentProjectUsage = totals.ProjectTotal, + FairShareLimit = maxEventsPerMonth / projectCount + }; + + await ActivateSmartThrottleAsync(utcNow, organization, project, result); + return result; + } + + private async Task ActivateSmartThrottleAsync(DateTime utcNow, Organization organization, Project project, SmartThrottleResult result) + { + string stateKey = GetProjectSmartThrottleKey(utcNow, organization.Id, project.Id); + bool isNewTransition; + try + { + isNewTransition = await _cache.AddAsync(stateKey, result, _bucketSize + TimeSpan.FromMinutes(1)); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to record smart throttle state for project {ProjectId}", project.Id); + return; + } + + if (!isNewTransition) + return; + + _logger.LogInformation("Smart project throttling activated for {OrganizationId}/{ProjectId} at {SampleRate:P0}; project usage {ProjectUsage}, fair-share limit {FairShareLimit}", + organization.Id, project.Id, result.SampleRate, result.CurrentProjectUsage, result.FairShareLimit); + + string notificationKey = GetSmartThrottleNotificationKey(utcNow, organization.Id, project.Id); + bool notificationClaimed = false; + try + { + var notificationTtl = utcNow.EndOfMonth() - utcNow + TimeSpan.FromDays(1); + notificationClaimed = await _cache.AddAsync(notificationKey, true, notificationTtl); + if (!notificationClaimed) + return; + + await _messagePublisher.PublishAsync(new ProjectSmartThrottleApplied + { + OrganizationId = organization.Id, + ProjectId = project.Id, + SampleRate = result.SampleRate, + CurrentEventCount = result.CurrentProjectUsage, + EventLimit = result.FairShareLimit + }); + } + catch (Exception ex) + { + if (notificationClaimed) + { + try + { + await _cache.RemoveAsync(notificationKey); + } + catch (Exception cleanupException) + { + _logger.LogWarning(cleanupException, "Failed to release smart throttle notification claim for project {ProjectId}", project.Id); + } + } + + _logger.LogError(ex, "Failed to publish smart throttle notification for project {ProjectId}", project.Id); + } + } + + public async Task> GetProjectSmartThrottleStatesAsync(IReadOnlyCollection<(string OrganizationId, string ProjectId)> projects) + { + if (!_appOptions.EnableSmartProjectThrottling || projects.Count == 0) + return new Dictionary(); + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + var keysByProjectId = projects.ToDictionary( + project => project.ProjectId, + project => GetProjectSmartThrottleKey(utcNow, project.OrganizationId, project.ProjectId)); + var values = await _cache.GetAllAsync(keysByProjectId.Values); + + return keysByProjectId + .Where(pair => values.TryGetValue(pair.Value, out var value) && value.HasValue && value.Value.IsThrottled) + .ToDictionary(pair => pair.Key, pair => values[pair.Value].Value); + } + + public async Task IsProjectSmartThrottledAsync(string organizationId, string projectId) + { + if (!_appOptions.EnableSmartProjectThrottling) + return false; + + var state = await _cache.GetAsync(GetProjectSmartThrottleKey(_timeProvider.GetUtcNow().UtcDateTime, organizationId, projectId)); + return state is { HasValue: true, Value.IsThrottled: true }; + } + + public void RecordSmartThrottle(int blockedCount) + { + if (blockedCount > 0) + AppDiagnostics.EventsSmartThrottled.Add(blockedCount); + } + + public async Task EvaluateBudgetAlertsAfterSettingsChangeAsync(Organization organization, IReadOnlyCollection thresholds) + { + if (organization.BudgetAlertSettings is not { Enabled: true } || thresholds.Count == 0) + return; + + int maxEventsPerMonth = organization.GetMaxEventsPerMonthWithBonus(_timeProvider); + if (maxEventsPerMonth <= 0) + return; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + string totalKey = GetTotalCacheKey(utcNow, organization.Id); + string currentKey = GetBucketTotalCacheKey(utcNow, organization.Id); + string previousKey = GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organization.Id); + var values = await _cache.GetAllAsync([totalKey, currentKey, previousKey]); + int currentTotal = GetCachedValue(values, totalKey, organization.GetCurrentUsage(_timeProvider).Total) + + GetCachedValue(values, currentKey) + + GetCachedValue(values, previousKey); + + await PublishCrossedBudgetAlertsAsync(organization, 0, currentTotal, maxEventsPerMonth, thresholds); + } + + private async Task PublishCrossedBudgetAlertsAsync(Organization organization, int previousTotal, int currentTotal, int maxEventsPerMonth, IReadOnlyCollection? thresholds = null) + { + if (organization.BudgetAlertSettings is not { Enabled: true, Thresholds: not null } || maxEventsPerMonth <= 0) + return; + + thresholds ??= organization.BudgetAlertSettings.Thresholds; + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + foreach (int threshold in thresholds.Order()) + { + if (threshold is <= 0 or >= 100) + continue; + + int thresholdEventCount = (int)Math.Ceiling((double)threshold / 100 * maxEventsPerMonth); + if (previousTotal >= thresholdEventCount || currentTotal < thresholdEventCount) + continue; + + string alertSentKey = GetBudgetAlertSentKey(utcNow, organization.Id, threshold); + bool alertClaimed = false; + try + { + var alertTtl = utcNow.EndOfMonth() - utcNow + TimeSpan.FromDays(1); + alertClaimed = await _cache.AddAsync(alertSentKey, true, alertTtl); + if (!alertClaimed) + continue; + + await _messagePublisher.PublishAsync(new OrganizationBudgetAlert + { + OrganizationId = organization.Id, + Threshold = threshold, + ThresholdEventCount = thresholdEventCount, + CurrentEventCount = currentTotal, + EventLimit = maxEventsPerMonth + }); + } + catch (Exception ex) + { + if (alertClaimed) + { + try + { + await _cache.RemoveAsync(alertSentKey); + } + catch (Exception cleanupException) + { + _logger.LogWarning(cleanupException, "Failed to release budget alert {Threshold}% claim for organization {OrganizationId}", threshold, organization.Id); + } + } + + _logger.LogError(ex, "Failed to publish budget alert {Threshold}% for organization {OrganizationId}", threshold, organization.Id); + } + } + } + + private sealed record AcceptedUsageTotals(int OrganizationTotal, int ProjectTotal, int OrganizationCurrentBucket, int ProjectCurrentBucket); +} diff --git a/src/Exceptionless.Core/Services/UsageService.cs b/src/Exceptionless.Core/Services/UsageService.cs index 4a12491675..2c7484134d 100644 --- a/src/Exceptionless.Core/Services/UsageService.cs +++ b/src/Exceptionless.Core/Services/UsageService.cs @@ -1,4 +1,5 @@ -using Exceptionless.Core.Extensions; +using Exceptionless.Core; +using Exceptionless.Core.Extensions; using Exceptionless.Core.Messaging.Models; using Exceptionless.Core.Models; using Exceptionless.Core.Repositories; @@ -10,19 +11,21 @@ namespace Exceptionless.Core.Services; -public class UsageService +public partial class UsageService { private readonly IOrganizationRepository _organizationRepository; private readonly IProjectRepository _projectRepository; private readonly ICacheClient _cache; private readonly IMessagePublisher _messagePublisher; private readonly NotificationService _notificationService; + private readonly AppOptions _appOptions; private readonly TimeProvider _timeProvider; private readonly ILogger _logger; private readonly TimeSpan _bucketSize = TimeSpan.FromMinutes(5); public UsageService(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, ICacheClient cache, IMessagePublisher messagePublisher, NotificationService notificationService, + AppOptions appOptions, TimeProvider timeProvider, ILoggerFactory loggerFactory) { @@ -31,6 +34,7 @@ public UsageService(IOrganizationRepository organizationRepository, IProjectRepo _cache = cache; _messagePublisher = messagePublisher; _notificationService = notificationService; + _appOptions = appOptions; _timeProvider = timeProvider; _logger = loggerFactory.CreateLogger(); } @@ -422,38 +426,63 @@ public async Task GetEventsLeftAsync(string organizationId) return Math.Max(eventsLeftInBucket, 0); } - public async Task IncrementTotalAsync(string organizationId, string projectId, int eventCount = 1) + public Task IncrementTotalAsync(string organizationId, string projectId, int eventCount = 1) + { + return IncrementTotalAsync(null, organizationId, projectId, eventCount); + } + + public Task IncrementTotalAsync(Organization organization, string projectId, int eventCount = 1) + { + ArgumentNullException.ThrowIfNull(organization); + return IncrementTotalAsync(organization, organization.Id, projectId, eventCount); + } + + private async Task IncrementTotalAsync(Organization? organization, string organizationId, string projectId, int eventCount) { if (eventCount <= 0) return; var utcNow = _timeProvider.GetUtcNow().UtcDateTime; - long bucketTotal = await _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId), eventCount, TimeSpan.FromHours(8)); - await _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId, projectId), eventCount, TimeSpan.FromHours(8)); - await _cache.ListAddAsync(GetOrganizationSetKey(utcNow), organizationId, TimeSpan.FromHours(8)); - await _cache.ListAddAsync(GetProjectSetKey(utcNow), projectId, TimeSpan.FromHours(8)); + await Task.WhenAll( + _cache.IncrementAsync(GetBucketTotalCacheKey(utcNow, organizationId, projectId), eventCount, TimeSpan.FromHours(8)), + _cache.ListAddAsync(GetOrganizationSetKey(utcNow), organizationId, TimeSpan.FromHours(8)), + _cache.ListAddAsync(GetProjectSetKey(utcNow), projectId, TimeSpan.FromHours(8)) + ); - int maxEventsPerMonth = await GetMaxEventsPerMonthAsync(organizationId); + int maxEventsPerMonth = organization?.GetMaxEventsPerMonthWithBonus(_timeProvider) ?? await GetMaxEventsPerMonthAsync(organizationId); int bucketLimit = GetBucketEventLimit(maxEventsPerMonth); - var currentTotalCache = await _cache.GetAsync(GetTotalCacheKey(utcNow, organizationId)); - if (currentTotalCache.HasValue) + if (maxEventsPerMonth > 0) { - long monthTotal = currentTotalCache.Value + bucketTotal; - if (monthTotal >= maxEventsPerMonth && monthTotal - maxEventsPerMonth < eventCount) + organization ??= await _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + int persistedTotal = organization?.GetCurrentUsage(_timeProvider).Total ?? 0; + var totals = await _cache.GetAllAsync([ + GetTotalCacheKey(utcNow, organizationId), + GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organizationId) + ]); + + int baseTotal = GetCachedValue(totals, GetTotalCacheKey(utcNow, organizationId), persistedTotal); + int previousBucketTotal = GetCachedValue(totals, GetBucketTotalCacheKey(utcNow.Subtract(_bucketSize), organizationId)); + int currentTotal = checked(baseTotal + previousBucketTotal + (int)bucketTotal); + int previousTotal = Math.Max(0, currentTotal - eventCount); + + if (previousTotal < maxEventsPerMonth && currentTotal >= maxEventsPerMonth) await _messagePublisher.PublishAsync(new PlanOverage { OrganizationId = organizationId }); + + if (organization is not null) + await PublishCrossedBudgetAlertsAsync(organization, previousTotal, currentTotal, maxEventsPerMonth); } - if (bucketTotal >= bucketLimit && bucketTotal - bucketLimit < eventCount) + if (bucketLimit >= 0 && bucketTotal >= bucketLimit && bucketTotal - bucketLimit < eventCount) { - // org will be throttled during the current bucket of time await _messagePublisher.PublishAsync(new PlanOverage { OrganizationId = organizationId, IsHourly = true }); - await _cache.SetAsync(GetThrottledKey(utcNow, organizationId), true, TimeSpan.FromMinutes(5)); + await _cache.SetAsync(GetThrottledKey(utcNow, organizationId), true, _bucketSize); } } + public async Task IncrementBlockedAsync(string organizationId, string? projectId, int eventCount = 1) { if (eventCount <= 0) @@ -620,6 +649,49 @@ private string GetProjectSetKey(DateTime utcTime) return $"usage:{bucket}:projects"; } + private string GetBudgetAlertSentKey(DateTime utcTime, string organizationId, int threshold) + { + int monthBucket = GetTotalBucket(utcTime); + return $"usage:budget-alert:{monthBucket}:{organizationId}:{threshold}"; + } + + private string GetProjectSmartThrottleKey(DateTime utcTime, string organizationId, string projectId) + { + int bucket = GetCurrentBucket(utcTime); + return $"usage:{bucket}:{organizationId}:{projectId}:smart-throttle"; + } + + private string GetSmartThrottleNotificationKey(DateTime utcTime, string organizationId, string projectId) + { + int monthBucket = GetTotalBucket(utcTime); + return $"usage:smart-throttle-notified:{monthBucket}:{organizationId}:{projectId}"; + } + private int GetCurrentBucket(DateTime utcTime) => utcTime.Floor(_bucketSize).ToEpoch(); private int GetTotalBucket(DateTime utcTime) => utcTime.StartOfMonth().ToEpoch(); } + +public class EventIngestAllowance +{ + public static readonly EventIngestAllowance Unlimited = new() { EventsLeft = Int32.MaxValue }; + + public int EventsLeft { get; init; } + public bool IsOverOrgLimit { get; init; } + public bool IsOverProjectLimit { get; init; } + public int EffectiveProjectLimit { get; init; } = -1; + public SmartThrottleResult SmartThrottle { get; init; } = SmartThrottleResult.NoThrottle; + public double SampleRate => SmartThrottle.SampleRate; +} + +public class SmartThrottleResult +{ + public const double DefaultSampleRate = 0.05; + public static readonly SmartThrottleResult NoThrottle = new() { SampleRate = 1.0 }; + + public double SampleRate { get; init; } = 1.0; + public bool IsThrottled { get; init; } + public double ProjectShare { get; init; } + public double FairShareRatio { get; init; } + public int CurrentProjectUsage { get; init; } + public int FairShareLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Utility/AppDiagnostics.cs b/src/Exceptionless.Core/Utility/AppDiagnostics.cs index c081925b6c..a88ec49519 100644 --- a/src/Exceptionless.Core/Utility/AppDiagnostics.cs +++ b/src/Exceptionless.Core/Utility/AppDiagnostics.cs @@ -96,6 +96,7 @@ public GaugeInfo(Meter meter, string name) internal static readonly Counter EventsProcessErrors = Meter.CreateCounter("ex.events.processing.errors", description: "Errors processing events"); internal static readonly Counter EventsDiscarded = Meter.CreateCounter("ex.events.discarded", description: "Events that were discarded"); internal static readonly Counter EventsBlocked = Meter.CreateCounter("ex.events.blocked", description: "Events that were blocked"); + internal static readonly Counter EventsSmartThrottled = Meter.CreateCounter("ex.events.smart_throttled", description: "Events blocked by automatic project throttling"); internal static readonly Counter EventsProcessCancelled = Meter.CreateCounter("ex.events.processing.cancelled", description: "Events that started processing and were cancelled"); internal static readonly Counter EventsDeleted = Meter.CreateCounter("ex.events.deleted", description: "Events that were deleted"); internal static readonly Counter EventsRetryCount = Meter.CreateCounter("ex.events.retry.count", description: "Events where processing was retried"); diff --git a/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html b/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html new file mode 100644 index 0000000000..c0f91e5f6f --- /dev/null +++ b/src/Exceptionless.EmailTemplates/src/pages/organization-budget-alert.html @@ -0,0 +1,20 @@ + + + + +

+ \{{OrganizationName}} has reached \{{Threshold}}% of its monthly event allowance. +

+

+ The organization has accepted \{{CurrentEventCount}} events. The \{{Threshold}}% alert is + \{{ThresholdEventCount}} events, the current allowance is \{{EventLimit}}, and \{{RemainingEventCount}} + events remain. +

+
+ +
+ + + diff --git a/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html b/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html new file mode 100644 index 0000000000..5269a9d8fd --- /dev/null +++ b/src/Exceptionless.EmailTemplates/src/pages/project-smart-throttle.html @@ -0,0 +1,23 @@ + + + + +

+ Smart throttling is active for \{{ProjectName}} in \{{OrganizationName}}. +

+

+ This project has accepted \{{CurrentEventCount}} events this month, above its current fair-share limit of + \{{EventLimit}} events. To protect the organization's remaining allowance, Exceptionless is accepting a + stable \{{SamplePercent}}% sample from this project. Events outside that sample are counted as blocked. +

+

+ Other projects are unaffected unless they independently meet the same smart-throttling criteria. +

+
+ +
+ + + diff --git a/src/Exceptionless.Web/Bootstrapper.cs b/src/Exceptionless.Web/Bootstrapper.cs index 7eb404ae13..dfb8bd47d3 100644 --- a/src/Exceptionless.Web/Bootstrapper.cs +++ b/src/Exceptionless.Web/Bootstrapper.cs @@ -41,5 +41,11 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); } } diff --git a/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs b/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs index 58935cf119..6180e71431 100644 --- a/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs +++ b/src/Exceptionless.Web/ClientApp/api-templates/schemas.ejs @@ -61,7 +61,7 @@ const typeToZod = (rawType, isRequired, nullable, field) => { // but outputs "X | null" for nullable arrays fixed via generate-api.mjs workaround. if (type.startsWith('null | ')) { let innerType = type.replace('null | ', ''); - if (innerType.startsWith('(') && innerType.endsWith(')') && !innerType.endsWith('[]')) { + if (innerType.startsWith('(') && innerType.endsWith(')')) { innerType = innerType.slice(1, -1); } return `${typeToZod(innerType, true, false, field)}.nullable()`; @@ -69,13 +69,13 @@ const typeToZod = (rawType, isRequired, nullable, field) => { if (type.endsWith(' | null')) { let innerType = type.replace(' | null', ''); - if (innerType.startsWith('(') && innerType.endsWith(')') && !innerType.endsWith('[]')) { + if (innerType.startsWith('(') && innerType.endsWith(')')) { innerType = innerType.slice(1, -1); } return `${typeToZod(innerType, true, false, field)}.nullable()`; } - if (type.startsWith('(') && type.endsWith(')') && !type.includes('[]')) { + if (type.startsWith('(') && type.endsWith(')')) { type = type.slice(1, -1); } @@ -191,7 +191,9 @@ const typeToZod = (rawType, isRequired, nullable, field) => { }; const buildZodField = (field) => { - const rawType = field.value || 'unknown'; + const hasNullableOneOf = Array.isArray(field.oneOf) && field.oneOf.some((alternative) => alternative.type === 'null'); + const nonNullAlternative = hasNullableOneOf ? field.oneOf.find((alternative) => alternative.type !== 'null') : undefined; + const rawType = nonNullAlternative?.$parsed?.content || field.value || 'unknown'; const type = rawType.replace(/\bany\b/g, 'unknown'); const cleanedType = cleanType(type); const readableName = toReadableName(field.name); @@ -210,13 +212,15 @@ const buildZodField = (field) => { const isUuidField = field.format === 'uuid' || fieldNameLower.endsWith('_uuid'); // Check if this is a nullable union type (e.g., "null | string" or "string | null") - const isNullableUnion = cleanedType.startsWith('null | ') || cleanedType.endsWith(' | null'); + const unwrappedType = cleanedType.startsWith('(') && cleanedType.endsWith(')') ? cleanedType.slice(1, -1) : cleanedType; + const isNullableUnion = unwrappedType.startsWith('null | ') || unwrappedType.endsWith(' | null'); + const isFieldNullable = field.nullable || field.isNullable || hasNullableOneOf || isNullableUnion; // Get the base type (without null union) - let baseType = cleanedType; + let baseType = unwrappedType; if (isNullableUnion) { baseType = cleanedType.replace('null | ', '').replace(' | null', ''); - if (baseType.startsWith('(') && baseType.endsWith(')') && !baseType.endsWith('[]')) { + if (baseType.startsWith('(') && baseType.endsWith(')')) { baseType = baseType.slice(1, -1); } } @@ -344,12 +348,12 @@ const buildZodField = (field) => { } // Handle nullable (adds .nullable()) - if (isNullableUnion && !zodType.includes('.nullable()')) { + if (isFieldNullable && !zodType.includes('.nullable()')) { zodType = zodType + '.nullable()'; } // Handle optional (non-required fields and nullable fields are optional) - if (field.nullable || !field.isRequired) { + if (isFieldNullable || !field.isRequired) { zodType = zodType + '.optional()'; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts index ecc2301730..e8764e67ff 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts @@ -8,7 +8,7 @@ import { queryKeys as userQueryKeys } from '$features/users/api.svelte'; import { type FetchClientResponse, type ProblemDetails, useFetchClient } from '@exceptionless/fetchclient'; import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query'; -import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, ViewOrganization } from './models'; +import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, UpdateOrganization, ViewOrganization } from './models'; export async function invalidateOrganizationQueries(queryClient: QueryClient, message: WebSocketMessageValue<'OrganizationChanged'>) { const { id } = message; @@ -413,9 +413,9 @@ export function getPlansQuery(request: GetPlansRequest) { export function patchOrganization(request: PatchOrganizationRequest) { const queryClient = useQueryClient(); - return createMutation(() => ({ + return createMutation(() => ({ enabled: () => !!accessToken.current && !!request.route.id, - mutationFn: async (data: NewOrganization) => { + mutationFn: async (data: UpdateOrganization) => { const client = useFetchClient(); const response = await client.patchJSON(`organizations/${request.route.id}`, data); return response.data!; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts new file mode 100644 index 0000000000..37b9f40a06 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from 'vitest'; + +import { getBudgetThresholdEventCount, parseBudgetThresholds } from './budget-utils'; +import { BudgetAlertCardSchema } from './schemas'; + +describe('organization budget controls', () => { + it('normalizes, sorts, and deduplicates threshold input', () => { + expect(parseBudgetThresholds('80, 50, 80, 90')).toEqual([50, 80, 90]); + expect(parseBudgetThresholds('')).toEqual([]); + expect(parseBudgetThresholds(' , ')).toEqual([]); + }); + + it('rejects decimals, out-of-range thresholds, and empty enabled settings', () => { + expect(BudgetAlertCardSchema.safeParse({ enabled: true, thresholds: '' }).success).toBe(false); + expect(BudgetAlertCardSchema.safeParse({ enabled: false, thresholds: '50.5' }).success).toBe(false); + expect(BudgetAlertCardSchema.safeParse({ enabled: false, thresholds: '100' }).success).toBe(false); + }); + + it('calculates threshold event counts and disables percentages for unlimited plans', () => { + expect(getBudgetThresholdEventCount(1001, 50)).toBe(501); + expect(getBudgetThresholdEventCount(-1, 50)).toBeNull(); + expect(getBudgetThresholdEventCount(1001, 50.5)).toBeNull(); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts new file mode 100644 index 0000000000..310def613c --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/budget-utils.ts @@ -0,0 +1,20 @@ +export function getBudgetThresholdEventCount(eventLimit: number, threshold: number): null | number { + if (!Number.isFinite(eventLimit) || !Number.isInteger(threshold) || eventLimit < 0 || threshold < 1 || threshold > 99) { + return null; + } + + return Math.ceil((eventLimit * threshold) / 100); +} + +export function parseBudgetThresholds(value: string): number[] { + return [ + ...new Set( + value + .split(',') + .map((threshold) => threshold.trim()) + .filter(Boolean) + .map(Number) + .filter(Number.isFinite) + ) + ].sort((left, right) => left - right); +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte index eae4237066..67831a2edf 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/notifications/organization-notifications.stories.svelte @@ -25,6 +25,7 @@ has_slack_integration: false, id: '1', is_configured: false, + is_smart_throttled: false, name: 'Frontend App', organization_id: 'org1', organization_name: 'Test Org', diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte new file mode 100644 index 0000000000..57439ac359 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/components/organization-budget-alert-card.svelte @@ -0,0 +1,164 @@ + + + + + Budget alerts + Receive an email when accepted organization usage crosses selected percentages of the monthly allowance. + +
{ + event.preventDefault(); + event.stopPropagation(); + form.handleSubmit(); + }} + > + + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + + {#snippet children(field)} + + + Enable budget alerts + Alerts are sent once per threshold per monthly usage period. + + field.handleChange(checked)} + disabled={update.isPending || isUnlimited} + /> + + {/snippet} + + + + {#snippet children(field)} + + Threshold percentages + field.handleChange(event.currentTarget.value)} + aria-invalid={ariaInvalid(field)} + aria-describedby="budget-threshold-help" + disabled={update.isPending} + /> + Comma-separated whole numbers from 1 to 99. + + +

+ Current allowance: + {#if isUnlimited} + unlimited; percentage alerts are inactive. + {:else} + events. + {/if} +

+ {#if !isUnlimited} +
    + {#each parseBudgetThresholds(field.state.value).filter((threshold) => Number.isInteger(threshold) && threshold >= 1 && threshold <= 99) as threshold (threshold)} +
  • + {threshold}% = + events +
    • + {/each} +
+ {/if} + + {/snippet} + + + + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + + +
+ diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts index 85d1fe5de6..9bea17bcf4 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts @@ -1,4 +1,4 @@ -export type { Invoice, InvoiceGridModel, NewOrganization, ViewOrganization } from '$generated/api'; +export type { Invoice, InvoiceGridModel, NewOrganization, OrganizationBudgetAlertSettings, ViewOrganization } from '$generated/api'; // TODO: This should be generated from the backend enum - investigate why it wasn't included in the generated API export enum SuspensionCode { @@ -7,3 +7,13 @@ export enum SuspensionCode { Abuse = 2, Other = 100 } + +export interface OrganizationBudgetAlertSettingsUpdate { + enabled: boolean; + thresholds: number[]; +} + +export interface UpdateOrganization { + budget_alert_settings?: null | OrganizationBudgetAlertSettingsUpdate; + name?: string; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts index c380dca5e7..93459b1a9d 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts @@ -1,9 +1,45 @@ -import { date, type infer as Infer, number, object, string, enum as zodEnum } from 'zod'; +import { array, boolean, date, type infer as Infer, number, object, string, enum as zodEnum } from 'zod'; import { SuspensionCode } from './models'; export { type NewOrganizationFormData, NewOrganizationSchema } from '$generated/schemas'; +export const BudgetAlertSettingsSchema = object({ + enabled: boolean(), + thresholds: array(number().int().min(1).max(99)) +}).superRefine((settings, context) => { + if (settings.enabled && settings.thresholds.length === 0) { + context.addIssue({ code: 'custom', message: 'At least one threshold is required when alerts are enabled.', path: ['thresholds'] }); + } +}); +export type BudgetAlertSettingsFormData = Infer; + +export const BudgetAlertCardSchema = object({ + enabled: boolean(), + thresholds: string() +}).superRefine((settings, context) => { + const values = settings.thresholds + .split(',') + .map((value) => value.trim()) + .filter(Boolean); + + if (settings.enabled && values.length === 0) { + context.addIssue({ code: 'custom', message: 'Enter at least one threshold.', path: ['thresholds'] }); + return; + } + + if (values.some((value) => !/^\d+$/.test(value) || Number(value) < 1 || Number(value) > 99)) { + context.addIssue({ code: 'custom', message: 'Use whole-number percentages from 1 to 99.', path: ['thresholds'] }); + } +}); +export type BudgetAlertCardFormData = Infer; + +export const UpdateOrganizationSchema = object({ + budget_alert_settings: BudgetAlertSettingsSchema.nullable().optional(), + name: string().min(1, 'Name is required').optional() +}); +export type UpdateOrganizationFormData = Infer; + export const SetBonusOrganizationSchema = object({ bonusEvents: number().int('Bonus events must be a whole number'), expires: date().optional() diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts new file mode 100644 index 0000000000..d32aab88fd --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it } from 'vitest'; + +import type { ViewOrganization } from './models'; + +import { getEffectiveEventLimit } from './utils'; + +function organization(overrides: Partial): ViewOrganization { + return { bonus_events_per_month: 0, max_events_per_month: 1000, usage: [], ...overrides } as ViewOrganization; +} + +describe('getEffectiveEventLimit', () => { + it('uses the current usage limit so active bonus events match backend enforcement', () => { + const now = new Date(); + const usageDate = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 1)); + const value = organization({ + bonus_events_per_month: 250, + max_events_per_month: 1000, + usage: [{ blocked: 0, date: usageDate.toISOString(), deleted: 0, discarded: 0, limit: 1250, too_big: 0, total: 0 }] + }); + + expect(getEffectiveEventLimit(value)).toBe(1250); + }); + + it('preserves unlimited organizations', () => { + expect(getEffectiveEventLimit(organization({ max_events_per_month: -1 }))).toBe(-1); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts index 11cdb6569a..d0503640ba 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/utils.ts @@ -2,6 +2,25 @@ import { isSameUtcMonth } from '$features/shared/dates'; import type { ViewOrganization } from './models'; +export function getEffectiveEventLimit(organization?: ViewOrganization): number { + if (!organization?.max_events_per_month) { + return 0; + } + + if (organization.max_events_per_month < 0) { + return -1; + } + + const now = new Date(); + const currentUsage = organization.usage?.[organization.usage.length - 1]; + if (currentUsage && isSameUtcMonth(new Date(currentUsage.date), now)) { + return currentUsage.limit; + } + + const bonusEvents = organization.bonus_expiration && new Date(organization.bonus_expiration) > now ? organization.bonus_events_per_month : 0; + return organization.max_events_per_month + bonusEvents; +} + export function getNextBillingDateUtc(organization?: ViewOrganization): Date { if (organization?.subscribe_date) { console.log('Organization subscribe date for next billing date:', organization.subscribe_date); @@ -12,21 +31,20 @@ export function getNextBillingDateUtc(organization?: ViewOrganization): Date { } export function getRemainingEventLimit(organization?: ViewOrganization): number { - if (!organization?.max_events_per_month) { + const eventLimit = getEffectiveEventLimit(organization); + if (eventLimit <= 0) { return 0; } const now = new Date(); - const bonusEvents = organization.bonus_expiration && new Date(organization.bonus_expiration) > now ? organization.bonus_events_per_month : 0; - - const usage = organization.usage && organization.usage[organization.usage.length - 1]; + const usage = organization?.usage?.[organization.usage.length - 1]; if (usage) { const usageDate = new Date(usage.date); if (isSameUtcMonth(usageDate, now)) { - const remaining = usage.limit - usage.total; + const remaining = eventLimit - usage.total; return remaining > 0 ? remaining : 0; } } - return organization.max_events_per_month + bonusEvents; + return eventLimit; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts new file mode 100644 index 0000000000..a499538167 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from 'vitest'; + +import { createProjectIngestLimit, getEffectiveProjectLimit } from './budget-utils'; +import { ProjectBudgetCardSchema } from './schemas'; + +describe('project budget controls', () => { + it('rejects truncated fixed decimals and out-of-range percentages', () => { + expect(ProjectBudgetCardSchema.safeParse({ type: 'fixed', value: '1.5' }).success).toBe(false); + expect(ProjectBudgetCardSchema.safeParse({ type: 'fixed', value: '10' }).success).toBe(true); + expect(ProjectBudgetCardSchema.safeParse({ type: 'percent', value: '100.1' }).success).toBe(false); + }); + + it('maps modes to additive API payloads', () => { + expect(createProjectIngestLimit('none', '')).toBeNull(); + expect(createProjectIngestLimit('fixed', 'not-a-number')).toBeNull(); + expect(createProjectIngestLimit('fixed', '20000')).toMatchObject({ fixed_limit: 20000, type: 0 }); + expect(createProjectIngestLimit('percent', '25.5')).toMatchObject({ percent_of_organization_limit: 25.5, type: 1 }); + }); + + it('calculates percentage caps and clamps fixed caps to finite organization allowance', () => { + expect(getEffectiveProjectLimit(1001, createProjectIngestLimit('percent', '50'))).toBe(501); + expect(getEffectiveProjectLimit(1000, createProjectIngestLimit('fixed', '2000'))).toBe(1000); + expect(getEffectiveProjectLimit(-1, createProjectIngestLimit('percent', '50'))).toBeNull(); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts new file mode 100644 index 0000000000..ad481d6f4d --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/budget-utils.ts @@ -0,0 +1,39 @@ +import { type ProjectIngestLimit, ProjectIngestLimitType } from './models'; + +export type ProjectBudgetType = 'fixed' | 'none' | 'percent'; + +export function createProjectIngestLimit(type: ProjectBudgetType, value: string): null | ProjectIngestLimit { + if (type === 'none') { + return null; + } + + const numericValue = Number(value.trim()); + if (!Number.isFinite(numericValue)) { + return null; + } + + return type === 'fixed' + ? { fixed_limit: numericValue, percent_of_organization_limit: null, type: ProjectIngestLimitType.Fixed } + : { fixed_limit: null, percent_of_organization_limit: numericValue, type: ProjectIngestLimitType.PercentOfOrganizationLimit }; +} + +export function getEffectiveProjectLimit(organizationLimit: number, ingestLimit: null | ProjectIngestLimit): null | number { + if (!ingestLimit) { + return null; + } + + if (ingestLimit.type === ProjectIngestLimitType.Fixed) { + if (!Number.isInteger(ingestLimit.fixed_limit) || (ingestLimit.fixed_limit ?? 0) <= 0) { + return null; + } + + return organizationLimit < 0 ? ingestLimit.fixed_limit! : Math.min(ingestLimit.fixed_limit!, organizationLimit); + } + + const percentage = ingestLimit.percent_of_organization_limit; + if (organizationLimit < 0 || percentage == null || !Number.isFinite(percentage) || percentage <= 0 || percentage > 100) { + return null; + } + + return Math.min(organizationLimit, Math.ceil((organizationLimit * percentage) / 100)); +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte new file mode 100644 index 0000000000..69919215e6 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte @@ -0,0 +1,193 @@ + + + + + Project event budget + Optionally cap accepted events for this project without changing the organization allowance. + +
{ + event.preventDefault(); + event.stopPropagation(); + form.handleSubmit(); + }} + > + + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + + {#snippet children(field)} + + Limit type + { + const nextType = value as ProjectBudgetType; + selectedType = nextType; + field.handleChange(nextType); + if (nextType === 'none') { + form.setFieldValue('value', ''); + } + }} + disabled={update.isPending} + > + + {selectedType === 'none' + ? 'No project limit' + : selectedType === 'fixed' + ? 'Fixed event count' + : 'Percentage of organization allowance'} + + + + No project limit + Fixed event count + Percentage of organization allowance + + + + + {/snippet} + + + {#if selectedType !== 'none'} + + {#snippet children(field)} + + {selectedType === 'fixed' ? 'Monthly event count' : 'Percentage'} + field.handleChange(event.currentTarget.value)} + aria-invalid={ariaInvalid(field)} + disabled={update.isPending} + /> + + {selectedType === 'fixed' + ? 'Use a whole number greater than 0. The effective cap cannot exceed a finite organization allowance.' + : 'Use a value greater than 0 and no more than 100.'} + + + + {/snippet} + + {/if} + + state.values}> + {#snippet children(values)} + {@const draftLimit = createProjectIngestLimit(values.type, values.value)} + {@const effectiveLimit = getEffectiveProjectLimit(organizationLimit, draftLimit)} +

+ Effective project limit: + {#if effectiveLimit == null} + organization allowance + {:else} + events per month + {/if} +

+ {#if values.type === 'fixed' && organizationLimit >= 0 && (draftLimit?.fixed_limit ?? 0) > organizationLimit} + + Fixed limit will be clamped + The effective cap cannot exceed the organization's finite event allowance. + + {/if} + {/snippet} + + + {#if project.is_smart_throttled} + + Smart throttling active 5% sample + This project is being sampled independently; other projects remain unaffected. + + {/if} + + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + + +
+ diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts index a055dfb212..f30fa01b48 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts @@ -1,6 +1,10 @@ -export type { ClientConfiguration, NewProject, NotificationSettings, UpdateProject, ViewProject } from '$generated/api'; +import type { UpdateProject as GeneratedUpdateProject } from '$generated/api'; +export { ProjectIngestLimitType } from '$generated/api'; +export type { ClientConfiguration, NewProject, NotificationSettings, ProjectIngestLimit, ViewProject } from '$generated/api'; export interface ClientConfigurationSetting { key: string; value: string; } + +export type UpdateProject = Partial; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts index e495e7dd4f..797963c2c4 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts @@ -5,7 +5,7 @@ import { type NotificationSettingsFormData, NotificationSettingsSchema } from '$generated/schemas'; -import { type infer as Infer, object, string } from 'zod'; +import { discriminatedUnion, type infer as Infer, literal, nullable, number, object, optional, string, enum as zodEnum } from 'zod'; export { type NewProjectFormData, NewProjectSchema, type NotificationSettingsFormData, NotificationSettingsSchema }; @@ -15,5 +15,50 @@ export const ClientConfigurationSettingSchema = object({ }); export type ClientConfigurationSettingFormData = Infer; +export const FixedIngestLimitSchema = object({ + fixed_limit: number().int().min(1, 'Limit must be at least 1'), + percent_of_organization_limit: optional(nullable(number())), + type: literal(0) +}); + +export const PercentIngestLimitSchema = object({ + fixed_limit: optional(nullable(number())), + percent_of_organization_limit: number().positive('Percentage must be greater than 0').max(100, 'Percentage must be at most 100'), + type: literal(1) +}); + +export const IngestLimitSchema = discriminatedUnion('type', [FixedIngestLimitSchema, PercentIngestLimitSchema]); +export type IngestLimitFormData = Infer; + export const UpdateProjectSchema = GeneratedUpdateProjectSchema.partial(); export type UpdateProjectFormData = Infer; + +export const UpdateProjectIngestLimitSchema = object({ + ingest_limit: IngestLimitSchema.nullable().optional() +}); +export type UpdateProjectIngestLimitFormData = Infer; + +export const ProjectBudgetCardSchema = object({ + type: zodEnum(['none', 'fixed', 'percent']), + value: string() +}).superRefine((budget, context) => { + if (budget.type === 'none') { + return; + } + + const value = budget.value.trim(); + if (budget.type === 'fixed') { + const numericValue = Number(value); + if (!/^\d+$/.test(value) || !Number.isSafeInteger(numericValue) || numericValue < 1 || numericValue > 2_147_483_647) { + context.addIssue({ code: 'custom', message: 'Enter a whole number from 1 to 2,147,483,647.', path: ['value'] }); + } + + return; + } + + const numericValue = Number(value); + if (!/^(?:\d+\.?\d*|\.\d+)$/.test(value) || !Number.isFinite(numericValue) || numericValue <= 0 || numericValue > 100) { + context.addIssue({ code: 'custom', message: 'Enter a percentage greater than 0 and no more than 100.', path: ['value'] }); + } +}); +export type ProjectBudgetCardFormData = Infer; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts index 1e8554a074..9f5fc7bf41 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts @@ -7,6 +7,11 @@ export enum StackStatus { Discarded = "discarded", } +export enum ProjectIngestLimitType { + Fixed = 0, + PercentOfOrganizationLimit = 1, +} + export enum BillingStatus { Trialing = 0, Active = 1, @@ -130,6 +135,7 @@ export interface NewProject { organization_id: string; name: string; delete_bot_data_enabled: boolean; + ingest_limit?: null | ProjectIngestLimit; promoted_tabs?: string[] | null; } @@ -271,6 +277,15 @@ export interface OAuthTokenResponse { resource?: null | string; } +export interface OrganizationBudgetAlertSettings { + enabled: boolean; + /** + * Percentage thresholds of the organization's effective monthly event allowance. + * Example: [50, 80, 90]. + */ + thresholds: number[]; +} + export interface PersistentEvent { /** * Unique id that identifies an event. @@ -350,6 +365,23 @@ export interface PredefinedSavedViewDefinition { showChart?: null | boolean; } +export interface ProblemDetails { + type?: null | string; + title?: null | string; + /** @format int32 */ + status?: null | number; + detail?: null | string; + instance?: null | string; +} + +export interface ProjectIngestLimit { + type: ProjectIngestLimitType; + /** @format int32 */ + fixed_limit?: null | number; + /** @format double */ + percent_of_organization_limit?: null | number; +} + export interface ResetPasswordModel { password_reset_token: string; password: string; @@ -458,11 +490,18 @@ export interface UpdateEvent { description?: null | string; } +/** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ +export interface UpdateOrganization { + name: string; + budget_alert_settings?: null | OrganizationBudgetAlertSettings; +} + /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ export interface UpdateProject { name: string; delete_bot_data_enabled: boolean; - promoted_tabs?: string[] | null; + ingest_limit?: null | ProjectIngestLimit; + promoted_tabs?: null | string[]; } /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ @@ -474,7 +513,7 @@ export interface UpdateSavedView { slug?: null | string; filter_definitions?: null | string; columns?: null | Record; - column_order?: string[] | null; + column_order?: null | string[]; show_stats?: null | boolean; show_chart?: null | boolean; } @@ -660,6 +699,7 @@ export interface ViewOrganization { is_throttled: boolean; is_over_monthly_limit: boolean; is_over_request_limit: boolean; + budget_alert_settings?: null | OrganizationBudgetAlertSettings; } export interface ViewProject { @@ -681,6 +721,12 @@ export interface ViewProject { event_count: number; has_premium_features: boolean; has_slack_integration: boolean; + ingest_limit?: null | ProjectIngestLimit; + /** @format int32 */ + effective_ingest_limit?: null | number; + is_smart_throttled: boolean; + /** @format double */ + smart_throttle_sample_rate?: null | number; usage_hours: UsageHourInfo[]; usage: UsageInfo[]; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts index 32ef74643c..8f739347ed 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts @@ -27,6 +27,7 @@ export const StackStatusSchema = zodEnum([ "ignored", "discarded", ]); +export const ProjectIngestLimitTypeSchema = union([literal(0), literal(1)]); export const BillingStatusSchema = union([ literal(0), literal(1), @@ -91,8 +92,10 @@ export const CountResultSchema = object({ aggregations: record( string(), lazy(() => IAggregateSchema), - ), - data: record(string(), unknown()).nullable(), + ) + .nullable() + .optional(), + data: record(string(), unknown()).nullable().optional(), }); export type CountResultFormData = Infer; @@ -169,6 +172,9 @@ export const NewProjectSchema = object({ .regex(/^[a-fA-F0-9]{24}$/, "Organization id has invalid format"), name: string().min(1, "Name is required"), delete_bot_data_enabled: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), promoted_tabs: array(string()).nullable().optional(), }); export type NewProjectFormData = Infer; @@ -380,6 +386,14 @@ export const OAuthTokenResponseSchema = object({ }); export type OAuthTokenResponseFormData = Infer; +export const OrganizationBudgetAlertSettingsSchema = object({ + enabled: boolean(), + thresholds: array(number()), +}); +export type OrganizationBudgetAlertSettingsFormData = Infer< + typeof OrganizationBudgetAlertSettingsSchema +>; + export const PersistentEventSchema = object({ id: string() .length(24, "Id must be exactly 24 characters") @@ -399,7 +413,8 @@ export const PersistentEventSchema = object({ type: string() .min(1, "Type is required") .max(100, "Type must be at most 100 characters") - .nullable(), + .nullable() + .optional(), source: string() .min(1, "Source is required") .max(2000, "Source must be at most 2000 characters") @@ -431,7 +446,7 @@ export const PredefinedSavedViewDefinitionSchema = object({ filter: string().min(1, "Filter is required").nullable().optional(), time: string().min(1, "Time is required").nullable().optional(), sort: string().min(1, "Sort is required").nullable().optional(), - filterDefinitions: unknown().optional(), + filterDefinitions: unknown().nullable().optional(), columns: record(string(), boolean()).nullable().optional(), columnOrder: array(string()).nullable().optional(), showStats: boolean().nullable().optional(), @@ -441,6 +456,22 @@ export type PredefinedSavedViewDefinitionFormData = Infer< typeof PredefinedSavedViewDefinitionSchema >; +export const ProblemDetailsSchema = object({ + type: string().min(1, "Type is required").nullable().optional(), + title: string().min(1, "Title is required").nullable().optional(), + status: int32().nullable().optional(), + detail: string().min(1, "Detail is required").nullable().optional(), + instance: string().min(1, "Instance is required").nullable().optional(), +}); +export type ProblemDetailsFormData = Infer; + +export const ProjectIngestLimitSchema = object({ + type: ProjectIngestLimitTypeSchema, + fixed_limit: int32().nullable().optional(), + percent_of_organization_limit: number().nullable().optional(), +}); +export type ProjectIngestLimitFormData = Infer; + export const ResetPasswordModelSchema = object({ password_reset_token: string().length( 40, @@ -503,7 +534,7 @@ export const StackSchema = object({ export type StackFormData = Infer; export const StringStringValuesKeyValuePairSchema = object({ - key: string().min(1, "Key is required").nullable(), + key: string().min(1, "Key is required").nullable().optional(), value: array(string()), }); export type StringStringValuesKeyValuePairFormData = Infer< @@ -511,7 +542,7 @@ export type StringStringValuesKeyValuePairFormData = Infer< >; export const StringValueFromBodySchema = object({ - value: string().min(1, "Value is required").nullable(), + value: string().min(1, "Value is required").nullable().optional(), }); export type StringValueFromBodyFormData = Infer< typeof StringValueFromBodySchema @@ -535,9 +566,20 @@ export const UpdateEventSchema = object({ }); export type UpdateEventFormData = Infer; +export const UpdateOrganizationSchema = object({ + name: string().min(1, "Name is required").optional(), + budget_alert_settings: lazy(() => OrganizationBudgetAlertSettingsSchema) + .nullable() + .optional(), +}); +export type UpdateOrganizationFormData = Infer; + export const UpdateProjectSchema = object({ name: string().min(1, "Name is required").optional(), delete_bot_data_enabled: boolean().optional(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), promoted_tabs: array(string()).nullable().optional(), }); export type UpdateProjectFormData = Infer; @@ -725,6 +767,9 @@ export const ViewOrganizationSchema = object({ is_throttled: boolean(), is_over_monthly_limit: boolean(), is_over_request_limit: boolean(), + budget_alert_settings: lazy(() => OrganizationBudgetAlertSettingsSchema) + .nullable() + .optional(), }); export type ViewOrganizationFormData = Infer; @@ -746,6 +791,12 @@ export const ViewProjectSchema = object({ event_count: int(), has_premium_features: boolean(), has_slack_integration: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema) + .nullable() + .optional(), + effective_ingest_limit: int32().nullable().optional(), + is_smart_throttled: boolean(), + smart_throttle_sample_rate: number().nullable().optional(), usage_hours: array(lazy(() => UsageHourInfoSchema)), usage: array(lazy(() => UsageInfoSchema)), }); diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte index 0c42001cd2..627e61dc1a 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte @@ -10,6 +10,7 @@ import { env } from '$env/dynamic/public'; import { ChangePlanDialog } from '$features/billing'; import { getOrganizationQuery } from '$features/organizations/api.svelte'; + import OrganizationBudgetAlertCard from '$features/organizations/components/organization-budget-alert-card.svelte'; import { getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; import { formatDateLabel, formatLongDate } from '$shared/dates'; import { scaleUtc } from 'd3-scale'; @@ -139,6 +140,10 @@ {/if} +{#if organizationQuery.data} + +{/if} + {#if changePlanDialogOpen && organizationQuery.data} (changePlanDialogOpen = false)} /> {/if} diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte index 27f5590cde..2f7449a51a 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte @@ -11,8 +11,9 @@ import { ChangePlanDialog } from '$features/billing'; import { getOrganizationQuery } from '$features/organizations/api.svelte'; import { organization } from '$features/organizations/context.svelte'; - import { getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; + import { getEffectiveEventLimit, getNextBillingDateUtc, getRemainingEventLimit } from '$features/organizations/utils'; import { getProjectQuery } from '$features/projects/api.svelte'; + import ProjectIngestLimitCard from '$features/projects/components/project-ingest-limit-card.svelte'; import { formatDateLabel, formatLongDate } from '$shared/dates'; import { scaleUtc } from 'd3-scale'; import { curveNatural } from 'd3-shape'; @@ -32,6 +33,7 @@ const hasMonthlyUsage = $derived((organizationQuery.data?.max_events_per_month ?? 0) > 0); const canChangePlan = $derived(organizationQuery.isSuccess && !!env.PUBLIC_STRIPE_PUBLISHABLE_KEY); const remainingEventLimit = $derived(getRemainingEventLimit(organizationQuery.data)); + const organizationLimit = $derived(getEffectiveEventLimit(organizationQuery.data)); const nextBillingDate = $derived(getNextBillingDateUtc(organizationQuery.data)); const projectQuery = getProjectQuery({ @@ -44,15 +46,15 @@ let changePlanDialogOpen = $state(false); - const chartConfig = { + const chartConfig = $derived({ blocked: { color: 'var(--chart-2)', label: 'Blocked' }, deleted: { color: 'var(--chart-7)', label: 'Deleted' }, discarded: { color: 'var(--chart-3)', label: 'Discarded' }, - limit: { color: 'var(--chart-6)', label: 'Limit' }, + limit: { color: 'var(--chart-6)', label: projectQuery.data?.effective_ingest_limit == null ? 'Organization limit' : 'Project limit' }, org_total: { color: 'var(--chart-5)', label: 'Total in Organization' }, too_big: { color: 'var(--chart-4)', label: 'Too Big' }, total: { color: 'var(--chart-1)', label: 'Total' } - } satisfies Chart.ChartConfig; + } satisfies Chart.ChartConfig); const chartData = $derived.by(() => { const project = projectQuery.data; @@ -76,7 +78,7 @@ date: new Date(projItem.date), deleted: projItem.deleted, discarded: projItem.discarded, - limit: orgItem?.limit || 0, + limit: project.effective_ingest_limit ?? orgItem?.limit ?? 0, org_total: orgItem?.total || 0, too_big: projItem.too_big, total: projItem.total @@ -84,7 +86,7 @@ }); }); - const series = [ + const series = $derived([ { key: 'org_total', ...chartConfig.org_total }, { key: 'total', ...chartConfig.total }, { key: 'discarded', ...chartConfig.discarded }, @@ -99,7 +101,7 @@ line: { class: '[stroke-dasharray:4]' } } } - ]; + ]);
@@ -168,6 +170,10 @@ {/if}
+{#if projectQuery.data && organizationQuery.data} + +{/if} + {#if changePlanDialogOpen && organizationQuery.data} (changePlanDialogOpen = false)} /> {/if} diff --git a/src/Exceptionless.Web/Controllers/OrganizationController.cs b/src/Exceptionless.Web/Controllers/OrganizationController.cs index b29f0cd516..73ab443643 100644 --- a/src/Exceptionless.Web/Controllers/OrganizationController.cs +++ b/src/Exceptionless.Web/Controllers/OrganizationController.cs @@ -32,7 +32,7 @@ namespace Exceptionless.Web.Controllers; [Route(API_PREFIX + "/organizations")] [Authorize(Policy = AuthorizationRoles.UserPolicy)] -public class OrganizationController : RepositoryApiController +public class OrganizationController : RepositoryApiController { private readonly OrganizationService _organizationService; private readonly ICacheClient _cacheClient; @@ -185,7 +185,7 @@ public Task> PostAsync(NewOrganization organizati [HttpPut] [Consumes("application/json")] [Route("{id:objectid}")] - public Task> PatchAsync(string id, Delta changes) + public Task> PatchAsync(string id, Delta changes) { return PatchImplAsync(id, changes); } @@ -968,14 +968,16 @@ public async Task IsNameAvailableAsync(string name) return StatusCode(StatusCodes.Status201Created); } - private async Task IsOrganizationNameAvailableInternalAsync(string name) + private async Task IsOrganizationNameAvailableInternalAsync(string name, string? excludeOrganizationId = null) { if (String.IsNullOrWhiteSpace(name)) return false; string decodedName = Uri.UnescapeDataString(name).Trim().ToLowerInvariant(); var results = await _repository.GetByIdsAsync(GetAssociatedOrganizationIds().ToArray(), o => o.Cache()); - return !results.Any(o => String.Equals(o.Name.Trim().ToLowerInvariant(), decodedName, StringComparison.OrdinalIgnoreCase)); + return !results.Any(o => + o.Id != excludeOrganizationId && + String.Equals(o.Name.Trim().ToLowerInvariant(), decodedName, StringComparison.OrdinalIgnoreCase)); } protected override async Task CanAddAsync(Organization value) @@ -1014,15 +1016,55 @@ await _messagePublisher.PublishAsync(new UserMembershipChanged return organization; } - protected override async Task CanUpdateAsync(Organization original, Delta changes) + protected override async Task CanUpdateAsync(Organization original, Delta changes) { var changed = changes.GetEntity(); - if (!await IsOrganizationNameAvailableInternalAsync(changed.Name)) + if (changes.ContainsChangedProperty(p => p.Name) && !await IsOrganizationNameAvailableInternalAsync(changed.Name, original.Id)) return PermissionResult.DenyWithMessage("A organization with this name already exists."); + if (changes.ContainsChangedProperty(p => p.BudgetAlertSettings!) && + changed.BudgetAlertSettings is { Enabled: true } && + original.GetMaxEventsPerMonthWithBonus(_timeProvider) < 0) + return PermissionResult.DenyWithMessage("Budget alerts cannot be enabled for an organization with an unlimited event allowance."); + return await base.CanUpdateAsync(original, changes); } + protected override async Task UpdateModelAsync(Organization original, Delta changes) + { + var changed = changes.GetEntity(); + bool budgetSettingsChanged = changes.ContainsChangedProperty(p => p.BudgetAlertSettings!); + bool wereAlertsEnabled = original.BudgetAlertSettings is { Enabled: true }; + var previousThresholds = original.BudgetAlertSettings?.Thresholds?.ToHashSet() ?? []; + + changes.Patch(original); + if (budgetSettingsChanged) + original.BudgetAlertSettings = changed.BudgetAlertSettings; + + var saved = await _repository.SaveAsync(original, o => o.Cache()); + + if (budgetSettingsChanged && saved.BudgetAlertSettings is { Enabled: true, Thresholds: not null }) + { + int[] thresholdsToEvaluate = wereAlertsEnabled + ? saved.BudgetAlertSettings.Thresholds.Except(previousThresholds).ToArray() + : saved.BudgetAlertSettings.Thresholds.ToArray(); + + if (thresholdsToEvaluate.Length > 0) + { + try + { + await _usageService.EvaluateBudgetAlertsAfterSettingsChangeAsync(saved, thresholdsToEvaluate); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to evaluate newly enabled budget alerts for organization {OrganizationId}", saved.Id); + } + } + } + + return saved; + } + protected override async Task CanDeleteAsync(Organization value) { if (!String.IsNullOrEmpty(value.StripeCustomerId) && !User.IsInRole(AuthorizationRoles.GlobalAdmin)) diff --git a/src/Exceptionless.Web/Controllers/ProjectController.cs b/src/Exceptionless.Web/Controllers/ProjectController.cs index 6bf7f4c0f3..df8bc3b140 100644 --- a/src/Exceptionless.Web/Controllers/ProjectController.cs +++ b/src/Exceptionless.Web/Controllers/ProjectController.cs @@ -738,7 +738,13 @@ protected override async Task AfterResultMapAsync(ICollection().ToList(); - var organizations = await _organizationRepository.GetByIdsAsync(viewProjects.Select(p => p.OrganizationId).ToArray(), o => o.Cache()); + var organizationsTask = _organizationRepository.GetByIdsAsync(viewProjects.Select(p => p.OrganizationId).ToArray(), o => o.Cache()); + var throttleStatesTask = _usageService.GetProjectSmartThrottleStatesAsync( + viewProjects.Select(project => (project.OrganizationId, project.Id)).ToArray()); + await Task.WhenAll(organizationsTask, throttleStatesTask); + + var organizations = await organizationsTask; + var throttleStates = await throttleStatesTask; foreach (var viewProject in viewProjects) { if (!viewProject.IsConfigured.HasValue) @@ -775,6 +781,18 @@ await _workItemQueue.EnqueueAsync(new SetProjectIsConfiguredWorkItem currentHourUsage.Discarded = realTimeUsage.CurrentHourUsage.Discarded; currentHourUsage.TooBig = realTimeUsage.CurrentHourUsage.TooBig; currentHourUsage.Deleted = realTimeUsage.CurrentHourUsage.Deleted; + + if (viewProject.IngestLimit is not null) + { + int effectiveLimit = _usageService.GetEffectiveProjectLimit(viewProject.IngestLimit, organization); + viewProject.EffectiveIngestLimit = effectiveLimit > 0 ? effectiveLimit : null; + } + + if (throttleStates.TryGetValue(viewProject.Id, out var throttleState)) + { + viewProject.IsSmartThrottled = true; + viewProject.SmartThrottleSampleRate = throttleState.SampleRate; + } } } @@ -810,16 +828,36 @@ protected override async Task CanUpdateAsync(Project original, if (changes.ContainsChangedProperty(p => p.Name) && !await IsProjectNameAvailableInternalAsync(original.OrganizationId, changed.Name)) return PermissionResult.DenyWithMessage("A project with this name already exists."); + if (changes.ContainsChangedProperty(p => p.IngestLimit!) && changed.IngestLimit?.Type is ProjectIngestLimitType.PercentOfOrganizationLimit) + { + var organization = await _organizationRepository.GetByIdAsync(original.OrganizationId, o => o.Cache()); + if (organization?.GetMaxEventsPerMonthWithBonus(_timeProvider) < 0) + return PermissionResult.DenyWithMessage("A percentage project ingest limit cannot be used with an unlimited organization allowance."); + } + return await base.CanUpdateAsync(original, changes); } protected override Task UpdateModelAsync(Project original, Delta changes) { + var changed = changes.GetEntity(); + bool ingestLimitChanged = changes.ContainsChangedProperty(p => p.IngestLimit!); + changes.Patch(original); + if (ingestLimitChanged) + original.IngestLimit = changed.IngestLimit; if (changes.ContainsChangedProperty(p => p.PromotedTabs!)) original.PromotedTabs = NormalizePromotedTabs(original.PromotedTabs); + if (ingestLimitChanged && original.IngestLimit is not null) + { + if (original.IngestLimit.Type is ProjectIngestLimitType.Fixed) + original.IngestLimit.PercentOfOrganizationLimit = null; + else if (original.IngestLimit.Type is ProjectIngestLimitType.PercentOfOrganizationLimit) + original.IngestLimit.FixedLimit = null; + } + return _repository.SaveAsync(original, o => o.Cache()); } diff --git a/src/Exceptionless.Web/Exceptionless.Web.csproj b/src/Exceptionless.Web/Exceptionless.Web.csproj index 0ab2d63e52..143d15f50a 100644 --- a/src/Exceptionless.Web/Exceptionless.Web.csproj +++ b/src/Exceptionless.Web/Exceptionless.Web.csproj @@ -17,6 +17,7 @@ + diff --git a/src/Exceptionless.Web/Mapping/OrganizationMapper.cs b/src/Exceptionless.Web/Mapping/OrganizationMapper.cs index 69b0e27556..4b4fa164e0 100644 --- a/src/Exceptionless.Web/Mapping/OrganizationMapper.cs +++ b/src/Exceptionless.Web/Mapping/OrganizationMapper.cs @@ -20,6 +20,7 @@ public OrganizationMapper(TimeProvider timeProvider) } public partial Organization MapToOrganization(NewOrganization source); + public partial Organization MapToOrganization(UpdateOrganization source); [MapperIgnoreTarget(nameof(ViewOrganization.IsOverMonthlyLimit))] [MapperIgnoreTarget(nameof(ViewOrganization.IsOverRequestLimit))] diff --git a/src/Exceptionless.Web/Mapping/ProjectMapper.cs b/src/Exceptionless.Web/Mapping/ProjectMapper.cs index 6d69ef583d..faabf4891d 100644 --- a/src/Exceptionless.Web/Mapping/ProjectMapper.cs +++ b/src/Exceptionless.Web/Mapping/ProjectMapper.cs @@ -18,6 +18,9 @@ public partial class ProjectMapper [MapperIgnoreTarget(nameof(ViewProject.OrganizationName))] [MapperIgnoreTarget(nameof(ViewProject.StackCount))] [MapperIgnoreTarget(nameof(ViewProject.EventCount))] + [MapperIgnoreTarget(nameof(ViewProject.EffectiveIngestLimit))] + [MapperIgnoreTarget(nameof(ViewProject.IsSmartThrottled))] + [MapperIgnoreTarget(nameof(ViewProject.SmartThrottleSampleRate))] private partial ViewProject MapToViewProjectCore(Project source); public ViewProject MapToViewProject(Project source) diff --git a/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs b/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs new file mode 100644 index 0000000000..dd1946a773 --- /dev/null +++ b/src/Exceptionless.Web/Models/Organization/UpdateOrganization.cs @@ -0,0 +1,12 @@ +using System.ComponentModel.DataAnnotations; +using Exceptionless.Core.Models; + +namespace Exceptionless.Web.Models; + +public record UpdateOrganization +{ + [Required] + public string Name { get; set; } = null!; + + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} diff --git a/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs b/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs index 90bdd3fffd..f3db6838af 100644 --- a/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs +++ b/src/Exceptionless.Web/Models/Organization/ViewOrganization.cs @@ -46,6 +46,7 @@ public record ViewOrganization : IIdentity, IData, IHaveDates public bool IsThrottled { get; set; } public bool IsOverMonthlyLimit { get; set; } public bool IsOverRequestLimit { get; set; } + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } } public static class ViewOrganizationExtensions diff --git a/src/Exceptionless.Web/Models/Project/UpdateProject.cs b/src/Exceptionless.Web/Models/Project/UpdateProject.cs index fe76403b6e..73096d0b2b 100644 --- a/src/Exceptionless.Web/Models/Project/UpdateProject.cs +++ b/src/Exceptionless.Web/Models/Project/UpdateProject.cs @@ -1,8 +1,11 @@ -namespace Exceptionless.Web.Models; +using Exceptionless.Core.Models; + +namespace Exceptionless.Web.Models; public record UpdateProject { public string Name { get; set; } = null!; public bool DeleteBotDataEnabled { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } public List? PromotedTabs { get; set; } } diff --git a/src/Exceptionless.Web/Models/Project/ViewProject.cs b/src/Exceptionless.Web/Models/Project/ViewProject.cs index 52f5576e9a..26e1204abd 100644 --- a/src/Exceptionless.Web/Models/Project/ViewProject.cs +++ b/src/Exceptionless.Web/Models/Project/ViewProject.cs @@ -23,6 +23,10 @@ public class ViewProject : IIdentity, IData, IHaveCreatedDate public long EventCount { get; set; } public bool HasPremiumFeatures { get; set; } public bool HasSlackIntegration { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } + public int? EffectiveIngestLimit { get; set; } + public bool IsSmartThrottled { get; set; } + public double? SmartThrottleSampleRate { get; set; } public ICollection UsageHours { get; set; } = new SortedSet(Comparer.Create((a, b) => a.Date.CompareTo(b.Date))); public ICollection Usage { get; set; } = new SortedSet(Comparer.Create((a, b) => a.Date.CompareTo(b.Date))); } diff --git a/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs b/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs index 34f000b90a..b2026b1d11 100644 --- a/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs +++ b/src/Exceptionless.Web/Utility/OpenApi/DeltaSchemaTransformer.cs @@ -14,18 +14,18 @@ public class DeltaSchemaTransformer : IOpenApiSchemaTransformer { private static readonly NullabilityInfoContext NullabilityContext = new(); - public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext context, CancellationToken cancellationToken) + public async Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext context, CancellationToken cancellationToken) { var type = context.JsonTypeInfo.Type; // Check if this is a Delta type if (!IsDeltaType(type)) - return Task.CompletedTask; + return; // Get the inner type T from Delta var innerType = type.GetGenericArguments().FirstOrDefault(); if (innerType is null) - return Task.CompletedTask; + return; // Set the type to object schema.Type = JsonSchemaType.Object; @@ -37,7 +37,26 @@ public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext .Where(p => p.CanRead && p.CanWrite)) { bool isNullable = IsPropertyNullable(property); - var propertySchema = CreateSchemaForType(property.PropertyType, isNullable); + var generatedSchema = await context.GetOrCreateSchemaAsync(property.PropertyType, cancellationToken: cancellationToken); + OpenApiSchema propertySchema; + + if (isNullable && RequiresNullableWrapper(property.PropertyType)) + { + propertySchema = new OpenApiSchema + { + OneOf = + [ + new OpenApiSchema { Type = JsonSchemaType.Null }, + generatedSchema + ] + }; + } + else + { + propertySchema = generatedSchema; + if (isNullable) + propertySchema.Type = propertySchema.Type.GetValueOrDefault() | JsonSchemaType.Null; + } // Apply data annotations from the inner type's property DataAnnotationHelper.ApplyToSchema(propertySchema, property); @@ -49,8 +68,6 @@ public Task TransformAsync(OpenApiSchema schema, OpenApiSchemaTransformerContext // Ensure no required array - all properties are optional for PATCH schema.Required = null; - - return Task.CompletedTask; } private static bool IsDeltaType(Type type) @@ -77,76 +94,10 @@ private static bool IsPropertyNullable(PropertyInfo property) } } - private static OpenApiSchema CreateSchemaForType(Type type, bool isNullable) + private static bool RequiresNullableWrapper(Type type) { - var schema = new OpenApiSchema(); - JsonSchemaType schemaType = default; - - // Handle nullable value types (int?, DateTime?, etc.) - var underlyingType = Nullable.GetUnderlyingType(type); - if (underlyingType is not null) - { - type = underlyingType; - isNullable = true; - } - - // Add null type if nullable - if (isNullable) - { - schemaType |= JsonSchemaType.Null; - } - - if (type == typeof(string)) - { - schemaType |= JsonSchemaType.String; - } - else if (type == typeof(bool)) - { - schemaType |= JsonSchemaType.Boolean; - } - else if (type == typeof(int) || type == typeof(long) || type == typeof(short) || type == typeof(byte)) - { - schemaType |= JsonSchemaType.Integer; - } - else if (type == typeof(double) || type == typeof(float) || type == typeof(decimal)) - { - schemaType |= JsonSchemaType.Number; - } - else if (type == typeof(DateTime) || type == typeof(DateTimeOffset)) - { - schemaType |= JsonSchemaType.String; - schema.Format = "date-time"; - } - else if (type == typeof(Guid)) - { - schemaType |= JsonSchemaType.String; - schema.Format = "uuid"; - } - else if (type.IsEnum) - { - schemaType |= JsonSchemaType.String; - } - else if (type.IsGenericType && type.GetInterfaces().Concat([type]).Any(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IDictionary<,>))) - { - schemaType |= JsonSchemaType.Object; - var valueType = type.GetGenericArguments().ElementAtOrDefault(1); - if (valueType is not null) - { - schema.AdditionalProperties = CreateSchemaForType(valueType, false); - } - } - else if (TryGetEnumerableElementType(type, out var elementType)) - { - schemaType |= JsonSchemaType.Array; - schema.Items = CreateSchemaForType(elementType, false); - } - else - { - schemaType = JsonSchemaType.Object; - } - - schema.Type = schemaType; - return schema; + type = Nullable.GetUnderlyingType(type) ?? type; + return type != typeof(string) && !type.IsValueType; } private static void ApplyArrayAnnotations(OpenApiSchema schema, PropertyInfo property) @@ -162,26 +113,4 @@ private static void ApplyArrayAnnotations(OpenApiSchema schema, PropertyInfo pro schema.MaxItems = maxLength.Length; } } - - private static bool TryGetEnumerableElementType(Type type, out Type elementType) - { - if (type.IsArray) - { - elementType = type.GetElementType() ?? typeof(object); - return true; - } - - if (type == typeof(string) || !typeof(System.Collections.IEnumerable).IsAssignableFrom(type)) - { - elementType = typeof(object); - return false; - } - - var enumerableType = type.IsGenericType && type.GetGenericTypeDefinition() == typeof(IEnumerable<>) - ? type - : type.GetInterfaces().FirstOrDefault(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IEnumerable<>)); - - elementType = enumerableType?.GetGenericArguments()[0] ?? typeof(object); - return true; - } } diff --git a/tests/Exceptionless.Tests/Controllers/ContactControllerTests.cs b/tests/Exceptionless.Tests/Controllers/ContactControllerTests.cs index 5a6445d784..166b7b612f 100644 --- a/tests/Exceptionless.Tests/Controllers/ContactControllerTests.cs +++ b/tests/Exceptionless.Tests/Controllers/ContactControllerTests.cs @@ -143,6 +143,16 @@ public Task SendOrganizationNoticeAsync(User user, Organization organization, bo return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + public Task SendOrganizationPaymentFailedAsync(User owner, Organization organization) { return Task.CompletedTask; diff --git a/tests/Exceptionless.Tests/Controllers/Data/openapi.json b/tests/Exceptionless.Tests/Controllers/Data/openapi.json index a15d8682e1..c5cbd330e9 100644 --- a/tests/Exceptionless.Tests/Controllers/Data/openapi.json +++ b/tests/Exceptionless.Tests/Controllers/Data/openapi.json @@ -673,7 +673,7 @@ "Token" ], "summary": "Create for organization", - "description": "This is a helper action that makes it easier to create a token for a specific organization.\r\nYou may also specify a scope when creating a token. There are three valid scopes: client, user and admin.", + "description": "This is a helper action that makes it easier to create a token for a specific organization.\nYou may also specify a scope when creating a token. There are three valid scopes: client, user and admin.", "parameters": [ { "name": "organizationId", @@ -797,7 +797,7 @@ "Token" ], "summary": "Create for project", - "description": "This is a helper action that makes it easier to create a token for a specific project.\r\nYou may also specify a scope when creating a token. There are three valid scopes: client, user and admin.", + "description": "This is a helper action that makes it easier to create a token for a specific project.\nYou may also specify a scope when creating a token. There are three valid scopes: client, user and admin.", "parameters": [ { "name": "projectId", @@ -1304,7 +1304,7 @@ "Auth" ], "summary": "Login", - "description": "Log in with your email address and password to generate a token scoped with your users roles.\r\n\r\n```{ \"email\": \"noreply@exceptionless.io\", \"password\": \"exceptionless\" }```\r\n\r\nThis token can then be used to access the api. You can use this token in the header (bearer authentication)\r\nor append it onto the query string: ?access_token=MY_TOKEN\r\n\r\nPlease note that you can also use this token on the documentation site by placing it in the\r\nheaders api_key input box.", + "description": "Log in with your email address and password to generate a token scoped with your users roles.\n\n```{ \"email\": \"noreply@exceptionless.io\", \"password\": \"exceptionless\" }```\n\nThis token can then be used to access the api. You can use this token in the header (bearer authentication)\nor append it onto the query string: ?access_token=MY_TOKEN\n\nPlease note that you can also use this token on the documentation site by placing it in the\nheaders api_key input box.", "requestBody": { "content": { "application/json": { @@ -2179,7 +2179,7 @@ "Event" ], "summary": "Submit event by POST", - "description": " You can create an event by posting any uncompressed or compressed (gzip or deflate) string or json object. If we know how to handle it\r\n we will create a new event. If none of the JSON properties match the event object then we will create a new event and place your JSON\r\n object into the events data collection.\r\n\r\n You can also post a multi-line string. We automatically split strings by the \\n character and create a new log event for every line.\r\n\r\n Simple event:\r\n ```{ \"message\": \"Exceptionless is amazing!\" }```\r\n\r\n Simple log event with user identity:\r\n ```{\r\n \"type\": \"log\",\r\n \"message\": \"Exceptionless is amazing!\",\r\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\r\n \"@user\":{ \"identity\":\"123456789\", \"name\": \"Test User\" }\r\n}```\r\n\r\n Multiple events from string content:\r\n ```Exceptionless is amazing!\r\nExceptionless is really amazing!```\r\n\r\n Simple error:\r\n ```{\r\n \"type\": \"error\",\r\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\r\n \"@simple_error\": {\r\n \"message\": \"Simple Exception\",\r\n \"type\": \"System.Exception\",\r\n \"stack_trace\": \" at Client.Tests.ExceptionlessClientTests.CanSubmitSimpleException() in ExceptionlessClientTests.cs:line 77\"\r\n }\r\n}```", + "description": " You can create an event by posting any uncompressed or compressed (gzip or deflate) string or json object. If we know how to handle it\n we will create a new event. If none of the JSON properties match the event object then we will create a new event and place your JSON\n object into the events data collection.\n\n You can also post a multi-line string. We automatically split strings by the \\n character and create a new log event for every line.\n\n Simple event:\n ```{ \"message\": \"Exceptionless is amazing!\" }```\n\n Simple log event with user identity:\n ```{\n \"type\": \"log\",\n \"message\": \"Exceptionless is amazing!\",\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\n \"@user\":{ \"identity\":\"123456789\", \"name\": \"Test User\" }\n}```\n\n Multiple events from string content:\n ```Exceptionless is amazing!\nExceptionless is really amazing!```\n\n Simple error:\n ```{\n \"type\": \"error\",\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\n \"@simple_error\": {\n \"message\": \"Simple Exception\",\n \"type\": \"System.Exception\",\n \"stack_trace\": \" at Client.Tests.ExceptionlessClientTests.CanSubmitSimpleException() in ExceptionlessClientTests.cs:line 77\"\n }\n}```", "parameters": [ { "name": "userAgent", @@ -2496,7 +2496,7 @@ "Event" ], "summary": "Submit event by POST for a specific project", - "description": " You can create an event by posting any uncompressed or compressed (gzip or deflate) string or json object. If we know how to handle it\r\n we will create a new event. If none of the JSON properties match the event object then we will create a new event and place your JSON\r\n object into the events data collection.\r\n\r\n You can also post a multi-line string. We automatically split strings by the \\n character and create a new log event for every line.\r\n\r\n Simple event:\r\n ```{ \"message\": \"Exceptionless is amazing!\" }```\r\n\r\n Simple log event with user identity:\r\n ```{\r\n \"type\": \"log\",\r\n \"message\": \"Exceptionless is amazing!\",\r\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\r\n \"@user\":{ \"identity\":\"123456789\", \"name\": \"Test User\" }\r\n}```\r\n\r\n Multiple events from string content:\r\n ```Exceptionless is amazing!\r\nExceptionless is really amazing!```\r\n\r\n Simple error:\r\n ```{\r\n \"type\": \"error\",\r\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\r\n \"@simple_error\": {\r\n \"message\": \"Simple Exception\",\r\n \"type\": \"System.Exception\",\r\n \"stack_trace\": \" at Client.Tests.ExceptionlessClientTests.CanSubmitSimpleException() in ExceptionlessClientTests.cs:line 77\"\r\n }\r\n}```", + "description": " You can create an event by posting any uncompressed or compressed (gzip or deflate) string or json object. If we know how to handle it\n we will create a new event. If none of the JSON properties match the event object then we will create a new event and place your JSON\n object into the events data collection.\n\n You can also post a multi-line string. We automatically split strings by the \\n character and create a new log event for every line.\n\n Simple event:\n ```{ \"message\": \"Exceptionless is amazing!\" }```\n\n Simple log event with user identity:\n ```{\n \"type\": \"log\",\n \"message\": \"Exceptionless is amazing!\",\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\n \"@user\":{ \"identity\":\"123456789\", \"name\": \"Test User\" }\n}```\n\n Multiple events from string content:\n ```Exceptionless is amazing!\nExceptionless is really amazing!```\n\n Simple error:\n ```{\n \"type\": \"error\",\n \"date\":\"2030-01-01T12:00:00.0000000-05:00\",\n \"@simple_error\": {\n \"message\": \"Simple Exception\",\n \"type\": \"System.Exception\",\n \"stack_trace\": \" at Client.Tests.ExceptionlessClientTests.CanSubmitSimpleException() in ExceptionlessClientTests.cs:line 77\"\n }\n}```", "parameters": [ { "name": "projectId", @@ -3908,7 +3908,7 @@ "Event" ], "summary": "Submit event by GET", - "description": "You can submit an event using an HTTP GET and query string parameters. Any unknown query string parameters will be added to the extended data of the event.\r\n\r\nFeature usage named build with a duration of 10:\r\n```/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\r\n\r\nLog with message, geo and extended data\r\n```/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", + "description": "You can submit an event using an HTTP GET and query string parameters. Any unknown query string parameters will be added to the extended data of the event.\n\nFeature usage named build with a duration of 10:\n```/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\n\nLog with message, geo and extended data\n```/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", "parameters": [ { "name": "type", @@ -4042,7 +4042,7 @@ "Event" ], "summary": "Submit event type by GET", - "description": "You can submit an event using an HTTP GET and query string parameters.\r\n\r\nFeature usage event named build with a value of 10:\r\n```/events/submit/usage?access_token=YOUR_API_KEY&source=build&value=10```\r\n\r\nLog event with message, geo and extended data\r\n```/events/submit/log?access_token=YOUR_API_KEY&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", + "description": "You can submit an event using an HTTP GET and query string parameters.\n\nFeature usage event named build with a value of 10:\n```/events/submit/usage?access_token=YOUR_API_KEY&source=build&value=10```\n\nLog event with message, geo and extended data\n```/events/submit/log?access_token=YOUR_API_KEY&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", "parameters": [ { "name": "type", @@ -4178,7 +4178,7 @@ "Event" ], "summary": "Submit event type by GET for a specific project", - "description": "You can submit an event using an HTTP GET and query string parameters.\r\n\r\nFeature usage named build with a duration of 10:\r\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\r\n\r\nLog with message, geo and extended data\r\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", + "description": "You can submit an event using an HTTP GET and query string parameters.\n\nFeature usage named build with a duration of 10:\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\n\nLog with message, geo and extended data\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", "parameters": [ { "name": "projectId", @@ -4314,7 +4314,7 @@ "Event" ], "summary": "Submit event type by GET for a specific project", - "description": "You can submit an event using an HTTP GET and query string parameters.\r\n\r\nFeature usage named build with a duration of 10:\r\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\r\n\r\nLog with message, geo and extended data\r\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", + "description": "You can submit an event using an HTTP GET and query string parameters.\n\nFeature usage named build with a duration of 10:\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=usage&source=build&value=10```\n\nLog with message, geo and extended data\n```/projects/{projectId}/events/submit?access_token=YOUR_API_KEY&type=log&message=Hello World&source=server01&geo=32.85,-96.9613&randomproperty=true```", "parameters": [ { "name": "projectId", @@ -5162,12 +5162,12 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } }, "application/*+json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } } }, @@ -5214,12 +5214,12 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } }, "application/*+json": { "schema": { - "$ref": "#/components/schemas/NewOrganization" + "$ref": "#/components/schemas/UpdateOrganization" } } }, @@ -5562,7 +5562,7 @@ "Organization" ], "summary": "Change plan", - "description": "Upgrades or downgrades the organization's plan.\r\nAccepts parameters via JSON body (preferred) or query string (legacy).", + "description": "Upgrades or downgrades the organization's plan.\nAccepts parameters via JSON body (preferred) or query string (legacy).", "parameters": [ { "name": "id", @@ -8924,6 +8924,16 @@ "delete_bot_data_enabled": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, "promoted_tabs": { "type": [ "null", @@ -9516,6 +9526,27 @@ } } }, + "OrganizationBudgetAlertSettings": { + "required": [ + "enabled", + "thresholds" + ], + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "thresholds": { + "uniqueItems": true, + "type": "array", + "items": { + "type": "integer", + "format": "int32" + }, + "description": "Percentage thresholds of the organization's effective monthly event allowance.\nExample: [50, 80, 90]." + } + } + }, "PersistentEvent": { "required": [ "organization_id", @@ -9581,7 +9612,7 @@ "null", "string" ], - "description": "The event type (ie. error, log message, feature usage). Check KnownTypes for standard event types.\r\nNullable in transit; the pipeline infers a default before save. Validated as required on repository save." + "description": "The event type (ie. error, log message, feature usage). Check KnownTypes for standard event types.\nNullable in transit; the pipeline infers a default before save. Validated as required on repository save." }, "source": { "maxLength": 2000, @@ -9774,6 +9805,42 @@ } } }, + "ProjectIngestLimit": { + "required": [ + "type" + ], + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/ProjectIngestLimitType" + }, + "fixed_limit": { + "type": [ + "null", + "integer" + ], + "format": "int32" + }, + "percent_of_organization_limit": { + "type": [ + "null", + "number" + ], + "format": "double" + } + } + }, + "ProjectIngestLimitType": { + "enum": [ + 0, + 1 + ], + "type": "integer", + "x-enumNames": [ + "Fixed", + "PercentOfOrganizationLimit" + ] + }, "ResetPasswordModel": { "required": [ "password_reset_token", @@ -10074,6 +10141,25 @@ }, "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." }, + "UpdateOrganization": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "budget_alert_settings": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/OrganizationBudgetAlertSettings" + } + ] + } + }, + "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." + }, "UpdateProject": { "type": "object", "properties": { @@ -10083,14 +10169,28 @@ "delete_bot_data_enabled": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, "promoted_tabs": { - "type": [ - "null", - "array" - ], - "items": { - "type": "string" - } + "oneOf": [ + { + "type": "null" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] } }, "description": "A class the tracks changes (i.e. the Delta) for a particular TEntityType." @@ -10135,23 +10235,30 @@ ] }, "columns": { - "type": [ - "null", - "object" - ], - "additionalProperties": { - "type": "boolean" - } + "oneOf": [ + { + "type": "null" + }, + { + "type": "object", + "additionalProperties": { + "type": "boolean" + } + } + ] }, "column_order": { - "maxItems": 50, - "type": [ - "null", - "array" - ], - "items": { - "type": "string" - } + "oneOf": [ + { + "type": "null" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] }, "show_stats": { "type": [ @@ -10788,6 +10895,16 @@ }, "is_over_request_limit": { "type": "boolean" + }, + "budget_alert_settings": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/OrganizationBudgetAlertSettings" + } + ] } } }, @@ -10804,6 +10921,7 @@ "event_count", "has_premium_features", "has_slack_integration", + "is_smart_throttled", "usage_hours", "usage" ], @@ -10867,6 +10985,33 @@ "has_slack_integration": { "type": "boolean" }, + "ingest_limit": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ProjectIngestLimit" + } + ] + }, + "effective_ingest_limit": { + "type": [ + "null", + "integer" + ], + "format": "int32" + }, + "is_smart_throttled": { + "type": "boolean" + }, + "smart_throttle_sample_rate": { + "type": [ + "null", + "number" + ], + "format": "double" + }, "usage_hours": { "type": "array", "items": { diff --git a/tests/Exceptionless.Tests/Controllers/OpenApiControllerTests.cs b/tests/Exceptionless.Tests/Controllers/OpenApiControllerTests.cs index 09ee908ed1..12d7089780 100644 --- a/tests/Exceptionless.Tests/Controllers/OpenApiControllerTests.cs +++ b/tests/Exceptionless.Tests/Controllers/OpenApiControllerTests.cs @@ -95,6 +95,16 @@ public async Task GetOpenApiJson_ContainsExpectedSchemasAndSecuritySchemes() Assert.Equal("access_token", token.GetProperty("name").GetString()); } + [Fact] + public async Task GetOpenApiJson_DeltaNullableComplexProperties_UseReferencedSchemas() + { + using var document = await GetOpenApiDocumentAsync(); + var schemas = document.RootElement.GetProperty("components").GetProperty("schemas"); + + AssertNullableReference(schemas, "UpdateOrganization", "budget_alert_settings", "OrganizationBudgetAlertSettings"); + AssertNullableReference(schemas, "UpdateProject", "ingest_limit", "ProjectIngestLimit"); + } + private static string NormalizeOpenApiJson(string json) { return json @@ -121,4 +131,20 @@ private static void AssertResponseCodes(JsonElement operation, params string[] e foreach (string statusCode in expectedStatusCodes) Assert.True(responses.TryGetProperty(statusCode, out _), $"Expected response status code '{statusCode}'."); } + + private static void AssertNullableReference(JsonElement schemas, string schemaName, string propertyName, string referenceName) + { + Assert.True(schemas.TryGetProperty(referenceName, out _), $"Expected referenced schema '{referenceName}'."); + + var propertySchema = schemas + .GetProperty(schemaName) + .GetProperty("properties") + .GetProperty(propertyName); + var alternatives = propertySchema.GetProperty("oneOf").EnumerateArray().ToArray(); + + Assert.Contains(alternatives, alternative => alternative.TryGetProperty("type", out var type) && type.GetString() == "null"); + Assert.Contains(alternatives, alternative => + alternative.TryGetProperty("$ref", out var reference) && + reference.GetString() == $"#/components/schemas/{referenceName}"); + } } diff --git a/tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs b/tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs index 6e99232bcc..164b9a9b80 100644 --- a/tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs +++ b/tests/Exceptionless.Tests/Controllers/OrganizationControllerTests.cs @@ -1,6 +1,7 @@ using Exceptionless.Core; using Exceptionless.Core.Billing; using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; using Exceptionless.Core.Models; using Exceptionless.Core.Models.Billing; using Exceptionless.Core.Repositories; @@ -11,6 +12,8 @@ using Exceptionless.Web.Utility; using Foundatio.Repositories; using Foundatio.Repositories.Utility; +using Foundatio.AsyncEx; +using Foundatio.Messaging; using Microsoft.Extensions.DependencyInjection; using Stripe; using Xunit; @@ -1803,6 +1806,115 @@ public async Task PatchAsync_EmptyJsonBody_ReturnsOriginalOrganizationUnchanged( Assert.Equal(originalOrganization.UpdatedUtc, updated.UpdatedUtc); } + [Fact] + public async Task PatchAsync_SameNameWithBudgetAlertSettings_DoesNotRejectAsDuplicate() + { + // Regression: If PATCH sends the org's current name alongside budget_alert_settings, + // IsOrganizationNameAvailableInternalAsync previously found the org itself and returned + // false ("already exists"), rejecting an idempotent name update. + var originalOrg = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(originalOrg); + originalOrg.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(originalOrg, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content(new { name = originalOrg.Name, budget_alert_settings = new { enabled = true, thresholds = new[] { 75 } } }) + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Equal(originalOrg.Name, updated.Name); + Assert.NotNull(updated.BudgetAlertSettings); + Assert.True(updated.BudgetAlertSettings.Enabled); + } + + [Fact] + public async Task PatchAsync_BudgetAlertSettingsNull_ClearsSettings() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = [50] }; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", organization.Id) + .Content("""{"budget_alert_settings":null}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Null(updated.BudgetAlertSettings); + } + + [Theory] + [InlineData("{\"budget_alert_settings\":{\"enabled\":false,\"thresholds\":[100]}}")] + [InlineData("{\"budget_alert_settings\":{\"enabled\":true,\"thresholds\":[]}}")] + public async Task PatchAsync_InvalidBudgetAlertSettings_ReturnsValidationError(string content) + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content(content, "application/json") + .StatusCodeShouldBeUnprocessableEntity() + ); + } + + [Fact] + public Task PatchAsync_NullBudgetAlertThresholds_ReturnsBadRequest() + { + return SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID) + .Content("""{"budget_alert_settings":{"enabled":true,"thresholds":null}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } + + [Fact] + public async Task PatchAsync_EnablingAlertAboveCurrentUsage_PublishesOnceImmediately() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + organization.BudgetAlertSettings = null; + organization.GetCurrentUsage(TimeProvider).Total = 600; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var messageBus = GetService(); + var countdown = new AsyncCountdownEvent(1); + OrganizationBudgetAlert? alert = null; + await messageBus.SubscribeAsync(message => + { + alert = message; + countdown.Signal(); + }, TestCancellationToken); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("organizations", organization.Id) + .Content("""{"budget_alert_settings":{"enabled":true,"thresholds":[50]}}""", "application/json") + .StatusCodeShouldBeOk() + ); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.NotNull(alert); + Assert.Equal(50, alert.Threshold); + Assert.Equal(600, alert.CurrentEventCount); + } + private static MultipartFormDataContent CreateProfileImageContent() { byte[] bytes = new byte[256 * 1024]; diff --git a/tests/Exceptionless.Tests/Controllers/ProjectControllerTests.cs b/tests/Exceptionless.Tests/Controllers/ProjectControllerTests.cs index d8843620cd..340a42c6d9 100644 --- a/tests/Exceptionless.Tests/Controllers/ProjectControllerTests.cs +++ b/tests/Exceptionless.Tests/Controllers/ProjectControllerTests.cs @@ -1348,4 +1348,147 @@ public async Task CanGetProjectListStats() Assert.Equal(newStacks.Count, project.StackCount); Assert.Equal(newEvents.Count, project.EventCount); } + + [Fact] + public async Task PatchAsync_WithFixedIngestLimit_SavesCorrectly() + { + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":500}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(ProjectIngestLimitType.Fixed, updated.IngestLimit.Type); + Assert.Equal(500, updated.IngestLimit.FixedLimit); + + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + Assert.NotNull(project.IngestLimit); + Assert.Equal(ProjectIngestLimitType.Fixed, project.IngestLimit.Type); + Assert.Equal(500, project.IngestLimit.FixedLimit); + } + + [Fact] + public async Task PatchAsync_WithPercentageIngestLimit_SavesCorrectly() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":1,"percent_of_organization_limit":75.0}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(ProjectIngestLimitType.PercentOfOrganizationLimit, updated.IngestLimit.Type); + Assert.Equal(75.0m, updated.IngestLimit.PercentOfOrganizationLimit); + + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + Assert.NotNull(project.IngestLimit); + Assert.Equal(75.0m, project.IngestLimit.PercentOfOrganizationLimit); + } + + [Fact] + public async Task PatchAsync_RemoveIngestLimit_SavesNull() + { + var project = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(project); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 }; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":null}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Null(updated.IngestLimit); + + var saved = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(saved); + Assert.Null(saved.IngestLimit); + } + + [Fact] + public async Task PatchAsync_OnlyIngestLimit_NoNameValidationError() + { + var original = await _projectRepository.GetByIdAsync(SampleDataService.TEST_PROJECT_ID); + Assert.NotNull(original); + + var updated = await SendRequestAsAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":200}}""", "application/json") + .StatusCodeShouldBeOk() + ); + + Assert.NotNull(updated); + Assert.Equal(original.Name, updated.Name); + Assert.NotNull(updated.IngestLimit); + Assert.Equal(200, updated.IngestLimit.FixedLimit); + } + + [Theory] + [InlineData("{\"ingest_limit\":{\"type\":0,\"fixed_limit\":0}}")] + [InlineData("{\"ingest_limit\":{\"type\":1,\"percent_of_organization_limit\":0}}")] + [InlineData("{\"ingest_limit\":{\"type\":1,\"percent_of_organization_limit\":100.1}}")] + public async Task PatchAsync_InvalidIngestLimit_ReturnsValidationError(string content) + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content(content, "application/json") + .StatusCodeShouldBeUnprocessableEntity() + ); + } + + [Fact] + public Task PatchAsync_FractionalFixedIngestLimit_ReturnsBadRequest() + { + return SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":0,"fixed_limit":1.5}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } + + [Fact] + public async Task PatchAsync_PercentageIngestLimitForUnlimitedOrganization_ReturnsValidationError() + { + var organization = await _organizationRepository.GetByIdAsync(SampleDataService.TEST_ORG_ID); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = -1; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + await SendRequestAsync(r => r + .Patch() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID) + .Content("""{"ingest_limit":{"type":1,"percent_of_organization_limit":50}}""", "application/json") + .StatusCodeShouldBeBadRequest() + ); + } } diff --git a/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs b/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs index fb07d885e1..5a86695711 100644 --- a/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs +++ b/tests/Exceptionless.Tests/Jobs/EventPostJobTests.cs @@ -268,4 +268,73 @@ private PersistentEvent GenerateEvent(DateTimeOffset? occurrenceDate = null, str occurrenceDate ??= DateTimeOffset.Now; return _eventData.GenerateEvent(projectId: TestConstants.ProjectId, organizationId: TestConstants.OrganizationId, generateTags: false, generateData: false, occurrenceDate: occurrenceDate, userIdentity: userIdentity, type: type, source: source, sessionId: sessionId); } + + [Fact] + public async Task CanRunJob_WhenProjectOverLimit_BlocksEvents() + { + var project = await _projectRepository.GetByIdAsync(TestConstants.ProjectId); + Assert.NotNull(project); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 1 }; + project.GetCurrentUsage(TimeProvider).Total = 1; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + Assert.NotNull(await EnqueueEventPostAsync(GenerateEvent())); + Assert.Equal(1, (await _eventQueue.GetQueueStatsAsync()).Enqueued); + + var result = await _job.RunAsync(TestCancellationToken); + Assert.True(result.IsSuccess); + + await RefreshDataAsync(); + Assert.Equal(0, await _eventRepository.CountAsync()); + + var usage = await _usageService.GetUsageAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.Equal(1, usage.CurrentUsage.Blocked); + } + + [Fact] + public async Task CanRunJob_WhenSmartThrottleApplied_BlocksEventsOnce() + { + var organization = await _organizationRepository.GetByIdAsync(TestConstants.OrganizationId); + Assert.NotNull(organization); + organization.MaxEventsPerMonth = 1_000_000; + await _organizationRepository.SaveAsync(organization, o => o.ImmediateConsistency().Cache()); + + var project = await _projectRepository.GetByIdAsync(TestConstants.ProjectId); + Assert.NotNull(project); + await _projectRepository.AddAsync(_projectData.GenerateProject(generateId: true, organizationId: organization.Id), o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(_projectData.GenerateProject(generateId: true, organizationId: organization.Id), o => o.ImmediateConsistency().Cache()); + var utcNow = TimeProvider.GetUtcNow().UtcDateTime; + var endOfMonth = new DateTime(utcNow.Year, utcNow.Month, 1, 0, 0, 0, DateTimeKind.Utc).AddMonths(1); + double windowsLeft = Math.Max(1, Math.Ceiling((endOfMonth - utcNow).TotalMinutes / 5)); + int currentWindowSpike = (int)Math.Floor(organization.MaxEventsPerMonth / windowsLeft * 10 * 0.9); + await _usageService.IncrementTotalAsync(organization, project.Id, currentWindowSpike); + + var throttleResult = await _usageService.GetSmartThrottleRateAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.True(throttleResult.IsThrottled); + Assert.Equal(0.05, throttleResult.SampleRate); + + var events = Enumerable.Range(0, 100).Select(index => GenerateEvent(source: $"source-{index}")).ToList(); + Assert.NotNull(await EnqueueEventPostAsync(events)); + + var result = await _job.RunAsync(TestCancellationToken); + Assert.True(result.IsSuccess); + + var usage = await _usageService.GetUsageAsync(TestConstants.OrganizationId, TestConstants.ProjectId); + Assert.True(usage.CurrentUsage.Blocked > 0); + Assert.Equal(0, usage.CurrentUsage.Discarded); + + await RefreshDataAsync(); + Assert.InRange(await _eventRepository.CountAsync(), 1, 20); + } + + [Fact] + public void SmartThrottleSelection_SingleEventPosts_UsesStableFivePercentSample() + { + var persistentEvent = GenerateEvent(); + + int accepted = Enumerable.Range(0, 1_000) + .Count(index => EventPostsJob.IsSelectedForSmartThrottle($"post-{index}", persistentEvent, 0)); + + Assert.InRange(accepted, 30, 70); + } } diff --git a/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs b/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs new file mode 100644 index 0000000000..97bd7da6bc --- /dev/null +++ b/tests/Exceptionless.Tests/Jobs/WorkItemHandlers/UsageBudgetNotificationWorkItemHandlerTests.cs @@ -0,0 +1,165 @@ +using Exceptionless.Core.Billing; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Jobs.WorkItemHandlers; +using Exceptionless.Core.Mail; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.Tests.Mail; +using Exceptionless.Tests.Extensions; +using Exceptionless.Tests.Utility; +using Foundatio.Jobs; +using Foundatio.Repositories; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Exceptionless.Tests.Jobs.WorkItemHandlers; + +public class UsageBudgetNotificationWorkItemHandlerTests : IntegrationTestsBase +{ + private const string OrganizationId = "664ec4c1f12e4f2b7a0d3101"; + private const string ProjectId = "664ec4c1f12e4f2b7a0d3201"; + private const string UserId = "664ec4c1f12e4f2b7a0d3301"; + + public UsageBudgetNotificationWorkItemHandlerTests(ITestOutputHelper output, AppWebHostFactory factory) : base(output, factory) { } + + private CountingMailer Mailer => GetService(); + private IOrganizationRepository OrganizationRepository => GetService(); + private IProjectRepository ProjectRepository => GetService(); + private IUserRepository UserRepository => GetService(); + private UsageService UsageService => GetService(); + + protected override void RegisterServices(IServiceCollection services) + { + base.RegisterServices(services); + services.AddSingleton(); + services.ReplaceSingleton(serviceProvider => serviceProvider.GetRequiredService()); + } + + protected override async Task ResetDataAsync() + { + await base.ResetDataAsync(); + Mailer.Reset(); + + var billingManager = GetService(); + var billingPlans = GetService(); + var organization = GetService().GenerateOrganization(billingManager, billingPlans, id: OrganizationId, name: "Budget Organization", plan: billingPlans.ExtraLargePlan); + organization.MaxEventsPerMonth = 1_000_000; + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = [50] }; + await OrganizationRepository.AddAsync(organization, options => options.ImmediateConsistency().Cache()); + + var projectData = GetService(); + await ProjectRepository.AddAsync([ + projectData.GenerateProject(id: ProjectId, organizationId: OrganizationId, name: "Noisy Project"), + projectData.GenerateProject(generateId: true, organizationId: OrganizationId, name: "Quiet Project"), + projectData.GenerateProject(generateId: true, organizationId: OrganizationId, name: "Another Project") + ], options => options.ImmediateConsistency().Cache()); + + var user = GetService().GenerateUser(id: UserId, organizationId: OrganizationId, emailAddress: "budget-owner@example.org"); + user.IsEmailAddressVerified = true; + user.EmailNotificationsEnabled = true; + await UserRepository.AddAsync(user, options => options.ImmediateConsistency().Cache()); + } + + [Fact] + public async Task OrganizationHandler_CurrentCrossing_SendsCurrentUsageAndPlan() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + Assert.NotNull(organization); + organization.GetCurrentUsage(TimeProvider).Total = 600_000; + await OrganizationRepository.SaveAsync(organization, options => options.ImmediateConsistency().Cache()); + + await HandleAsync(GetService(), new OrganizationBudgetAlertWorkItem + { + OrganizationId = OrganizationId, + Threshold = 50, + ThresholdEventCount = 1, + CurrentEventCount = 1, + EventLimit = 1 + }); + + var call = Assert.Single(Mailer.OrganizationBudgetAlertCalls); + Assert.Equal(500_000, call.ThresholdEventCount); + Assert.Equal(600_000, call.CurrentEventCount); + Assert.Equal(1_000_000, call.EventLimit); + } + + [Fact] + public async Task OrganizationHandler_DisabledAfterQueue_SuppressesStaleEmail() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + Assert.NotNull(organization); + organization.BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = false, Thresholds = [50] }; + await OrganizationRepository.SaveAsync(organization, options => options.ImmediateConsistency().Cache()); + + await HandleAsync(GetService(), new OrganizationBudgetAlertWorkItem + { + OrganizationId = OrganizationId, + Threshold = 50, + ThresholdEventCount = 500_000, + CurrentEventCount = 600_000, + EventLimit = 1_000_000 + }); + + Assert.Empty(Mailer.OrganizationBudgetAlertCalls); + } + + [Fact] + public async Task ProjectHandler_CurrentThrottle_SendsProjectUsageAndFairShare() + { + var organization = await OrganizationRepository.GetByIdAsync(OrganizationId); + var project = await ProjectRepository.GetByIdAsync(ProjectId); + Assert.NotNull(organization); + Assert.NotNull(project); + + int spike = GetCurrentWindowSpike(organization.MaxEventsPerMonth); + await UsageService.IncrementTotalAsync(organization, project.Id, spike); + Assert.True((await UsageService.GetEventIngestAllowanceAsync(organization, project)).SmartThrottle.IsThrottled); + + await HandleAsync(GetService(), new ProjectSmartThrottleWorkItem + { + OrganizationId = OrganizationId, + ProjectId = ProjectId, + SampleRate = 0.05, + CurrentEventCount = 1, + EventLimit = 1 + }); + + var call = Assert.Single(Mailer.ProjectThrottleCalls); + Assert.Equal(spike, call.CurrentEventCount); + Assert.Equal(333_333, call.EventLimit); + Assert.Equal(0.05, call.SampleRate); + } + + [Fact] + public async Task ProjectHandler_ThrottleExpiredBeforeHandling_SuppressesStaleEmail() + { + await HandleAsync(GetService(), new ProjectSmartThrottleWorkItem + { + OrganizationId = OrganizationId, + ProjectId = ProjectId, + SampleRate = 0.05, + CurrentEventCount = 1, + EventLimit = 1 + }); + + Assert.Empty(Mailer.ProjectThrottleCalls); + } + + private int GetCurrentWindowSpike(int maxEventsPerMonth) + { + var utcNow = TimeProvider.GetUtcNow().UtcDateTime; + var endOfMonth = new DateTime(utcNow.Year, utcNow.Month, 1, 0, 0, 0, DateTimeKind.Utc).AddMonths(1); + double windowsLeft = Math.Max(1, Math.Ceiling((endOfMonth - utcNow).TotalMinutes / 5)); + return (int)Math.Floor(maxEventsPerMonth / windowsLeft * 10 * 0.9); + } + + private static async Task HandleAsync(WorkItemHandlerBase handler, object workItem) + { + await using var workItemLock = await handler.GetWorkItemLockAsync(workItem, TestContext.Current.CancellationToken); + Assert.NotNull(workItemLock); + var context = new WorkItemContext(workItem, "test-job", workItemLock, TestContext.Current.CancellationToken, static (_, _) => Task.CompletedTask); + await handler.HandleItemAsync(context); + } +} diff --git a/tests/Exceptionless.Tests/Mail/CountingMailer.cs b/tests/Exceptionless.Tests/Mail/CountingMailer.cs index 2967a84886..97a088108e 100644 --- a/tests/Exceptionless.Tests/Mail/CountingMailer.cs +++ b/tests/Exceptionless.Tests/Mail/CountingMailer.cs @@ -10,6 +10,8 @@ public class CountingMailer : IMailer public int OrganizationNoticeCount => _organizationNoticeCount; public List OrganizationNoticeCalls { get; } = []; + public List OrganizationBudgetAlertCalls { get; } = []; + public List ProjectThrottleCalls { get; } = []; /// /// When true, throws instead of recording a call. @@ -55,6 +57,24 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + lock (OrganizationBudgetAlertCalls) + { + OrganizationBudgetAlertCalls.Add(new OrganizationBudgetAlertCall(user.Id, organization.Id, threshold, thresholdEventCount, currentEventCount, eventLimit)); + } + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + lock (ProjectThrottleCalls) + { + ProjectThrottleCalls.Add(new ProjectThrottleCall(user.Id, organization.Id, project.Id, sampleRate, currentEventCount, eventLimit)); + } + return Task.CompletedTask; + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { return Task.CompletedTask; @@ -77,8 +97,18 @@ public void Reset() { OrganizationNoticeCalls.Clear(); } + lock (OrganizationBudgetAlertCalls) + { + OrganizationBudgetAlertCalls.Clear(); + } + lock (ProjectThrottleCalls) + { + ProjectThrottleCalls.Clear(); + } ShouldThrow = false; } } public record OrganizationNoticeCall(string UserId, string OrganizationId, bool IsOverMonthlyLimit, bool IsOverHourlyLimit); +public record OrganizationBudgetAlertCall(string UserId, string OrganizationId, int Threshold, int ThresholdEventCount, int CurrentEventCount, int EventLimit); +public record ProjectThrottleCall(string UserId, string OrganizationId, string ProjectId, double SampleRate, int CurrentEventCount, int EventLimit); diff --git a/tests/Exceptionless.Tests/Mail/NullMailer.cs b/tests/Exceptionless.Tests/Mail/NullMailer.cs index 7ce54fcde6..86b854c649 100644 --- a/tests/Exceptionless.Tests/Mail/NullMailer.cs +++ b/tests/Exceptionless.Tests/Mail/NullMailer.cs @@ -35,6 +35,16 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza return Task.CompletedTask; } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + return Task.CompletedTask; + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { return Task.CompletedTask; diff --git a/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs b/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs new file mode 100644 index 0000000000..3ba07c5e03 --- /dev/null +++ b/tests/Exceptionless.Tests/Services/UsageBudgetServiceTests.cs @@ -0,0 +1,514 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using Exceptionless.Core.Billing; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; +using Exceptionless.Tests.Extensions; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Repositories; +using Xunit; +using LogLevel = Microsoft.Extensions.Logging.LogLevel; + +namespace Exceptionless.Tests.Services; + +public sealed partial class UsageServiceTests +{ + [Fact] + public async Task GetSmartThrottleRateAsync_OrganizationNotFound_ReturnsNoThrottle() + { + // Regression test: When the organization does not exist, GetMaxEventsPerMonthAsync + // returns 0 (default). GetSmartThrottleRateAsync must not divide by zero. + string nonExistentOrgId = "000000000000000000000099"; + string nonExistentProjectId = "000000000000000000000098"; + + // Act - this would divide by zero before the fix (maxEventsPerMonth=0 returned for missing org) + var result = await _usageService.GetSmartThrottleRateAsync(nonExistentOrgId, nonExistentProjectId); + + // Assert - should return NoThrottle since maxEventsPerMonth <= 0 means unlimited/invalid + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + // ── GetEventIngestAllowanceAsync ──────────────────────────────────────── + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithNoIngestLimit_ReturnsOrgLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.True(result.EventsLeft > 0); + Assert.Equal(1.0, result.SampleRate); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedIngestLimit_BelowOrgLimit_ReturnsFixed() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(100, result.EventsLeft); + Assert.Equal(100, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedIngestLimit_AboveOrgLimit_ReturnsOrgLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 10000 } + }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 749); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(1, result.EventsLeft); + Assert.Equal(750, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithPercentageIngestLimit_50Percent_ReturnsCorrectLimit() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 50 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(500, result.EventsLeft); + Assert.Equal(500, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithInvalidPersistedPercentage_IsInactive() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks + }, o => o.ImmediateConsistency().Cache()); + project.IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 150 }; + + var result = await _usageService.GetEventIngestAllowanceAsync(organization, project); + + Assert.Equal(-1, result.EffectiveProjectLimit); + Assert.True(result.EventsLeft > 0); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Project_WithFixedLimit_OverLimit_IsOverProjectLimitTrue() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 100 } + }, o => o.ImmediateConsistency().Cache()); + project.GetCurrentUsage(TimeProvider).Total = 100; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(0, result.EventsLeft); + Assert.True(result.IsOverProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.Equal(100, result.EffectiveProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_Organization_OverLimit_IsOverOrgLimitTrue() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 750, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 750); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(0, result.EventsLeft); + Assert.True(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_UnlimitedOrg_WithFixedProjectLimit_ReturnsFixed() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = _plans.UnlimitedPlan.MaxEventsPerMonth, PlanId = _plans.UnlimitedPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.Fixed, FixedLimit = 500 } + }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + + Assert.Equal(500, result.EventsLeft); + Assert.Equal(500, result.EffectiveProjectLimit); + Assert.False(result.IsOverOrgLimit); + Assert.False(result.IsOverProjectLimit); + } + + // ── GetSmartThrottleRateAsync ──────────────────────────────────────────── + + [Fact] + public async Task GetSmartThrottleRateAsync_BelowThreshold_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 600; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + project1.GetCurrentUsage(TimeProvider).Total = 600; + await _projectRepository.SaveAsync(project1, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_SingleProjectInOrg_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + project.GetCurrentUsage(TimeProvider).Total = 900; + await _projectRepository.SaveAsync(project, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_NoOrgUsage_ReturnsNoThrottle() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_NoProjectUsage_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_AboveThreshold_FairShare_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1000, PlanId = _plans.SmallPlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + var project2 = await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + // fairShare = 1000/2 = 500; projectTotal = 900 → fairShareRatio = 1.8 ≤ 2.0 + project1.GetCurrentUsage(TimeProvider).Total = 900; + await _projectRepository.SaveAsync(project1, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + _ = project2; + } + + [Fact] + public async Task GetSmartThrottleRateAsync_CurrentWindowSpike_UsesFixedFivePercentSample() + { + var organization = await _organizationRepository.AddAsync(new Organization + { + Name = "Test", + MaxEventsPerMonth = 1_000_000, + PlanId = _plans.ExtraLargePlan.Id + }, o => o.ImmediateConsistency().Cache()); + var project1 = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P3", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.IncrementTotalAsync(organization, project1.Id, 1_900); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.True(result.IsThrottled); + Assert.Equal(0.05, result.SampleRate); + Assert.Equal(1_900, result.CurrentProjectUsage); + Assert.Equal(333_333, result.FairShareLimit); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_HighMonthlyUsageWithoutCurrentSpike_ReturnsNoThrottle() + { + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 1_000_000, PlanId = _plans.ExtraLargePlan.Id }; + organization.GetCurrentUsage(TimeProvider).Total = 900_000; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project1 = new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }; + project1.GetCurrentUsage(TimeProvider).Total = 900_000; + project1 = await _projectRepository.AddAsync(project1, o => o.ImmediateConsistency().Cache()); + await _projectRepository.AddAsync(new Project { Name = "P2", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project1.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + [Fact] + public async Task GetSmartThrottleRateAsync_UnlimitedOrg_ReturnsNoThrottle() + { + var organization = await _organizationRepository.AddAsync(new Organization { Name = "Test", MaxEventsPerMonth = _plans.UnlimitedPlan.MaxEventsPerMonth, PlanId = _plans.UnlimitedPlan.Id }, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "P1", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + var result = await _usageService.GetSmartThrottleRateAsync(organization.Id, project.Id); + + Assert.False(result.IsThrottled); + Assert.Equal(1.0, result.SampleRate); + } + + // ── Budget Alert Threshold tests ───────────────────────────────────────── + + [Fact] + public async Task IncrementTotalAsync_CrossingThreshold_PublishesAlertMessage() + { + var messageBus = GetService(); + var countdown = new AsyncCountdownEvent(1); + OrganizationBudgetAlert? alert = null; + await messageBus.SubscribeAsync(a => + { + alert = a; + countdown.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 15); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.NotNull(alert); + Assert.Equal(organization.Id, alert.OrganizationId); + Assert.Equal(50, alert.Threshold); + Assert.Equal(500, alert.ThresholdEventCount); + Assert.Equal(1000, alert.EventLimit); + } + + [Fact] + public async Task IncrementTotalAsync_Threshold_NotDuplicated_WhenCrossedTwice() + { + var messageBus = GetService(); + int alertCount = 0; + var firstAlert = new AsyncCountdownEvent(1); + await messageBus.SubscribeAsync(a => + { + alertCount++; + if (alertCount == 1) firstAlert.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 15); + await firstAlert.WaitAsync(TimeSpan.FromSeconds(5)); + + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 20); + await Task.Delay(200, TestCancellationToken); + + Assert.Equal(1, alertCount); + } + + [Fact] + public async Task CheckBudgetAlertThresholdsAsync_ConcurrentCalls_PublishesAlertExactlyOnce() + { + // Regression: non-atomic get-then-set allowed two concurrent workers to both see the + // key absent and both publish. The fix uses AddAsync (atomic) so only the worker + // whose increment returns 1 sends the alert. + var messageBus = GetService(); + int alertCount = 0; + var firstAlert = new AsyncCountdownEvent(1); + await messageBus.SubscribeAsync(_ => + { + alertCount++; + if (alertCount == 1) firstAlert.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + // Simulate concurrent workers by calling GetEventsLeftAsync + IncrementTotalAsync + // concurrently. Both workers see usage crossing 50%; only one should publish the alert. + await _usageService.GetEventsLeftAsync(organization.Id); + var tasks = Enumerable.Range(0, 5) + .Select(_ => _usageService.IncrementTotalAsync(organization.Id, project.Id, 15)) + .ToArray(); + await Task.WhenAll(tasks); + + // Wait for the first (and only) alert, then give extra time for any duplicates. + await firstAlert.WaitAsync(TimeSpan.FromSeconds(5)); + await Task.Delay(300, TestCancellationToken); + + Assert.Equal(1, alertCount); + } + + [Fact] + public async Task IncrementTotalAsync_DisabledAlerts_DoNotPublishMessages() + { + var messageBus = GetService(); + int alertCount = 0; + await messageBus.SubscribeAsync(_ => { alertCount++; }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = false, Thresholds = new SortedSet { 50 } } + }; + organization.GetCurrentUsage(TimeProvider).Total = 490; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 100); + + await Task.Delay(200, TestCancellationToken); + Assert.Equal(0, alertCount); + } + + [Fact] + public async Task IncrementTotalAsync_AllThresholds_AllFireOnce() + { + var messageBus = GetService(); + var firedThresholds = new System.Collections.Concurrent.ConcurrentBag(); + var countdown = new AsyncCountdownEvent(3); + await messageBus.SubscribeAsync(a => + { + firedThresholds.Add(a.Threshold); + countdown.Signal(); + }, TestCancellationToken); + + var organization = new Organization + { + Name = "Test", + MaxEventsPerMonth = 1000, + PlanId = _plans.SmallPlan.Id, + BudgetAlertSettings = new OrganizationBudgetAlertSettings { Enabled = true, Thresholds = new SortedSet { 50, 80, 90 } } + }; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project { Name = "Test", OrganizationId = organization.Id, NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks }, o => o.ImmediateConsistency().Cache()); + + await _usageService.GetEventsLeftAsync(organization.Id); + await _usageService.IncrementTotalAsync(organization.Id, project.Id, 950); + await countdown.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.Equal(3, firedThresholds.Count); + Assert.Contains(50, firedThresholds); + Assert.Contains(80, firedThresholds); + Assert.Contains(90, firedThresholds); + } + + [Fact] + public async Task GetEventIngestAllowanceAsync_SmallPercentageLimit_UsesRoundUp_NotZero() + { + // Regression: 1% of 50 events floored to 0 with (int)cast, making the project + // permanently over-budget on the first event. Math.Ceiling must be used. + var organization = new Organization { Name = "Test", MaxEventsPerMonth = 50, PlanId = _plans.SmallPlan.Id }; + organization = await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency().Cache()); + var project = await _projectRepository.AddAsync(new Project + { + Name = "Test", + OrganizationId = organization.Id, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks, + IngestLimit = new ProjectIngestLimit { Type = ProjectIngestLimitType.PercentOfOrganizationLimit, PercentOfOrganizationLimit = 1 } + }, o => o.ImmediateConsistency().Cache()); + + // 1% of 50 = 0.5 → must ceil to 1, not floor to 0 + var result = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + Assert.True(result.EffectiveProjectLimit >= 1, $"EffectiveProjectLimit was {result.EffectiveProjectLimit}, expected >= 1 (floor bug would produce 0)"); + } +} diff --git a/tests/Exceptionless.Tests/Services/UsageServiceTests.cs b/tests/Exceptionless.Tests/Services/UsageServiceTests.cs index 8a54c6cbd7..4be2f70b97 100644 --- a/tests/Exceptionless.Tests/Services/UsageServiceTests.cs +++ b/tests/Exceptionless.Tests/Services/UsageServiceTests.cs @@ -1,3 +1,4 @@ +using System.Collections.Concurrent; using System.Diagnostics; using Exceptionless.Core.Billing; using Exceptionless.Core.Extensions; @@ -14,7 +15,7 @@ namespace Exceptionless.Tests.Services; -public sealed class UsageServiceTests : IntegrationTestsBase +public sealed partial class UsageServiceTests : IntegrationTestsBase { private readonly IOrganizationRepository _organizationRepository; private readonly IProjectRepository _projectRepository; @@ -655,4 +656,5 @@ public async Task RunBenchmarkAsync() sw.Stop(); _logger.LogInformation("Time: {Duration:g}, Avg: ({AverageTickDuration:g}ticks | {AverageDuration}ms)", sw.Elapsed, sw.ElapsedTicks / iterations, sw.ElapsedMilliseconds / iterations); } + } diff --git a/tests/http/organizations.http b/tests/http/organizations.http index 09119dbf67..22dc921876 100644 --- a/tests/http/organizations.http +++ b/tests/http/organizations.http @@ -71,6 +71,20 @@ Content-Type: application/json { "name": "My updated organization" } +### Enable Budget Alerts +PATCH {{apiUrl}}/organizations/{{organizationId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "budget_alert_settings": { "enabled": true, "thresholds": [50, 80, 90] } } + +### Clear Budget Alert Settings +PATCH {{apiUrl}}/organizations/{{organizationId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "budget_alert_settings": null } + ### Upload Icon # Replace {{iconFile}} with a local PNG, JPEG, GIF, or WebP image. POST {{apiUrl}}/organizations/{{organizationId}}/icon diff --git a/tests/http/projects.http b/tests/http/projects.http index 45bbcbf6df..36f523dcd7 100644 --- a/tests/http/projects.http +++ b/tests/http/projects.http @@ -153,6 +153,27 @@ Content-Type: application/json { "Name": "My Updated Project" } +### Set Fixed Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": { "type": 0, "fixed_limit": 20000 } } + +### Set Percentage Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": { "type": 1, "percent_of_organization_limit": 50 } } + +### Clear Project Event Budget +PATCH {{apiUrl}}/projects/{{projectId}} +Authorization: Bearer {{token}} +Content-Type: application/json + +{ "ingest_limit": null } + ### Delete DELETE {{apiUrl}}/projects/{{projectId}} Authorization: Bearer {{token}}