diff --git a/docs/docs/source-maps.md b/docs/docs/source-maps.md new file mode 100644 index 0000000000..1aa16515e6 --- /dev/null +++ b/docs/docs/source-maps.md @@ -0,0 +1,57 @@ +--- +title: "JavaScript Source Maps" +--- + +# JavaScript Source Maps + +Exceptionless uses source maps to turn minified JavaScript stack frames into the original file names, line and column numbers, and function names. Symbolication happens before an event is assigned to a stack, so readable function names also improve stack grouping. + +## Automatic discovery + +No domain allowlist or project setup is required for public source maps. When an error contains an absolute HTTPS JavaScript URL, Exceptionless checks the generated file's `SourceMap` or `X-SourceMap` response header and its `sourceMappingURL` comment. If neither is present, it also checks the conventional `.map` URL. + +Downloaded maps are validated and cached in project-scoped file storage. Automatically downloaded maps are revalidated after one hour so a stable generated-file URL cannot retain a map from an older deployment indefinitely. If refresh fails, Exceptionless leaves the generated frame unchanged instead of risking a misleading stack trace from the stale map. + +Exceptionless only makes anonymous HTTPS requests on the standard port to public network addresses. Redirects and every resolved address are revalidated. New automatic discoveries use plan-aware limits per client key, project, and organization; outbound requests are also limited per destination, IP address, and cluster. Free plans allow five new discoveries per client key and ten per project or organization in each 15-minute window by default. Paid plans have higher limits. Failed URLs are cached for 15 minutes, duplicate discoveries share the same in-flight work, and refreshes use a separate smaller outbound budget. If any limit is reached, Exceptionless leaves the generated frame unchanged without rejecting or delaying the event. + +Downloads also have time, redirect, size, and local and cluster-wide concurrency limits. Parsed maps use a bounded in-memory cache. Manual and deployment uploads do not consume automatic-discovery quotas. Self-hosted installations can tune these safeguards under the `SourceMaps` configuration section, including `AutoDownloadRateLimitPeriodMinutes`, `MaximumAutoDiscoveriesPerFreeClientKey`, `MaximumAutoDiscoveriesPerProject`, `MaximumAutoDownloadRequestsPerDestination`, `MaximumAutoRefreshRequestsGlobally`, `AutoDownloadRefreshIntervalMinutes`, `ParsedSourceMapCacheLifetimeMinutes`, and `MaximumParsedSourceMapCacheSize`. + +## Uploading a source map + +Upload a map when it is private or is not deployed next to the generated JavaScript: + +1. Open the project and select **Source Maps** under **Project Settings**. +2. Enter the exact absolute URL that appears in the generated stack frame, including any path or query string used to identify the build. +3. Select the corresponding source map and upload it. + +Uploading another map for the same generated file URL replaces the previous map. Uploaded and automatically discovered maps appear together on the Source Maps page and can be deleted there. + +### Uploading during deployment + +For build automation, create a project-scoped token that has only the `source-maps:write` scope. Set `EXCEPTIONLESS_SERVER_URL` to `https://be.exceptionless.io` for the hosted service or to the root URL of your self-hosted installation. Create the token once with a user-scoped token, then store the returned `id` as a protected CI/CD secret: + +```shell +curl --fail-with-body --request POST \ + "${EXCEPTIONLESS_SERVER_URL}/api/v2/projects/${EXCEPTIONLESS_PROJECT_ID}/tokens" \ + --header "Authorization: Bearer ${EXCEPTIONLESS_USER_TOKEN}" \ + --header "Content-Type: application/json" \ + --data "{\"organization_id\":\"${EXCEPTIONLESS_ORGANIZATION_ID}\",\"project_id\":\"${EXCEPTIONLESS_PROJECT_ID}\",\"scopes\":[\"source-maps:write\"],\"notes\":\"CI source map uploads\"}" +``` + +Upload each map after the generated JavaScript has been deployed. The `generated_file_url` must be the exact absolute URL that will appear in stack frames: + +```shell +curl --fail-with-body --request POST \ + "${EXCEPTIONLESS_SERVER_URL}/api/v2/projects/${EXCEPTIONLESS_PROJECT_ID}/source-maps" \ + --header "Authorization: Bearer ${EXCEPTIONLESS_SOURCE_MAP_TOKEN}" \ + --form "generated_file_url=https://cdn.example.com/assets/app.a1b2c3.js" \ + --form "file=@dist/assets/app.a1b2c3.js.map;type=application/json" +``` + +The upload token is accepted only for its assigned project and cannot read events, manage the project, list source maps, or use ordinary user APIs. Do not use a normal client API key for source map uploads because client keys are commonly embedded in distributed applications. + +Source maps must use the version 3 flat-map format. Indexed source maps with a `sections` property and authenticated automatic downloads are planned follow-up capabilities; private maps can be uploaded in the meantime. + +## Deployment guidance + +Generate source maps as part of the same build that produces the minified JavaScript. Content-hashed generated file names are preferred because the generated URL then identifies a specific build. You can publish the `.map` file for zero-configuration discovery or keep it private and upload it to Exceptionless during deployment. diff --git a/src/Exceptionless.Core/Authorization/AuthorizationRoles.cs b/src/Exceptionless.Core/Authorization/AuthorizationRoles.cs index 546b06e864..0ac1565c2e 100644 --- a/src/Exceptionless.Core/Authorization/AuthorizationRoles.cs +++ b/src/Exceptionless.Core/Authorization/AuthorizationRoles.cs @@ -12,12 +12,14 @@ public static class AuthorizationRoles public const string ProjectsReadPolicy = nameof(ProjectsReadPolicy); public const string StacksReadPolicy = nameof(StacksReadPolicy); public const string StacksWritePolicy = nameof(StacksWritePolicy); + public const string SourceMapsWritePolicy = nameof(SourceMapsWritePolicy); public const string EventsReadPolicy = nameof(EventsReadPolicy); public const string McpRead = "mcp:read"; public const string ProjectsRead = "projects:read"; public const string StacksRead = "stacks:read"; public const string StacksWrite = "stacks:write"; + public const string SourceMapsWrite = "source-maps:write"; public const string EventsRead = "events:read"; public const string OfflineAccess = "offline_access"; - public static readonly ISet AllScopes = new HashSet([Client, User, GlobalAdmin]); + public static readonly ISet AllScopes = new HashSet([Client, User, GlobalAdmin, SourceMapsWrite]); } diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index efcfae9efb..701ca2128c 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -25,6 +25,7 @@ using Exceptionless.Core.Seed; using Exceptionless.Core.Serialization; using Exceptionless.Core.Services; +using Exceptionless.Core.Services.SourceMaps; using Exceptionless.Core.Utility; using Exceptionless.Core.Validation; using Foundatio.Caching; @@ -193,6 +194,19 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO AllowAutoRedirect = false, ConnectCallback = ConnectToPublicAddressAsync }); + services.AddSingleton(); + services.AddHttpClient(SourceMapService.HttpClientName) + .ConfigurePrimaryHttpMessageHandler(serviceProvider => new SocketsHttpHandler + { + ActivityHeadersPropagator = null, + AllowAutoRedirect = false, + AutomaticDecompression = DecompressionMethods.All, + ConnectCallback = serviceProvider.GetRequiredService().ConnectToPublicAddressAsync, + PreAuthenticate = false, + UseCookies = false, + UseProxy = false + }); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); @@ -223,7 +237,7 @@ private static async ValueTask ConnectToPublicAddressAsync(SocketsHttpCo } } - throw new HttpRequestException($"OAuth client metadata host '{context.DnsEndPoint.Host}' did not resolve to a reachable public address.", lastException); + throw new HttpRequestException($"Host '{context.DnsEndPoint.Host}' did not resolve to a reachable public address.", lastException); } public static void LogConfiguration(IServiceProvider serviceProvider, AppOptions appOptions, ILogger logger) diff --git a/src/Exceptionless.Core/Configuration/AppOptions.cs b/src/Exceptionless.Core/Configuration/AppOptions.cs index ff6217829a..99da0ce9d3 100644 --- a/src/Exceptionless.Core/Configuration/AppOptions.cs +++ b/src/Exceptionless.Core/Configuration/AppOptions.cs @@ -80,6 +80,7 @@ public class AppOptions public StripeOptions StripeOptions { get; internal set; } = null!; public AuthOptions AuthOptions { get; internal set; } = null!; public OAuthServerOptions OAuthServerOptions { get; internal set; } = null!; + public SourceMapOptions SourceMapOptions { get; internal set; } = null!; public static AppOptions ReadFromConfiguration(IConfiguration config) { @@ -133,6 +134,7 @@ public static AppOptions ReadFromConfiguration(IConfiguration config) options.StripeOptions = StripeOptions.ReadFromConfiguration(config); options.AuthOptions = AuthOptions.ReadFromConfiguration(config); options.OAuthServerOptions = OAuthServerOptions.ReadFromConfiguration(config); + options.SourceMapOptions = SourceMapOptions.ReadFromConfiguration(config); return options; } diff --git a/src/Exceptionless.Core/Configuration/SourceMapOptions.cs b/src/Exceptionless.Core/Configuration/SourceMapOptions.cs new file mode 100644 index 0000000000..f9db1f3678 --- /dev/null +++ b/src/Exceptionless.Core/Configuration/SourceMapOptions.cs @@ -0,0 +1,74 @@ +using Microsoft.Extensions.Configuration; + +namespace Exceptionless.Core.Configuration; + +public sealed class SourceMapOptions +{ + public bool EnableAutoDownload { get; internal set; } + public int RequestTimeoutMilliseconds { get; internal set; } + public int MaximumGeneratedFileSize { get; internal set; } + public int MaximumSourceMapSize { get; internal set; } + public int MaximumMappingSegments { get; internal set; } + public int MaximumRedirects { get; internal set; } + public int MaximumConcurrentDownloads { get; internal set; } + public int MaximumConcurrentDownloadsGlobally { get; internal set; } + public int AutoDownloadRateLimitPeriodMinutes { get; internal set; } + public int MaximumAutoDiscoveriesPerFreeClientKey { get; internal set; } + public int MaximumAutoDiscoveriesPerClientKey { get; internal set; } + public int MaximumAutoDiscoveriesPerFreeProject { get; internal set; } + public int MaximumAutoDiscoveriesPerProject { get; internal set; } + public int MaximumAutoDiscoveriesPerFreeOrganization { get; internal set; } + public int MaximumAutoDiscoveriesPerOrganization { get; internal set; } + public int MaximumAutoDownloadRequestsPerDestination { get; internal set; } + public int MaximumAutoDownloadConnectionsPerIpAddress { get; internal set; } + public int MaximumAutoDownloadRequestsGlobally { get; internal set; } + public int MaximumAutoRefreshRequestsPerDestination { get; internal set; } + public int MaximumAutoRefreshRequestsGlobally { get; internal set; } + public int MaximumFramesPerError { get; internal set; } + public int MaximumProcessingTimeMilliseconds { get; internal set; } + public int AutoDownloadRefreshIntervalMinutes { get; internal set; } + public int ParsedSourceMapCacheLifetimeMinutes { get; internal set; } + public long MaximumParsedSourceMapCacheSize { get; internal set; } + + public TimeSpan RequestTimeout => TimeSpan.FromMilliseconds(RequestTimeoutMilliseconds); + public TimeSpan MaximumProcessingTime => TimeSpan.FromMilliseconds(MaximumProcessingTimeMilliseconds); + public TimeSpan AutoDownloadRefreshInterval => TimeSpan.FromMinutes(AutoDownloadRefreshIntervalMinutes); + public TimeSpan ParsedSourceMapCacheLifetime => TimeSpan.FromMinutes(ParsedSourceMapCacheLifetimeMinutes); + public TimeSpan AutoDownloadRateLimitPeriod => TimeSpan.FromMinutes(AutoDownloadRateLimitPeriodMinutes); + + public static SourceMapOptions ReadFromConfiguration(IConfiguration configuration) + { + var section = configuration.GetSection("SourceMaps"); + return new SourceMapOptions + { + EnableAutoDownload = section.GetValue(nameof(EnableAutoDownload), true), + RequestTimeoutMilliseconds = ReadPositive(section, nameof(RequestTimeoutMilliseconds), 3000), + MaximumGeneratedFileSize = ReadPositive(section, nameof(MaximumGeneratedFileSize), 5 * 1024 * 1024), + MaximumSourceMapSize = ReadPositive(section, nameof(MaximumSourceMapSize), 20 * 1024 * 1024), + MaximumMappingSegments = ReadPositive(section, nameof(MaximumMappingSegments), 1_000_000), + MaximumRedirects = Math.Max(0, section.GetValue(nameof(MaximumRedirects), 3)), + MaximumConcurrentDownloads = ReadPositive(section, nameof(MaximumConcurrentDownloads), 4), + MaximumConcurrentDownloadsGlobally = ReadPositive(section, nameof(MaximumConcurrentDownloadsGlobally), 16), + AutoDownloadRateLimitPeriodMinutes = ReadPositive(section, nameof(AutoDownloadRateLimitPeriodMinutes), 15), + MaximumAutoDiscoveriesPerFreeClientKey = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerFreeClientKey), 5)), + MaximumAutoDiscoveriesPerClientKey = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerClientKey), 25)), + MaximumAutoDiscoveriesPerFreeProject = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerFreeProject), 10)), + MaximumAutoDiscoveriesPerProject = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerProject), 50)), + MaximumAutoDiscoveriesPerFreeOrganization = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerFreeOrganization), 10)), + MaximumAutoDiscoveriesPerOrganization = Math.Max(0, section.GetValue(nameof(MaximumAutoDiscoveriesPerOrganization), 100)), + MaximumAutoDownloadRequestsPerDestination = Math.Max(0, section.GetValue(nameof(MaximumAutoDownloadRequestsPerDestination), 100)), + MaximumAutoDownloadConnectionsPerIpAddress = Math.Max(0, section.GetValue(nameof(MaximumAutoDownloadConnectionsPerIpAddress), 200)), + MaximumAutoDownloadRequestsGlobally = Math.Max(0, section.GetValue(nameof(MaximumAutoDownloadRequestsGlobally), 1000)), + MaximumAutoRefreshRequestsPerDestination = Math.Max(0, section.GetValue(nameof(MaximumAutoRefreshRequestsPerDestination), 20)), + MaximumAutoRefreshRequestsGlobally = Math.Max(0, section.GetValue(nameof(MaximumAutoRefreshRequestsGlobally), 200)), + MaximumFramesPerError = ReadPositive(section, nameof(MaximumFramesPerError), 100), + MaximumProcessingTimeMilliseconds = ReadPositive(section, nameof(MaximumProcessingTimeMilliseconds), 5000), + AutoDownloadRefreshIntervalMinutes = ReadPositive(section, nameof(AutoDownloadRefreshIntervalMinutes), 60), + ParsedSourceMapCacheLifetimeMinutes = ReadPositive(section, nameof(ParsedSourceMapCacheLifetimeMinutes), 5), + MaximumParsedSourceMapCacheSize = Math.Max(1, section.GetValue(nameof(MaximumParsedSourceMapCacheSize), 100L * 1024 * 1024)) + }; + } + + private static int ReadPositive(IConfiguration section, string name, int defaultValue) + => Math.Max(1, section.GetValue(name, defaultValue)); +} diff --git a/src/Exceptionless.Core/Exceptionless.Core.csproj b/src/Exceptionless.Core/Exceptionless.Core.csproj index 9ff9a9608d..ffc6a91408 100644 --- a/src/Exceptionless.Core/Exceptionless.Core.csproj +++ b/src/Exceptionless.Core/Exceptionless.Core.csproj @@ -31,6 +31,7 @@ + diff --git a/src/Exceptionless.Core/Jobs/CleanupDataJob.cs b/src/Exceptionless.Core/Jobs/CleanupDataJob.cs index f212c67cf4..a3fd5a27d7 100644 --- a/src/Exceptionless.Core/Jobs/CleanupDataJob.cs +++ b/src/Exceptionless.Core/Jobs/CleanupDataJob.cs @@ -286,6 +286,8 @@ private async Task RemoveOrganizationAsync(Organization organization, JobContext await RenewLockAsync(context); long removedStacks = await _stackRepository.RemoveAllByOrganizationIdAsync(organization.Id); + await RemoveOrganizationProjectFilesAsync(organization.Id, context); + await RenewLockAsync(context); long removedProjects = await _projectRepository.RemoveAllByOrganizationIdAsync(organization.Id); @@ -316,6 +318,23 @@ private async Task RemoveSyntheticUserAsync(User user) private Task RemoveOrganizationFilesAsync(Organization organization, JobContext context) => RemoveFilesAsync(OrganizationStoragePaths.GetProfileImagesPath(organization.Id), context.CancellationToken); + private async Task RemoveOrganizationProjectFilesAsync(string organizationId, JobContext context) + { + var projects = await _projectRepository.GetByOrganizationIdAsync( + organizationId, + options => options.Include(project => project.Id).SoftDeleteMode(SoftDeleteQueryMode.All).SearchAfterPaging().PageLimit(100)); + + while (projects.Documents.Count > 0 && !context.CancellationToken.IsCancellationRequested) + { + foreach (var project in projects.Documents) + await _fileStorage.DeleteFilesAsync($"source-maps/{project.Id}/*", context.CancellationToken); + + await RenewLockAsync(context); + if (!await projects.NextPageAsync()) + break; + } + } + private async Task RemoveFilesAsync(string path, CancellationToken cancellationToken) { string searchPattern = $"{path}/*"; @@ -341,6 +360,7 @@ private async Task RemoveProjectsAsync(Project project, JobContext context) await RenewLockAsync(context); long removedStacks = await _stackRepository.RemoveAllByProjectIdAsync(project.OrganizationId, project.Id); + await _fileStorage.DeleteFilesAsync($"source-maps/{project.Id}/*", context.CancellationToken); await _projectRepository.RemoveAsync(project); _logger.RemoveProjectComplete(project.Name, project.Id, removedStacks, removedEvents); } diff --git a/src/Exceptionless.Core/Jobs/EventPostsJob.cs b/src/Exceptionless.Core/Jobs/EventPostsJob.cs index 0b7aaffec3..40db24f038 100644 --- a/src/Exceptionless.Core/Jobs/EventPostsJob.cs +++ b/src/Exceptionless.Core/Jobs/EventPostsJob.cs @@ -312,6 +312,7 @@ await _eventPostService.EnqueueAsync(new EventPost(false) { ApiVersion = ep.ApiVersion, CharSet = ep.CharSet, + ClientKeyHash = ep.ClientKeyHash, ContentEncoding = null, IpAddress = ep.IpAddress, MediaType = ep.MediaType, diff --git a/src/Exceptionless.Core/Models/Queues/EventPostInfo.cs b/src/Exceptionless.Core/Models/Queues/EventPostInfo.cs index 5ba9ecb819..69223071d8 100644 --- a/src/Exceptionless.Core/Models/Queues/EventPostInfo.cs +++ b/src/Exceptionless.Core/Models/Queues/EventPostInfo.cs @@ -10,6 +10,7 @@ public record EventPostInfo public string? UserAgent { get; init; } public string? ContentEncoding { get; init; } public string? IpAddress { get; init; } + public string? ClientKeyHash { get; init; } } public record EventPost : EventPostInfo diff --git a/src/Exceptionless.Core/Plugins/EventProcessor/Default/15_SourceMapPlugin.cs b/src/Exceptionless.Core/Plugins/EventProcessor/Default/15_SourceMapPlugin.cs new file mode 100644 index 0000000000..a92c8bc692 --- /dev/null +++ b/src/Exceptionless.Core/Plugins/EventProcessor/Default/15_SourceMapPlugin.cs @@ -0,0 +1,44 @@ +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Billing; +using Exceptionless.Core.Models; +using Exceptionless.Core.Pipeline; +using Exceptionless.Core.Services.SourceMaps; +using Foundatio.Serializer; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Plugins.EventProcessor.Default; + +[Priority(15)] +public sealed class SourceMapPlugin : EventProcessorPluginBase +{ + private readonly SourceMapService _sourceMapService; + private readonly ITextSerializer _serializer; + private readonly BillingPlans _billingPlans; + + public SourceMapPlugin(SourceMapService sourceMapService, ITextSerializer serializer, BillingPlans billingPlans, AppOptions options, ILoggerFactory loggerFactory) + : base(options, loggerFactory) + { + _sourceMapService = sourceMapService; + _serializer = serializer; + _billingPlans = billingPlans; + ContinueOnError = true; + } + + public override async Task EventProcessingAsync(EventContext context) + { + if (!context.Event.IsError()) + return; + + var error = context.Event.GetError(_serializer, _logger); + if (error is null) + return; + + var request = new SourceMapRequest( + context.Organization.Id, + context.Project.Id, + context.EventPostInfo?.ClientKeyHash, + String.Equals(context.Organization.PlanId, _billingPlans.FreePlan.Id, StringComparison.OrdinalIgnoreCase)); + if (await _sourceMapService.SymbolicateAsync(request, error)) + context.Event.SetError(error); + } +} diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapArtifact.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapArtifact.cs new file mode 100644 index 0000000000..365c6a6fd3 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapArtifact.cs @@ -0,0 +1,12 @@ +namespace Exceptionless.Core.Services.SourceMaps; + +public sealed record SourceMapArtifact +{ + public required string Id { get; init; } + public required string GeneratedFileUrl { get; init; } + public string? SourceMapUrl { get; init; } + public string? FileName { get; init; } + public required long Size { get; init; } + public required bool IsAutoDownloaded { get; init; } + public required DateTime CreatedUtc { get; init; } +} diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapContent.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapContent.cs new file mode 100644 index 0000000000..59306e9d18 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapContent.cs @@ -0,0 +1,20 @@ +namespace Exceptionless.Core.Services.SourceMaps; + +internal static class SourceMapContent +{ + public static async Task ReadLimitedAsync(Stream stream, int maximumBytes, CancellationToken cancellationToken) + { + using var memoryStream = new MemoryStream(Math.Min(maximumBytes, 64 * 1024)); + byte[] buffer = new byte[81920]; + int read; + while ((read = await stream.ReadAsync(buffer, cancellationToken)) > 0) + { + if (memoryStream.Length + read > maximumBytes) + throw new InvalidOperationException("The file exceeded the configured maximum size."); + + await memoryStream.WriteAsync(buffer.AsMemory(0, read), cancellationToken); + } + + return memoryStream.ToArray(); + } +} diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapDocument.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapDocument.cs new file mode 100644 index 0000000000..da35da0bc8 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapDocument.cs @@ -0,0 +1,246 @@ +using System.Text.Json; + +namespace Exceptionless.Core.Services.SourceMaps; + +public sealed class SourceMapDocument +{ + private const string Base64Characters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + private const int MaximumSourceEntries = 1_000_000; + private readonly IReadOnlyList> _lines; + private readonly string[] _names; + private readonly string[] _sources; + + private SourceMapDocument( + string? sourceRoot, + string[] sources, + string[] names, + IReadOnlyList> lines, + long estimatedMemorySize) + { + SourceRoot = sourceRoot; + _sources = sources; + _names = names; + _lines = lines; + EstimatedMemorySize = estimatedMemorySize; + } + + public string? SourceRoot { get; } + internal long EstimatedMemorySize { get; } + + public static SourceMapDocument Parse(byte[] sourceMap, int maximumSegments = 1_000_000) + => Parse(sourceMap, maximumSegments, 100_000, MaximumSourceEntries); + + internal static SourceMapDocument Parse(byte[] sourceMap, int maximumSegments, int maximumLines, int maximumSourceEntries = MaximumSourceEntries) + { + using var document = JsonDocument.Parse(sourceMap); + var root = document.RootElement; + + if (!root.TryGetProperty("version", out var versionElement) + || versionElement.ValueKind != JsonValueKind.Number + || !versionElement.TryGetInt32(out int version) + || version != 3) + throw new JsonException("Only source map version 3 is supported."); + + if (root.TryGetProperty("sections", out _)) + throw new JsonException("Indexed source maps are not supported."); + + if (!root.TryGetProperty("mappings", out var mappingsElement) || mappingsElement.ValueKind != JsonValueKind.String) + throw new JsonException("The source map mappings are required."); + + string mappings = mappingsElement.GetString() ?? throw new JsonException("The source map mappings are required."); + string[] sources = ReadStringArray(root, "sources", maximumSourceEntries, out long sourcesMemorySize); + long namesMemorySize = 0; + string[] names = root.TryGetProperty("names", out _) + ? ReadStringArray(root, "names", maximumSourceEntries, out namesMemorySize) + : []; + string? sourceRoot = root.TryGetProperty("sourceRoot", out var sourceRootElement) ? sourceRootElement.GetString() : null; + + var lines = DecodeMappings(mappings, sources.Length, names.Length, maximumSegments, maximumLines, out int segmentCount); + long estimatedMemorySize = sourceMap.LongLength + + sourcesMemorySize + + namesMemorySize + + EstimateStringMemorySize(sourceRoot) + + (segmentCount * 64L) + + (lines.Count * 64L); + return new SourceMapDocument(sourceRoot, sources, names, lines, estimatedMemorySize); + } + + public SourceMapLocation? FindOriginalLocation(int generatedLine, int generatedColumn) + { + if (generatedLine < 0 || generatedLine >= _lines.Count || generatedColumn < 0) + return null; + + var segments = _lines[generatedLine]; + MappingSegment? match = null; + foreach (var segment in segments) + { + if (segment.GeneratedColumn > generatedColumn) + break; + + match = segment; + } + + if (match?.SourceIndex is not int sourceIndex || match.OriginalLine is not int originalLine || match.OriginalColumn is not int originalColumn) + return null; + + string? name = match.NameIndex is int nameIndex ? _names[nameIndex] : null; + return new SourceMapLocation(CombineSource(SourceRoot, _sources[sourceIndex]), originalLine, originalColumn, name); + } + + private static string[] ReadStringArray(JsonElement root, string propertyName, int maximumEntries, out long estimatedMemorySize) + { + if (!root.TryGetProperty(propertyName, out var element) || element.ValueKind != JsonValueKind.Array) + throw new JsonException($"The source map {propertyName} array is required."); + if (maximumEntries < 1) + throw new ArgumentOutOfRangeException(nameof(maximumEntries)); + + estimatedMemorySize = 0; + var values = new List(Math.Min(element.GetArrayLength(), maximumEntries)); + foreach (var value in element.EnumerateArray()) + { + if (values.Count >= maximumEntries) + throw new JsonException($"The source map {propertyName} array contains too many entries."); + + string text = value.GetString() ?? throw new JsonException($"The source map {propertyName} array contains a null value."); + values.Add(text); + estimatedMemorySize += IntPtr.Size + EstimateStringMemorySize(text); + } + + return values.ToArray(); + } + + private static long EstimateStringMemorySize(string? value) => value is null ? 0 : 24L + (value.Length * sizeof(char)); + + private static IReadOnlyList> DecodeMappings( + string mappings, + int sourceCount, + int nameCount, + int maximumSegments, + int maximumLines, + out int segmentCount) + { + if (maximumSegments < 1) + throw new ArgumentOutOfRangeException(nameof(maximumSegments)); + if (maximumLines < 1) + throw new ArgumentOutOfRangeException(nameof(maximumLines)); + + int lineCount = 1; + int encodedSegmentCount = 0; + bool hasEncodedSegment = false; + foreach (char character in mappings) + { + if (character is ',' or ';') + { + if (hasEncodedSegment && ++encodedSegmentCount > maximumSegments) + throw new JsonException("The source map contains too many mapping segments."); + hasEncodedSegment = false; + } + else + { + hasEncodedSegment = true; + } + + if (character == ';' && ++lineCount > maximumLines) + throw new JsonException("The source map contains too many generated lines."); + } + if (hasEncodedSegment && ++encodedSegmentCount > maximumSegments) + throw new JsonException("The source map contains too many mapping segments."); + + var lines = new List>(lineCount); + int sourceIndex = 0; + int originalLine = 0; + int originalColumn = 0; + int nameIndex = 0; + segmentCount = 0; + + foreach (string encodedLine in mappings.Split(';')) + { + int generatedColumn = 0; + var line = new List(); + foreach (string encodedSegment in encodedLine.Split(',', StringSplitOptions.RemoveEmptyEntries)) + { + if (++segmentCount > maximumSegments) + throw new JsonException("The source map contains too many mapping segments."); + + int index = 0; + generatedColumn += DecodeValue(encodedSegment, ref index); + if (generatedColumn < 0) + throw new JsonException("A source map generated column cannot be negative."); + + if (index == encodedSegment.Length) + { + line.Add(new MappingSegment(generatedColumn, null, null, null, null)); + continue; + } + + sourceIndex += DecodeValue(encodedSegment, ref index); + originalLine += DecodeValue(encodedSegment, ref index); + originalColumn += DecodeValue(encodedSegment, ref index); + if (sourceIndex < 0 || sourceIndex >= sourceCount || originalLine < 0 || originalColumn < 0) + throw new JsonException("A source map mapping points outside its source data."); + + int? segmentNameIndex = null; + if (index < encodedSegment.Length) + { + nameIndex += DecodeValue(encodedSegment, ref index); + if (nameIndex < 0 || nameIndex >= nameCount) + throw new JsonException("A source map mapping points outside its names array."); + + segmentNameIndex = nameIndex; + } + + if (index != encodedSegment.Length) + throw new JsonException("A source map segment has an invalid field count."); + + line.Add(new MappingSegment(generatedColumn, sourceIndex, originalLine, originalColumn, segmentNameIndex)); + } + + lines.Add(line); + } + + return lines; + } + + private static int DecodeValue(string segment, ref int index) + { + long result = 0; + int shift = 0; + bool hasContinuation; + + do + { + if (index >= segment.Length) + throw new JsonException("A source map VLQ value is incomplete."); + + int digit = Base64Characters.IndexOf(segment[index++]); + if (digit < 0) + throw new JsonException("A source map VLQ value contains an invalid character."); + + hasContinuation = (digit & 32) != 0; + digit &= 31; + result += (long)digit << shift; + shift += 5; + if (shift > 35 || result > (long)Int32.MaxValue * 2 + 1) + throw new JsonException("A source map VLQ value is too large."); + } while (hasContinuation); + + bool isNegative = (result & 1) == 1; + result >>= 1; + return isNegative ? -(int)result : (int)result; + } + + private static string CombineSource(string? sourceRoot, string source) + { + if (String.IsNullOrWhiteSpace(sourceRoot) || Uri.TryCreate(source, UriKind.Absolute, out _)) + return source; + + if (Uri.TryCreate(sourceRoot, UriKind.Absolute, out var sourceRootUri) && Uri.TryCreate(sourceRootUri, source, out var combinedUri)) + return combinedUri.ToString(); + + return $"{sourceRoot.TrimEnd('/')}/{source.TrimStart('/')}"; + } + + private sealed record MappingSegment(int GeneratedColumn, int? SourceIndex, int? OriginalLine, int? OriginalColumn, int? NameIndex); +} + +public sealed record SourceMapLocation(string Source, int Line, int Column, string? Name); diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapDownloader.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapDownloader.cs new file mode 100644 index 0000000000..28b3641b09 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapDownloader.cs @@ -0,0 +1,183 @@ +using System.Net; +using System.Net.Http.Headers; +using System.Text; +using Exceptionless.Core.Configuration; + +namespace Exceptionless.Core.Services.SourceMaps; + +internal sealed class SourceMapDownloader +{ + private readonly IHttpClientFactory _httpClientFactory; + private readonly SourceMapOptions _options; + private readonly SourceMapRequestThrottle _throttle; + + public SourceMapDownloader(IHttpClientFactory httpClientFactory, AppOptions options, SourceMapRequestThrottle throttle) + { + _httpClientFactory = httpClientFactory; + _options = options.SourceMapOptions; + _throttle = throttle; + } + + public async Task DownloadAsync(Uri generatedFileUri, bool isRefresh, CancellationToken cancellationToken) + { + using var generatedResponse = await SendAsync( + generatedFileUri, + _options.MaximumGeneratedFileSize, + validateContentLength: false, + request => + { + request.Headers.Range = new RangeHeaderValue(null, 64 * 1024); + request.Headers.AcceptEncoding.Clear(); + request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("identity")); + }, + isRefresh, + cancellationToken); + if (!generatedResponse.Response.IsSuccessStatusCode) + return null; + + string? sourceMapReference = GetSourceMapHeader(generatedResponse.Response); + if (String.IsNullOrWhiteSpace(sourceMapReference) + && (generatedResponse.Response.Content.Headers.ContentLength is not long contentLength + || contentLength <= _options.MaximumGeneratedFileSize)) + { + try + { + byte[] generatedContent = await SourceMapContent.ReadLimitedAsync( + await generatedResponse.Response.Content.ReadAsStreamAsync(cancellationToken), + _options.MaximumGeneratedFileSize, + cancellationToken); + sourceMapReference = FindSourceMapReference(Encoding.UTF8.GetString(generatedContent)); + } + catch (InvalidOperationException) + { + // A CDN may ignore the range request and return the entire bundle. Continue to the conventional .map fallback. + } + } + + if (String.IsNullOrWhiteSpace(sourceMapReference)) + { + var fallbackUriBuilder = new UriBuilder(generatedResponse.Uri) { Path = generatedResponse.Uri.AbsolutePath + ".map" }; + return await DownloadContentAsync(fallbackUriBuilder.Uri, isRefresh, cancellationToken); + } + + if (sourceMapReference.StartsWith("data:", StringComparison.OrdinalIgnoreCase)) + return new DownloadedSourceMap(DecodeDataUri(sourceMapReference, _options.MaximumSourceMapSize), null); + + if (!Uri.TryCreate(generatedResponse.Uri, sourceMapReference, out var sourceMapUri) || !IsAutoDownloadUri(sourceMapUri)) + return null; + + return await DownloadContentAsync(sourceMapUri, isRefresh, cancellationToken); + } + + private async Task DownloadContentAsync(Uri sourceMapUri, bool isRefresh, CancellationToken cancellationToken) + { + using var result = await SendAsync(sourceMapUri, _options.MaximumSourceMapSize, validateContentLength: true, null, isRefresh, cancellationToken); + if (!result.Response.IsSuccessStatusCode) + return null; + + byte[] content = await SourceMapContent.ReadLimitedAsync( + await result.Response.Content.ReadAsStreamAsync(cancellationToken), + _options.MaximumSourceMapSize, + cancellationToken); + return new DownloadedSourceMap(content, result.Uri.AbsoluteUri); + } + + private async Task SendAsync( + Uri uri, + int maximumBytes, + bool validateContentLength, + Action? configureRequest, + bool isRefresh, + CancellationToken cancellationToken) + { + Uri currentUri = uri; + for (int redirectCount = 0; ; redirectCount++) + { + if (!IsAutoDownloadUri(currentUri)) + throw new InvalidOperationException("Source map auto-download only supports public HTTPS URLs."); + if (!await _throttle.TryReserveOutboundRequestAsync(currentUri, isRefresh)) + throw new SourceMapRequestThrottledException(); + + using var request = new HttpRequestMessage(HttpMethod.Get, currentUri); + request.Headers.AcceptEncoding.ParseAdd("gzip, deflate, br"); + configureRequest?.Invoke(request); + + var client = _httpClientFactory.CreateClient(SourceMapService.HttpClientName); + var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken); + if (validateContentLength && response.Content.Headers.ContentLength > maximumBytes) + { + response.Dispose(); + throw new InvalidOperationException("The downloaded file exceeded the configured maximum size."); + } + + if (!IsRedirect(response.StatusCode)) + return new HttpDownloadResult(currentUri, response); + + if (redirectCount >= _options.MaximumRedirects || response.Headers.Location is null) + { + response.Dispose(); + throw new InvalidOperationException("The source map download exceeded the allowed redirects."); + } + + Uri redirectUri = response.Headers.Location.IsAbsoluteUri ? response.Headers.Location : new Uri(currentUri, response.Headers.Location); + response.Dispose(); + currentUri = redirectUri; + } + } + + private static bool IsAutoDownloadUri(Uri uri) + => uri.IsDefaultPort && SourceMapService.TryNormalizeGeneratedFileUrl(uri.AbsoluteUri, requireHttps: true, out _); + + private static bool IsRedirect(HttpStatusCode statusCode) => statusCode is HttpStatusCode.MovedPermanently + or HttpStatusCode.Redirect + or HttpStatusCode.RedirectMethod + or HttpStatusCode.TemporaryRedirect + or HttpStatusCode.PermanentRedirect; + + private static string? GetSourceMapHeader(HttpResponseMessage response) + { + if (response.Headers.TryGetValues("SourceMap", out var sourceMapValues)) + return sourceMapValues.FirstOrDefault(); + if (response.Headers.TryGetValues("X-SourceMap", out var legacySourceMapValues)) + return legacySourceMapValues.FirstOrDefault(); + return null; + } + + private static string? FindSourceMapReference(string generatedContent) + { + const string marker = "sourceMappingURL="; + int markerIndex = generatedContent.LastIndexOf(marker, StringComparison.Ordinal); + if (markerIndex < 0) + return null; + + int start = markerIndex + marker.Length; + int end = generatedContent.IndexOfAny(['\r', '\n'], start); + string value = generatedContent[start..(end < 0 ? generatedContent.Length : end)].Trim(); + if (value.EndsWith("*/", StringComparison.Ordinal)) + value = value[..^2].Trim(); + return value; + } + + private static byte[] DecodeDataUri(string value, int maximumBytes) + { + int commaIndex = value.IndexOf(','); + if (commaIndex < 0) + throw new FormatException("The inline source map data URI is invalid."); + + string metadata = value[..commaIndex]; + string data = value[(commaIndex + 1)..]; + byte[] decoded = metadata.Contains(";base64", StringComparison.OrdinalIgnoreCase) + ? Convert.FromBase64String(data) + : Encoding.UTF8.GetBytes(Uri.UnescapeDataString(data)); + if (decoded.Length > maximumBytes) + throw new InvalidOperationException("The inline source map exceeded the configured maximum size."); + return decoded; + } + + internal sealed record DownloadedSourceMap(byte[] Content, string? SourceMapUrl); + + private sealed record HttpDownloadResult(Uri Uri, HttpResponseMessage Response) : IDisposable + { + public void Dispose() => Response.Dispose(); + } +} diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapRequestThrottle.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapRequestThrottle.cs new file mode 100644 index 0000000000..7cb158c94d --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapRequestThrottle.cs @@ -0,0 +1,133 @@ +using System.Net; +using System.Net.Sockets; +using Exceptionless.Core.Configuration; +using Exceptionless.Core.Extensions; +using Foundatio.Caching; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services.SourceMaps; + +public sealed class SourceMapRequestThrottle +{ + private readonly ICacheClient _cache; + private readonly SourceMapOptions _options; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + + public SourceMapRequestThrottle(ICacheClient cache, AppOptions options, TimeProvider timeProvider, ILogger logger) + { + _cache = cache; + _options = options.SourceMapOptions; + _timeProvider = timeProvider; + _logger = logger; + } + + internal async Task TryReserveDiscoveryAsync(SourceMapRequest request) + { + int clientKeyLimit = request.IsFreePlan + ? _options.MaximumAutoDiscoveriesPerFreeClientKey + : _options.MaximumAutoDiscoveriesPerClientKey; + if (!String.IsNullOrEmpty(request.ClientKeyHash) + && !await TryReserveBucketAsync("client-key", $"{request.ProjectId}:{request.ClientKeyHash}", clientKeyLimit)) + { + return false; + } + + int projectLimit = request.IsFreePlan + ? _options.MaximumAutoDiscoveriesPerFreeProject + : _options.MaximumAutoDiscoveriesPerProject; + if (!await TryReserveBucketAsync("project", request.ProjectId, projectLimit)) + return false; + + int organizationLimit = request.IsFreePlan + ? _options.MaximumAutoDiscoveriesPerFreeOrganization + : _options.MaximumAutoDiscoveriesPerOrganization; + return await TryReserveBucketAsync("organization", request.OrganizationId, organizationLimit); + } + + internal async Task TryReserveOutboundRequestAsync(Uri uri, bool isRefresh = false) + { + string destination = uri.IdnHost.ToLowerInvariant().ToSHA256(); + string scopePrefix = isRefresh ? "refresh-" : String.Empty; + int destinationLimit = isRefresh + ? _options.MaximumAutoRefreshRequestsPerDestination + : _options.MaximumAutoDownloadRequestsPerDestination; + if (!await TryReserveBucketAsync(scopePrefix + "destination", destination, destinationLimit)) + return false; + + int globalLimit = isRefresh + ? _options.MaximumAutoRefreshRequestsGlobally + : _options.MaximumAutoDownloadRequestsGlobally; + return await TryReserveBucketAsync(scopePrefix + "global", "all", globalLimit); + } + + internal async ValueTask ConnectToPublicAddressAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken) + { + var addresses = await Dns.GetHostAddressesAsync(context.DnsEndPoint.Host, cancellationToken); + Exception? lastException = null; + bool addressThrottled = false; + foreach (var address in addresses) + { + if (!OAuthClientMetadataService.IsPublicAddress(address)) + continue; + + string addressHash = address.ToString().ToSHA256(); + if (!await TryReserveBucketAsync("ip-address", addressHash, _options.MaximumAutoDownloadConnectionsPerIpAddress)) + { + addressThrottled = true; + continue; + } + + var socket = new Socket(address.AddressFamily, SocketType.Stream, ProtocolType.Tcp) { NoDelay = true }; + try + { + await socket.ConnectAsync(new IPEndPoint(address, context.DnsEndPoint.Port), cancellationToken); + return new NetworkStream(socket, ownsSocket: true); + } + catch (SocketException ex) + { + lastException = ex; + socket.Dispose(); + } + } + + if (addressThrottled) + throw new SourceMapRequestThrottledException(); + + throw new HttpRequestException($"Host '{context.DnsEndPoint.Host}' did not resolve to a reachable public address.", lastException); + } + + private async Task TryReserveBucketAsync(string scope, string identifier, int limit) + { + var window = GetCurrentWindow(); + string counterKey = $"source-maps:rate:{scope}:{identifier}:{window.Id}"; + string blockedKey = counterKey + ":blocked"; + if ((await _cache.GetAsync(blockedKey)).HasValue) + return false; + + long count = await _cache.IncrementAsync(counterKey, 1, window.CounterLifetime); + if (count <= limit) + return true; + + await _cache.SetAsync(blockedKey, true, window.Remaining); + if (count == limit + 1) + _logger.LogWarning("Source map automatic download {RateLimitScope} rate limit reached.", scope); + + return false; + } + + private RateLimitWindow GetCurrentWindow() + { + long periodSeconds = Math.Max(1, (long)_options.AutoDownloadRateLimitPeriod.TotalSeconds); + long now = _timeProvider.GetUtcNow().ToUnixTimeSeconds(); + long id = now / periodSeconds; + long remainingSeconds = Math.Max(1, ((id + 1) * periodSeconds) - now); + return new RateLimitWindow(id, TimeSpan.FromSeconds(remainingSeconds), TimeSpan.FromSeconds(periodSeconds * 2)); + } + + private sealed record RateLimitWindow(long Id, TimeSpan Remaining, TimeSpan CounterLifetime); +} + +internal sealed record SourceMapRequest(string OrganizationId, string ProjectId, string? ClientKeyHash, bool IsFreePlan); + +internal sealed class SourceMapRequestThrottledException : Exception; diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapService.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapService.cs new file mode 100644 index 0000000000..abe2ffcc94 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapService.cs @@ -0,0 +1,470 @@ +using System.Collections.Concurrent; +using System.Text.Json; +using Exceptionless.Core.Configuration; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.Data; +using Foundatio.Caching; +using Foundatio.Lock; +using Foundatio.Storage; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services.SourceMaps; + +public sealed class SourceMapService : IDisposable +{ + public const string HttpClientName = "SourceMaps"; + public const int MaximumUploadRequestSize = 21 * 1024 * 1024; + private const string SourceMapDataKey = "@source_map"; + private static readonly TimeSpan FailureCacheLifetime = TimeSpan.FromMinutes(15); + private readonly SemaphoreSlim _downloadSemaphore; + private readonly ConcurrentDictionary>> _inflightSourceMaps = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _parsedSourceMapEntries = new(StringComparer.Ordinal); + private readonly MemoryCache _parsedSourceMaps; + private readonly SourceMapDownloader _downloader; + private readonly SourceMapStorage _storage; + private readonly SourceMapRequestThrottle _throttle; + private readonly ICacheClient _cache; + private readonly ILockProvider _lockProvider; + private readonly SourceMapOptions _options; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + + public SourceMapService( + IHttpClientFactory httpClientFactory, + IFileStorage storage, + ICacheClient cache, + ILockProvider lockProvider, + SourceMapRequestThrottle throttle, + JsonSerializerOptions serializerOptions, + AppOptions options, + TimeProvider timeProvider, + ILogger logger) + { + _downloader = new SourceMapDownloader(httpClientFactory, options, throttle); + _storage = new SourceMapStorage(storage, serializerOptions, logger); + _throttle = throttle; + _cache = cache; + _lockProvider = lockProvider; + _options = options.SourceMapOptions; + _downloadSemaphore = new SemaphoreSlim(_options.MaximumConcurrentDownloads); + _parsedSourceMaps = new MemoryCache(new MemoryCacheOptions { SizeLimit = _options.MaximumParsedSourceMapCacheSize }); + _timeProvider = timeProvider; + _logger = logger; + } + + public async Task SaveUploadedAsync( + string projectId, + string generatedFileUrl, + string? fileName, + Stream stream, + CancellationToken cancellationToken = default) + { + if (!TryNormalizeGeneratedFileUrl(generatedFileUrl, requireHttps: false, out var generatedFileUri)) + throw new ArgumentException("The generated file URL must be an absolute HTTP or HTTPS URL without credentials or a fragment.", nameof(generatedFileUrl)); + + byte[] sourceMap = await SourceMapContent.ReadLimitedAsync(stream, _options.MaximumSourceMapSize, cancellationToken); + _ = SourceMapDocument.Parse(sourceMap, _options.MaximumMappingSegments); + + string normalizedUrl = generatedFileUri.AbsoluteUri; + var artifact = new SourceMapArtifact + { + Id = GetArtifactId(normalizedUrl), + GeneratedFileUrl = normalizedUrl, + FileName = String.IsNullOrWhiteSpace(fileName) ? null : Path.GetFileName(fileName), + Size = sourceMap.LongLength, + IsAutoDownloaded = false, + CreatedUtc = _timeProvider.GetUtcNow().UtcDateTime + }; + + string cacheKey = GetMemoryCacheKey(projectId, normalizedUrl); + await WaitForInflightSourceMapAsync(cacheKey, cancellationToken); + await using var artifactLock = await _lockProvider.TryAcquireAsync(GetArtifactLockKey(projectId, artifact.Id), TimeSpan.FromSeconds(30), cancellationToken); + if (artifactLock is null) + { + cancellationToken.ThrowIfCancellationRequested(); + throw new IOException("Unable to acquire the source map storage lock."); + } + + await _storage.SaveAsync(projectId, artifact, sourceMap, cancellationToken); + await ClearCachesAsync(projectId, normalizedUrl); + return artifact; + } + + public Task> GetArtifactsAsync(string projectId, CancellationToken cancellationToken = default) + => _storage.GetArtifactsAsync(projectId, cancellationToken); + + public async Task DeleteArtifactAsync(string projectId, string artifactId, CancellationToken cancellationToken = default) + { + if (!IsArtifactId(artifactId)) + return false; + + await WaitForInflightArtifactAsync(projectId, artifactId, cancellationToken); + await using var artifactLock = await _lockProvider.TryAcquireAsync(GetArtifactLockKey(projectId, artifactId), TimeSpan.FromSeconds(30), cancellationToken); + if (artifactLock is null) + { + cancellationToken.ThrowIfCancellationRequested(); + return false; + } + + var artifact = await _storage.DeleteAsync(projectId, artifactId, cancellationToken); + if (artifact is null) + return false; + + await ClearCachesAsync(projectId, artifact.GeneratedFileUrl); + return true; + } + + public async Task DeleteAllArtifactsAsync(string projectId, CancellationToken cancellationToken = default) + { + await _storage.DeleteAllAsync(projectId, cancellationToken); + foreach (string key in _parsedSourceMapEntries.Keys.Where(key => key.StartsWith(projectId + ':', StringComparison.Ordinal))) + _parsedSourceMaps.Remove(key); + foreach (string key in _inflightSourceMaps.Keys.Where(key => key.StartsWith(projectId + ':', StringComparison.Ordinal))) + _inflightSourceMaps.TryRemove(key, out _); + } + + public Task SymbolicateAsync(string projectId, InnerError? error, CancellationToken cancellationToken = default) + => SymbolicateAsync(new SourceMapRequest(projectId, projectId, null, false), error, cancellationToken); + + internal async Task SymbolicateAsync(SourceMapRequest request, InnerError? error, CancellationToken cancellationToken = default) + { + bool changed = false; + int framesProcessed = 0; + using var processingCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + processingCancellationTokenSource.CancelAfter(_options.MaximumProcessingTime); + try + { + while (error is not null) + { + if (error.StackTrace is not null) + { + foreach (var frame in error.StackTrace) + { + if (++framesProcessed > _options.MaximumFramesPerError) + return changed; + + if (await SymbolicateFrameAsync(request, frame, processingCancellationTokenSource.Token)) + changed = true; + } + } + + error = error.Inner; + } + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + _logger.LogDebug("Source map processing exceeded its time budget for project {ProjectId}.", request.ProjectId); + } + + return changed; + } + + public void Dispose() + { + _parsedSourceMaps.Dispose(); + _downloadSemaphore.Dispose(); + } + + private async Task SymbolicateFrameAsync(SourceMapRequest request, StackFrame frame, CancellationToken cancellationToken) + { + if (frame.Data?.ContainsKey(SourceMapDataKey) == true || frame.LineNumber is null || frame.LineNumber < 1 || frame.Column is null || String.IsNullOrWhiteSpace(frame.FileName)) + return false; + + if (!TryNormalizeGeneratedFileUrl(frame.FileName, requireHttps: false, out var generatedFileUri)) + return false; + + var resolved = await GetSourceMapAsync(request, generatedFileUri, cancellationToken); + if (resolved is null) + return false; + + int generatedColumn = frame.Column.Value; + if (generatedColumn > 0) + generatedColumn--; + + var original = resolved.Document.FindOriginalLocation(frame.LineNumber.Value - 1, generatedColumn); + if (original is null) + return false; + + frame.Data ??= new DataDictionary(); + frame.Data[SourceMapDataKey] = new DataDictionary + { + ["generated_file_name"] = frame.FileName, + ["generated_line_number"] = frame.LineNumber, + ["generated_column"] = frame.Column, + ["source_map_id"] = resolved.Artifact.Id + }; + frame.FileName = original.Source; + frame.LineNumber = original.Line + 1; + frame.Column = original.Column + 1; + if (!String.IsNullOrWhiteSpace(original.Name)) + frame.Name = original.Name; + + return true; + } + + private async Task GetSourceMapAsync(SourceMapRequest request, Uri generatedFileUri, CancellationToken cancellationToken) + { + string cacheKey = GetMemoryCacheKey(request.ProjectId, generatedFileUri.AbsoluteUri); + if (_parsedSourceMaps.TryGetValue(cacheKey, out ResolvedSourceMap? cached)) + return cached; + + var lazy = _inflightSourceMaps.GetOrAdd(cacheKey, _ => new Lazy>( + () => LoadAndCacheSourceMapAsync(request, generatedFileUri, cacheKey), + LazyThreadSafetyMode.ExecutionAndPublication)); + Task loadTask = lazy.Value; + + try + { + return await loadTask.WaitAsync(cancellationToken); + } + finally + { + if (loadTask.IsCompleted) + RemoveInflightSourceMap(cacheKey, lazy); + else + _ = RemoveInflightSourceMapWhenCompleteAsync(cacheKey, lazy, loadTask); + } + } + + private async Task LoadAndCacheSourceMapAsync(SourceMapRequest request, Uri generatedFileUri, string cacheKey) + { + var resolved = await LoadSourceMapAsync(request, generatedFileUri); + if (resolved is not null && resolved.Document.EstimatedMemorySize <= _options.MaximumParsedSourceMapCacheSize) + { + _parsedSourceMapEntries[cacheKey] = resolved; + var cacheOptions = new MemoryCacheEntryOptions + { + AbsoluteExpirationRelativeToNow = _options.ParsedSourceMapCacheLifetime, + Size = Math.Max(1, resolved.Document.EstimatedMemorySize) + }.RegisterPostEvictionCallback( + static (key, _, _, state) => + { + if (key is string evictedCacheKey && state is ParsedSourceMapCacheRegistration registration) + registration.Service.RemoveParsedSourceMapEntry(evictedCacheKey, registration.SourceMap); + }, + new ParsedSourceMapCacheRegistration(this, resolved)); + _parsedSourceMaps.Set(cacheKey, resolved, cacheOptions); + } + + return resolved; + } + + private async Task RemoveInflightSourceMapWhenCompleteAsync( + string cacheKey, + Lazy> lazy, + Task loadTask) + { + try + { + await loadTask; + } + catch + { + // The caller observes the load failure; this continuation only releases the in-flight entry. + } + + RemoveInflightSourceMap(cacheKey, lazy); + } + + private void RemoveInflightSourceMap(string cacheKey, Lazy> lazy) + { + ICollection>>> entries = _inflightSourceMaps; + entries.Remove(new KeyValuePair>>(cacheKey, lazy)); + } + + private void RemoveParsedSourceMapEntry(string cacheKey, ResolvedSourceMap sourceMap) + { + ICollection> entries = _parsedSourceMapEntries; + entries.Remove(new KeyValuePair(cacheKey, sourceMap)); + } + + private async Task WaitForInflightSourceMapAsync(string cacheKey, CancellationToken cancellationToken) + { + if (_inflightSourceMaps.TryGetValue(cacheKey, out var sourceMap)) + await WaitForInflightSourceMapAsync(sourceMap, cancellationToken); + } + + private async Task WaitForInflightArtifactAsync(string projectId, string artifactId, CancellationToken cancellationToken) + { + string cacheKeyPrefix = projectId + ':'; + var sourceMaps = _inflightSourceMaps + .Where(entry => entry.Key.StartsWith(cacheKeyPrefix, StringComparison.Ordinal) + && GetArtifactId(entry.Key[cacheKeyPrefix.Length..]) == artifactId) + .Select(entry => entry.Value) + .ToArray(); + + foreach (var sourceMap in sourceMaps) + await WaitForInflightSourceMapAsync(sourceMap, cancellationToken); + } + + private static async Task WaitForInflightSourceMapAsync(Lazy> sourceMap, CancellationToken cancellationToken) + { + try + { + await sourceMap.Value.WaitAsync(cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch + { + // A failed lookup must not prevent an explicit upload or delete from repairing the artifact. + } + } + + private async Task LoadSourceMapAsync(SourceMapRequest request, Uri generatedFileUri) + { + string projectId = request.ProjectId; + string generatedFileUrl = generatedFileUri.AbsoluteUri; + string artifactId = GetArtifactId(generatedFileUrl); + var stored = await _storage.GetAsync(projectId, artifactId, _options.MaximumSourceMapSize, CancellationToken.None); + bool refreshStoredMap = ShouldRefresh(stored, generatedFileUri); + if (stored is not null && !refreshStoredMap) + return Resolve(stored.Artifact, stored.Content); + + if (!_options.EnableAutoDownload || !String.Equals(generatedFileUri.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase)) + return null; + + string failureCacheKey = GetFailureCacheKey(projectId, generatedFileUrl); + if ((await _cache.GetAsync(failureCacheKey)).HasValue) + return null; + + try + { + using var timeoutCancellationTokenSource = new CancellationTokenSource(_options.RequestTimeout); + await using var artifactLock = await _lockProvider.TryAcquireAsync( + GetArtifactLockKey(projectId, artifactId), + TimeSpan.FromSeconds(30), + timeoutCancellationTokenSource.Token); + if (artifactLock is null) + return await CacheFailureAsync(failureCacheKey); + + stored = await _storage.GetAsync(projectId, artifactId, _options.MaximumSourceMapSize, CancellationToken.None); + if (stored is not null && !ShouldRefresh(stored, generatedFileUri)) + return Resolve(stored.Artifact, stored.Content); + + if (stored is null && !await _throttle.TryReserveDiscoveryAsync(request)) + return null; + + if (!await _downloadSemaphore.WaitAsync(TimeSpan.Zero, timeoutCancellationTokenSource.Token)) + return null; + SourceMapDownloader.DownloadedSourceMap? downloaded; + try + { + await using var globalDownloadSlot = await TryAcquireGlobalDownloadSlotAsync(artifactId, timeoutCancellationTokenSource.Token); + if (globalDownloadSlot is null) + return null; + + downloaded = await _downloader.DownloadAsync(generatedFileUri, stored is not null, timeoutCancellationTokenSource.Token); + if (downloaded is null) + return await CacheFailureAsync(failureCacheKey); + } + finally + { + _downloadSemaphore.Release(); + } + + var artifact = new SourceMapArtifact + { + Id = artifactId, + GeneratedFileUrl = generatedFileUrl, + SourceMapUrl = downloaded.SourceMapUrl, + FileName = GetDownloadedFileName(downloaded.SourceMapUrl), + Size = downloaded.Content.LongLength, + IsAutoDownloaded = true, + CreatedUtc = _timeProvider.GetUtcNow().UtcDateTime + }; + var resolved = Resolve(artifact, downloaded.Content); + await _storage.SaveAsync(projectId, artifact, downloaded.Content, CancellationToken.None); + return resolved; + } + catch (OperationCanceledException ex) + { + _logger.LogDebug(ex, "Timed out downloading a source map for {GeneratedFileUrl}.", generatedFileUrl); + return await CacheFailureAsync(failureCacheKey); + } + catch (SourceMapRequestThrottledException) + { + return null; + } + catch (Exception ex) when (ex is HttpRequestException or IOException or JsonException or InvalidOperationException or FormatException) + { + _logger.LogWarning(ex, "Unable to download a source map for {GeneratedFileUrl}.", generatedFileUrl); + return await CacheFailureAsync(failureCacheKey); + } + } + + private ResolvedSourceMap Resolve(SourceMapArtifact artifact, byte[] content) + => new(artifact, SourceMapDocument.Parse(content, _options.MaximumMappingSegments)); + + private bool ShouldRefresh(SourceMapStorage.StoredSourceMap? stored, Uri generatedFileUri) + => stored is not null + && stored.Artifact.IsAutoDownloaded + && _options.EnableAutoDownload + && String.Equals(generatedFileUri.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase) + && _timeProvider.GetUtcNow().UtcDateTime - stored.Artifact.CreatedUtc >= _options.AutoDownloadRefreshInterval; + + private Task ClearCachesAsync(string projectId, string generatedFileUrl) + { + _parsedSourceMaps.Remove(GetMemoryCacheKey(projectId, generatedFileUrl)); + return _cache.RemoveAsync(GetFailureCacheKey(projectId, generatedFileUrl)); + } + + private async Task CacheFailureAsync(string failureCacheKey) + { + await _cache.SetAsync(failureCacheKey, true, FailureCacheLifetime); + return null; + } + + private async Task TryAcquireGlobalDownloadSlotAsync(string artifactId, CancellationToken cancellationToken) + { + int start = (int)(Convert.ToUInt32(artifactId[..8], 16) % _options.MaximumConcurrentDownloadsGlobally); + for (int offset = 0; offset < _options.MaximumConcurrentDownloadsGlobally; offset++) + { + cancellationToken.ThrowIfCancellationRequested(); + int slot = (start + offset) % _options.MaximumConcurrentDownloadsGlobally; + var globalDownloadSlot = await _lockProvider.TryAcquireAsync( + $"source-maps:download-slot:{slot}", + _options.RequestTimeout + TimeSpan.FromSeconds(1), + TimeSpan.Zero); + if (globalDownloadSlot is not null) + return globalDownloadSlot; + } + + return null; + } + + public static bool TryNormalizeGeneratedFileUrl(string? value, bool requireHttps, out Uri uri) + { + uri = null!; + if (String.IsNullOrWhiteSpace(value) || !Uri.TryCreate(value.Trim(), UriKind.Absolute, out var parsedUri)) + return false; + + bool validScheme = String.Equals(parsedUri.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase) + || (!requireHttps && String.Equals(parsedUri.Scheme, Uri.UriSchemeHttp, StringComparison.OrdinalIgnoreCase)); + if (!validScheme || !String.IsNullOrEmpty(parsedUri.UserInfo) || !String.IsNullOrEmpty(parsedUri.Fragment)) + return false; + + uri = parsedUri; + return true; + } + + private static bool IsArtifactId(string value) => value.Length == 64 && value.All(Uri.IsHexDigit); + private static string GetArtifactId(string generatedFileUrl) => generatedFileUrl.ToSHA256(); + private static string GetArtifactLockKey(string projectId, string artifactId) => $"source-maps:artifact:{projectId}:{artifactId}"; + private static string GetMemoryCacheKey(string projectId, string generatedFileUrl) => $"{projectId}:{generatedFileUrl}"; + private static string GetFailureCacheKey(string projectId, string generatedFileUrl) => $"source-maps:failure:{projectId}:{generatedFileUrl.ToSHA256()}"; + + private static string? GetDownloadedFileName(string? sourceMapUrl) + { + if (sourceMapUrl is null || !Uri.TryCreate(sourceMapUrl, UriKind.Absolute, out var uri)) + return null; + return Path.GetFileName(uri.AbsolutePath); + } + + private sealed record ResolvedSourceMap(SourceMapArtifact Artifact, SourceMapDocument Document); + private sealed record ParsedSourceMapCacheRegistration(SourceMapService Service, ResolvedSourceMap SourceMap); +} diff --git a/src/Exceptionless.Core/Services/SourceMaps/SourceMapStorage.cs b/src/Exceptionless.Core/Services/SourceMaps/SourceMapStorage.cs new file mode 100644 index 0000000000..42fee614d4 --- /dev/null +++ b/src/Exceptionless.Core/Services/SourceMaps/SourceMapStorage.cs @@ -0,0 +1,132 @@ +using System.Security.Cryptography; +using System.Text.Json; +using Foundatio.Storage; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services.SourceMaps; + +internal sealed class SourceMapStorage +{ + private const string RootPath = "source-maps"; + private readonly IFileStorage _storage; + private readonly JsonSerializerOptions _serializerOptions; + private readonly ILogger _logger; + + public SourceMapStorage(IFileStorage storage, JsonSerializerOptions serializerOptions, ILogger logger) + { + _storage = storage; + _serializerOptions = serializerOptions; + _logger = logger; + } + + public async Task SaveAsync(string projectId, SourceMapArtifact artifact, byte[] sourceMap, CancellationToken cancellationToken) + { + string metadataPath = GetMetadataPath(projectId, artifact.Id); + var previousMetadata = await ReadMetadataAsync(metadataPath, cancellationToken); + string mapFileName = $"{artifact.Id}-{Convert.ToHexString(SHA256.HashData(sourceMap)).ToLowerInvariant()}.map"; + string mapPath = GetMapPath(projectId, mapFileName); + + await using (var mapStream = new MemoryStream(sourceMap, writable: false)) + { + if (!await _storage.SaveFileAsync(mapPath, mapStream, cancellationToken)) + throw new IOException("Unable to save the source map."); + } + + try + { + var metadata = new StoredSourceMapMetadata(artifact, mapFileName); + byte[] metadataBytes = JsonSerializer.SerializeToUtf8Bytes(metadata, _serializerOptions); + await using var metadataStream = new MemoryStream(metadataBytes, writable: false); + if (!await _storage.SaveFileAsync(metadataPath, metadataStream, cancellationToken)) + throw new IOException("Unable to save the source map metadata."); + } + catch + { + if (previousMetadata?.MapFileName != mapFileName) + await _storage.DeleteFileAsync(mapPath, CancellationToken.None); + throw; + } + + if (previousMetadata is not null && previousMetadata.MapFileName != mapFileName) + await _storage.DeleteFileAsync(GetMapPath(projectId, previousMetadata.MapFileName), CancellationToken.None); + } + + public async Task GetAsync(string projectId, string artifactId, int maximumBytes, CancellationToken cancellationToken) + { + var metadata = await ReadMetadataAsync(GetMetadataPath(projectId, artifactId), cancellationToken); + if (metadata is null || !IsValidMapFileName(artifactId, metadata.MapFileName)) + return null; + + string mapPath = GetMapPath(projectId, metadata.MapFileName); + if (!await _storage.ExistsAsync(mapPath)) + return null; + + await using var stream = await _storage.GetFileStreamAsync(mapPath, StreamMode.Read, cancellationToken); + if (stream is null) + return null; + + byte[] content = await SourceMapContent.ReadLimitedAsync(stream, maximumBytes, cancellationToken); + return new StoredSourceMap(metadata.Artifact, content); + } + + public async Task> GetArtifactsAsync(string projectId, CancellationToken cancellationToken) + { + var files = await _storage.GetFileListAsync($"{RootPath}/{projectId}/*.json", cancellationToken: cancellationToken); + var artifacts = new List(files.Count); + foreach (var file in files) + { + try + { + var metadata = await ReadMetadataAsync(file.Path, cancellationToken); + if (metadata is not null) + artifacts.Add(metadata.Artifact); + } + catch (Exception ex) when (ex is JsonException or IOException) + { + _logger.LogWarning(ex, "Unable to read source map metadata {SourceMapMetadataPath}.", file.Path); + } + } + + return artifacts.OrderByDescending(artifact => artifact.CreatedUtc).ToArray(); + } + + public async Task DeleteAsync(string projectId, string artifactId, CancellationToken cancellationToken) + { + string metadataPath = GetMetadataPath(projectId, artifactId); + var metadata = await ReadMetadataAsync(metadataPath, cancellationToken); + bool mapDeleted = false; + if (metadata is not null && IsValidMapFileName(artifactId, metadata.MapFileName)) + mapDeleted = await _storage.DeleteFileAsync(GetMapPath(projectId, metadata.MapFileName), cancellationToken); + else + await _storage.DeleteFilesAsync($"{RootPath}/{projectId}/{artifactId}-*.map", cancellationToken); + + bool metadataDeleted = await _storage.DeleteFileAsync(metadataPath, cancellationToken); + return mapDeleted || metadataDeleted ? metadata?.Artifact : null; + } + + public Task DeleteAllAsync(string projectId, CancellationToken cancellationToken) + => _storage.DeleteFilesAsync($"{RootPath}/{projectId}/*", cancellationToken); + + private async Task ReadMetadataAsync(string path, CancellationToken cancellationToken) + { + if (!await _storage.ExistsAsync(path)) + return null; + + await using var stream = await _storage.GetFileStreamAsync(path, StreamMode.Read, cancellationToken); + return stream is null + ? null + : await JsonSerializer.DeserializeAsync(stream, _serializerOptions, cancellationToken); + } + + private static bool IsValidMapFileName(string artifactId, string mapFileName) + => mapFileName.Length == artifactId.Length + 1 + 64 + 4 + && mapFileName.StartsWith(artifactId + '-', StringComparison.Ordinal) + && mapFileName.EndsWith(".map", StringComparison.Ordinal) + && mapFileName.AsSpan(artifactId.Length + 1, 64).ToArray().All(character => Uri.IsHexDigit(character)); + + private static string GetMetadataPath(string projectId, string artifactId) => $"{RootPath}/{projectId}/{artifactId}.json"; + private static string GetMapPath(string projectId, string mapFileName) => $"{RootPath}/{projectId}/{mapFileName}"; + + internal sealed record StoredSourceMap(SourceMapArtifact Artifact, byte[] Content); + private sealed record StoredSourceMapMetadata(SourceMapArtifact Artifact, string MapFileName); +} diff --git a/src/Exceptionless.Web/Api/ApiEndpoints.cs b/src/Exceptionless.Web/Api/ApiEndpoints.cs index a804c7dd7d..1c649dcb08 100644 --- a/src/Exceptionless.Web/Api/ApiEndpoints.cs +++ b/src/Exceptionless.Web/Api/ApiEndpoints.cs @@ -23,6 +23,7 @@ public static WebApplication MapApiEndpoints(this WebApplication app) app.MapOAuthApplicationEndpoints(); app.MapOAuthEndpoints(); app.MapEventEndpoints(); + app.MapSourceMapEndpoints(); app.MapMediatorEndpoints(); return app; diff --git a/src/Exceptionless.Web/Api/Endpoints/SourceMapEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/SourceMapEndpoints.cs new file mode 100644 index 0000000000..402fac4e80 --- /dev/null +++ b/src/Exceptionless.Web/Api/Endpoints/SourceMapEndpoints.cs @@ -0,0 +1,183 @@ +using System.Text.Json; +using Exceptionless.Core.Authorization; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services.SourceMaps; +using Exceptionless.Web.Extensions; +using Exceptionless.Web.Utility.OpenApi; +using Foundatio.Repositories; +using Microsoft.AspNetCore.Mvc; +using HttpIResult = Microsoft.AspNetCore.Http.IResult; +using HttpResults = Microsoft.AspNetCore.Http.Results; + +namespace Exceptionless.Web.Api.Endpoints; + +public static class SourceMapEndpoints +{ + public static IEndpointRouteBuilder MapSourceMapEndpoints(this IEndpointRouteBuilder endpoints) + { + var group = endpoints.MapGroup("api/v2") + .WithTags("Source Map"); + + group.MapGet("projects/{projectId:objectid}/source-maps", GetAsync) + .RequireAuthorization(AuthorizationRoles.UserPolicy) + .Produces>() + .ProducesProblem(StatusCodes.Status404NotFound) + .WithSummary("Get source maps for a project") + .WithMetadata(new EndpointDocumentation { + ParameterDescriptions = new() { + ["projectId"] = "The identifier of the project.", + }, + ResponseDescriptions = new() { + ["404"] = "The project could not be found.", + } + }); + + group.MapPost("projects/{projectId:objectid}/source-maps", PostAsync) + .RequireAuthorization(AuthorizationRoles.SourceMapsWritePolicy) + .Accepts("multipart/form-data") + .Produces(StatusCodes.Status201Created) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) + .WithMetadata( + new MultipartFileUploadAttribute("file", "generated_file_url") { FileDescription = "The source map file to upload." }, + new RequestSizeLimitAttribute(SourceMapService.MaximumUploadRequestSize), + new RequestFormLimitsAttribute { MultipartBodyLengthLimit = SourceMapService.MaximumUploadRequestSize }) + .WithSummary("Upload a source map for a generated JavaScript file") + .WithMetadata(new EndpointDocumentation { + ParameterDescriptions = new() { + ["projectId"] = "The identifier of the project.", + }, + ResponseDescriptions = new() { + ["201"] = "Created", + ["404"] = "The project could not be found.", + ["422"] = "The source map file or generated file URL is invalid.", + } + }) + .DisableAntiforgery(); + + group.MapDelete("projects/{projectId:objectid}/source-maps/{sourceMapId}", DeleteAsync) + .RequireAuthorization(AuthorizationRoles.UserPolicy) + .Produces(StatusCodes.Status204NoContent) + .ProducesProblem(StatusCodes.Status404NotFound) + .WithSummary("Delete a source map from a project") + .WithMetadata(new EndpointDocumentation { + ParameterDescriptions = new() { + ["projectId"] = "The identifier of the project.", + ["sourceMapId"] = "The identifier of the source map.", + }, + ResponseDescriptions = new() { + ["404"] = "The project or source map could not be found.", + } + }); + + return endpoints; + } + + private static async Task GetAsync( + string projectId, + HttpContext httpContext, + IProjectRepository projectRepository, + SourceMapService sourceMapService, + CancellationToken cancellationToken) + { + if (!await CanAccessProjectAsync(projectId, httpContext, projectRepository)) + return HttpResults.NotFound(); + + return HttpResults.Ok(await sourceMapService.GetArtifactsAsync(projectId, cancellationToken)); + } + + private static async Task PostAsync( + string projectId, + HttpContext httpContext, + IProjectRepository projectRepository, + SourceMapService sourceMapService, + CancellationToken cancellationToken) + { + if (!await CanAccessProjectAsync(projectId, httpContext, projectRepository)) + return HttpResults.NotFound(); + + var (form, formError) = await ReadFormAsync(httpContext.Request, cancellationToken); + if (formError is not null) + return formError; + + var errors = new Dictionary(); + string? generatedFileUrl = form!["generated_file_url"].FirstOrDefault(); + var file = form.Files.GetFile("file"); + + if (String.IsNullOrWhiteSpace(generatedFileUrl)) + errors["generated_file_url"] = ["The generated file URL is required."]; + if (file is null || file.Length == 0) + errors["file"] = ["A source map file is required."]; + if (errors.Count > 0) + return HttpResults.ValidationProblem(errors, statusCode: StatusCodes.Status422UnprocessableEntity); + + try + { + await using var stream = file!.OpenReadStream(); + var artifact = await sourceMapService.SaveUploadedAsync(projectId, generatedFileUrl!, file.FileName, stream, cancellationToken); + return HttpResults.Json(artifact, statusCode: StatusCodes.Status201Created); + } + catch (ArgumentException ex) + { + errors["generated_file_url"] = [ex.Message]; + } + catch (Exception ex) when (ex is JsonException or InvalidOperationException) + { + errors["file"] = [ex.Message]; + } + + return HttpResults.ValidationProblem(errors, statusCode: StatusCodes.Status422UnprocessableEntity); + } + + internal static async Task<(IFormCollection? Form, HttpIResult? Error)> ReadFormAsync( + HttpRequest request, + CancellationToken cancellationToken) + { + if (!request.HasFormContentType) + { + return (null, HttpResults.ValidationProblem( + new Dictionary { ["file"] = ["The request must use multipart/form-data."] }, + statusCode: StatusCodes.Status422UnprocessableEntity)); + } + + try + { + return (await request.ReadFormAsync(cancellationToken), null); + } + catch (Exception ex) when (ex is InvalidDataException or BadHttpRequestException) + { + return (null, HttpResults.ValidationProblem( + new Dictionary { ["file"] = ["The multipart form data is invalid."] }, + statusCode: StatusCodes.Status422UnprocessableEntity)); + } + } + + private static async Task DeleteAsync( + string projectId, + string sourceMapId, + HttpContext httpContext, + IProjectRepository projectRepository, + SourceMapService sourceMapService, + CancellationToken cancellationToken) + { + if (!await CanAccessProjectAsync(projectId, httpContext, projectRepository)) + return HttpResults.NotFound(); + + return await sourceMapService.DeleteArtifactAsync(projectId, sourceMapId, cancellationToken) + ? HttpResults.NoContent() + : HttpResults.NotFound(); + } + + private static async Task CanAccessProjectAsync(string projectId, HttpContext httpContext, IProjectRepository projectRepository) + { + var project = await projectRepository.GetByIdAsync(projectId, options => options.Cache()); + if (project is null || !httpContext.Request.CanAccessOrganization(project.OrganizationId)) + return false; + + string? tokenProjectId = httpContext.Request.GetProjectId(); + if (httpContext.User.IsInRole(AuthorizationRoles.SourceMapsWrite) && !httpContext.User.IsInRole(AuthorizationRoles.User)) + return String.Equals(tokenProjectId, projectId, StringComparison.Ordinal); + + return String.IsNullOrEmpty(tokenProjectId) || String.Equals(tokenProjectId, projectId, StringComparison.Ordinal); + } +} diff --git a/src/Exceptionless.Web/Api/Endpoints/TokenEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/TokenEndpoints.cs index de940b0ec1..36bf374960 100644 --- a/src/Exceptionless.Web/Api/Endpoints/TokenEndpoints.cs +++ b/src/Exceptionless.Web/Api/Endpoints/TokenEndpoints.cs @@ -98,7 +98,7 @@ public static IEndpointRouteBuilder MapTokenEndpoints(this IEndpointRouteBuilder .ProducesProblem(StatusCodes.Status400BadRequest) .ProducesProblem(StatusCodes.Status409Conflict) .WithSummary("Create") - .WithDescription("To create a new token, you must specify an organization_id. There are three valid scopes: client, user and admin.") + .WithDescription("To create a new token, you must specify an organization_id and project_id. Valid scopes are client, user, admin, and source-maps:write. The source-maps:write scope requires a project-scoped token.") .WithMetadata(new EndpointDocumentation { RequestBodyDescription = "The token.", ResponseDescriptions = new() { @@ -119,7 +119,7 @@ public static IEndpointRouteBuilder MapTokenEndpoints(this IEndpointRouteBuilder .ProducesProblem(StatusCodes.Status404NotFound) .ProducesProblem(StatusCodes.Status409Conflict) .WithSummary("Create for project") - .WithDescription("This is a helper action that makes it easier to create a token for a specific project. You may also specify a scope when creating a token. There are three valid scopes: client, user and admin.") + .WithDescription("This is a helper action that makes it easier to create a token for a specific project. Valid scopes are client, user, admin, and source-maps:write.") .WithMetadata(new EndpointDocumentation { RequestBodyDescription = "The token.", ParameterDescriptions = new() { diff --git a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs index eb77c82c92..1d5cdd7105 100644 --- a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs @@ -535,6 +535,7 @@ await eventPostService.EnqueueAsync(new EventPost(appOptions.EnableArchive) MediaType = mediaType, OrganizationId = project.OrganizationId, ProjectId = project.Id, + ClientKeyHash = httpContext.Request.GetClientKeyHash(), UserAgent = message.UserAgent }, stream); } @@ -601,6 +602,7 @@ public async Task Handle(SubmitEventByPost message) MediaType = mediaType, OrganizationId = project.OrganizationId, ProjectId = project.Id, + ClientKeyHash = httpContext.Request.GetClientKeyHash(), UserAgent = message.UserAgent, }, requestBody, httpContext.RequestAborted); diff --git a/src/Exceptionless.Web/Api/Handlers/TokenHandler.cs b/src/Exceptionless.Web/Api/Handlers/TokenHandler.cs index 72547f4b6e..1148986b8f 100644 --- a/src/Exceptionless.Web/Api/Handlers/TokenHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/TokenHandler.cs @@ -235,6 +235,9 @@ private async Task> CreateTokenImplAsync(NewToken value) if (value.Scopes.Count == 0) value.Scopes.Add(AuthorizationRoles.Client); + if (value.Scopes.Contains(AuthorizationRoles.SourceMapsWrite) && String.IsNullOrEmpty(value.ProjectId)) + return Result.Invalid(ValidationError.Create("project_id", "The source-maps:write scope requires a project-scoped token.")); + if ((value.Scopes.Contains(AuthorizationRoles.Client) && !hasUserRole) || (value.Scopes.Contains(AuthorizationRoles.User) && !hasUserRole) || (value.Scopes.Contains(AuthorizationRoles.GlobalAdmin) && !hasGlobalAdminRole)) diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/api.svelte.ts index 189193d7cd..85a0120b24 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/api.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/api.svelte.ts @@ -1,8 +1,9 @@ -import type { ClientConfiguration, NewProject, NotificationSettings, UpdateProject, ViewProject } from '$features/projects/models'; +import type { ClientConfiguration, NewProject, NotificationSettings, SourceMapArtifact, UpdateProject, ViewProject } from '$features/projects/models'; import type { StringValueFromBody, WorkInProgressResult } from '$features/shared/models'; import type { WebSocketMessageValue } from '$features/websockets/models'; import { accessToken } from '$features/auth/index.svelte'; +import { fetchApiJson } from '$features/shared/api/api.svelte'; import { type FetchClientResponse, type ProblemDetails, useFetchClient } from '@exceptionless/fetchclient'; import { createMutation, createQuery, QueryClient, useQueryClient } from '@tanstack/svelte-query'; @@ -44,6 +45,7 @@ export const queryKeys = { putIntegrationNotificationSettings: (id: string | undefined, integration: string) => [...queryKeys.id(id), integration, 'put-notification-settings'] as const, resetData: (id: string | undefined) => [...queryKeys.id(id), 'reset-data'] as const, + sourceMaps: (id: string | undefined) => [...queryKeys.id(id), 'source-maps'] as const, type: ['Project'] as const, userNotificationSettings: (id: string | undefined, userId: string | undefined) => [...queryKeys.id(id), userId, 'notification-settings'] as const }; @@ -188,6 +190,17 @@ export interface ResetDataRequest { }; } +export interface SourceMapRequest { + route: { + id: string | undefined; + }; +} + +export interface SourceMapUpload { + file: File; + generated_file_url: string; +} + export interface UpdateProjectRequest { route: { id: string; @@ -283,6 +296,23 @@ export function deleteSlack(request: DeleteSlackRequest) { })); } +export function deleteSourceMapMutation(request: SourceMapRequest) { + const queryClient = useQueryClient(); + + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.id, + mutationFn: async (sourceMapId: string) => { + const client = useFetchClient(); + const response = await client.delete(`projects/${request.route.id}/source-maps/${sourceMapId}`); + return response.ok; + }, + mutationKey: [...queryKeys.sourceMaps(request.route.id), 'delete'], + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.sourceMaps(request.route.id) }); + } + })); +} + export function generateSampleData(request: GenerateSampleDataRequest) { return createMutation(() => ({ enabled: () => !!accessToken.current && !!request.route.id, @@ -412,6 +442,18 @@ export function getProjectUserNotificationSettings(request: GetProjectUserNotifi })); } +export function getSourceMapsQuery(request: SourceMapRequest) { + return createQuery(() => ({ + enabled: () => !!accessToken.current && !!request.route.id, + queryFn: async ({ signal }: { signal: AbortSignal }) => { + const client = useFetchClient(); + const response = await client.getJSON(`projects/${request.route.id}/source-maps`, { signal }); + return response.data!; + }, + queryKey: queryKeys.sourceMaps(request.route.id) + })); +} + export function postProject() { const queryClient = useQueryClient(); @@ -530,6 +572,27 @@ export function postSlack(request: PostSlackRequest) { })); } +export function postSourceMapMutation(request: SourceMapRequest) { + const queryClient = useQueryClient(); + + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.id, + mutationFn: async (sourceMap: SourceMapUpload) => { + const data = new FormData(); + data.append('generated_file_url', sourceMap.generated_file_url); + data.append('file', sourceMap.file); + return await fetchApiJson(`projects/${request.route.id}/source-maps`, { + body: data, + method: 'POST' + }); + }, + mutationKey: [...queryKeys.sourceMaps(request.route.id), 'post'], + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.sourceMaps(request.route.id) }); + } + })); +} + export function putProjectIntegrationNotificationSettings(request: PutProjectIntegrationNotificationSettingsRequest) { const queryClient = useQueryClient(); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts index a055dfb212..b9c8b05831 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts @@ -4,3 +4,13 @@ export interface ClientConfigurationSetting { key: string; value: string; } + +export interface SourceMapArtifact { + created_utc: string; + file_name?: string; + generated_file_url: string; + id: string; + is_auto_downloaded: boolean; + size: number; + source_map_url?: string; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts index e495e7dd4f..8c9b789bf3 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts @@ -5,7 +5,7 @@ import { type NotificationSettingsFormData, NotificationSettingsSchema } from '$generated/schemas'; -import { type infer as Infer, object, string } from 'zod'; +import { custom, type infer as Infer, object, string } from 'zod'; export { type NewProjectFormData, NewProjectSchema, type NotificationSettingsFormData, NotificationSettingsSchema }; @@ -17,3 +17,9 @@ export type ClientConfigurationSettingFormData = Infer; + +export const SourceMapUploadSchema = object({ + file: custom((value) => typeof File !== 'undefined' && value instanceof File, 'Source map file is required'), + generated_file_url: string().url('Enter the absolute URL of the generated JavaScript file') +}); +export type SourceMapUploadFormData = Infer; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/disable-token-dialog.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/disable-token-dialog.svelte index 9201f757d4..b53df29ac8 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/disable-token-dialog.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/disable-token-dialog.svelte @@ -20,14 +20,14 @@ - Disable API Key + Disable Token Are you sure you want to disable "{id}" {#if notes}({notes}){/if}? Cancel - Disable API Key + Disable Token diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/enable-token-dialog.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/enable-token-dialog.svelte index a0679bad08..10d581c7bd 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/enable-token-dialog.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/enable-token-dialog.svelte @@ -19,14 +19,14 @@ - Enable API Key + Enable Token Are you sure you want to enable "{id}" {#if notes}({notes}){/if}? Cancel - Enable API Key + Enable Token diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/remove-token-dialog.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/remove-token-dialog.svelte index 9382fd1d5f..49041755bf 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/remove-token-dialog.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/remove-token-dialog.svelte @@ -20,14 +20,14 @@ - Delete API Key + Delete Token Are you sure you want to delete "{id}" {#if notes}({notes}){/if}? Cancel - Delete API Key + Delete Token diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/update-token-notes-dialog.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/update-token-notes-dialog.svelte index f375843862..5226ef59ef 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/update-token-notes-dialog.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/dialogs/update-token-notes-dialog.svelte @@ -57,7 +57,7 @@ }} > - API Key Notes + Token Notes state.errors}> diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/options.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/options.svelte.ts index 8142bc8593..fef24c4c75 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/options.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/options.svelte.ts @@ -5,6 +5,7 @@ import type { CreateQueryResult } from '@tanstack/svelte-query'; import { getSharedTableOptions } from '$features/shared/table.svelte'; import TokenActionsCell from '$features/tokens/components/table/token-actions-cell.svelte'; import TokenIdCell from '$features/tokens/components/table/token-id-cell.svelte'; +import TokenScopesCell from '$features/tokens/components/table/token-scopes-cell.svelte'; import { type ColumnDef, renderComponent, type StockFeatures } from '@tanstack/svelte-table'; import type { GetProjectTokensParams } from '../../api.svelte'; @@ -16,7 +17,17 @@ export function getColumns(): ColumnDef renderComponent(TokenIdCell, { token: info.row.original }), enableHiding: false, enableSorting: false, - header: 'API Key', + header: 'Token', + meta: { + class: 'w-45' + } + }, + { + accessorKey: 'scopes', + cell: (info) => renderComponent(TokenScopesCell, { scopes: info.getValue() }), + enableHiding: true, + enableSorting: false, + header: 'Scope', meta: { class: 'w-45' } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-actions-cell.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-actions-cell.svelte index 0281e538b0..3f6e22616e 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-actions-cell.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-actions-cell.svelte @@ -31,7 +31,7 @@ const clipboard = new UseClipboard(); async function copyToClipboard() { - await clipboard.copy(token.id); // Assuming token.apiKey holds the API key to copy + await clipboard.copy(token.id); if (clipboard.copied) { toast.success('Copy to clipboard succeeded'); } else { @@ -49,7 +49,7 @@ async function updateDisabled(is_disabled: boolean) { await updateToken.mutateAsync({ is_disabled, notes: token.notes }); - toast.success(`Successfully ${is_disabled ? 'disabled' : 'enabled'} API key`); + toast.success(`Successfully ${is_disabled ? 'disabled' : 'enabled'} token`); } function onEnableDisableClick() { @@ -75,7 +75,7 @@ async function remove() { await removeToken.mutateAsync(); - toast.success('Successfully deleted API key'); + toast.success('Successfully deleted token'); } @@ -91,7 +91,7 @@ - Copy Api Key + Copy Token (showUpdateNotesDialog = true)}> @@ -100,10 +100,10 @@ {#if token.is_disabled} - Enable API Key + Enable Token {:else} - Disable API Key + Disable Token {/if} (showRemoveTokenDialog = true)} disabled={removeToken.isPending}> diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-scopes-cell.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-scopes-cell.svelte new file mode 100644 index 0000000000..712c707746 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/components/table/token-scopes-cell.svelte @@ -0,0 +1,15 @@ + + +
+ {#each scopes as scope (scope)} + {scope} + {/each} +
diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.test.ts new file mode 100644 index 0000000000..666b93b963 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it } from 'vitest'; + +import { createProjectToken } from './project-token'; + +describe('createProjectToken', () => { + it('creates a project-scoped source map upload token', () => { + const token = createProjectToken('organization-id', 'project-id', 'source-maps:write'); + + expect(token).toEqual({ + notes: 'Source map deployment', + organization_id: 'organization-id', + project_id: 'project-id', + scopes: ['source-maps:write'] + }); + }); + + it('preserves normal client API key creation', () => { + const token = createProjectToken('organization-id', 'project-id', 'client'); + + expect(token).toEqual({ + notes: undefined, + organization_id: 'organization-id', + project_id: 'project-id', + scopes: ['client'] + }); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.ts new file mode 100644 index 0000000000..286abcbe1c --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/tokens/project-token.ts @@ -0,0 +1,12 @@ +import type { NewToken } from '$features/tokens/models'; + +export type ProjectTokenScope = 'client' | 'source-maps:write'; + +export function createProjectToken(organizationId: string, projectId: string, scope: ProjectTokenScope): NewToken { + return { + notes: scope === 'source-maps:write' ? 'Source map deployment' : undefined, + organization_id: organizationId, + project_id: projectId, + scopes: [scope] + }; +} diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/api-keys/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/api-keys/+page.svelte index 4c69ffdaed..f25ff8cbed 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/api-keys/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/api-keys/+page.svelte @@ -1,16 +1,20 @@ + +
+
+

Source Map Discovery

+ + Exceptionless automatically downloads publicly available source maps over HTTPS using the generated file's SourceMap header or sourceMappingURL. + Upload a map only when it is private or not published with the generated JavaScript. + +
+ +
+

Upload Source Map

+ + Automating uploads from CI/CD? Create a project-scoped + source map upload token and use it with the source map API. + +
{ + event.preventDefault(); + event.stopPropagation(); + form.handleSubmit(); + }} + > + state.errors}> + {#snippet children(errors)} +
+ {/snippet} + + + + {#snippet children(field)} + + Generated JavaScript URL + field.handleChange(event.currentTarget.value)} + aria-invalid={ariaInvalid(field)} + /> + + + {/snippet} + + + + {#snippet children(field)} + + Source map + field.handleChange(event.currentTarget.files?.[0] ?? null)} + aria-invalid={ariaInvalid(field)} + /> + + + {/snippet} + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + +
+
+ +
+

Available Source Maps

+
+ + + + Generated file + Source + Size + Added + Actions + + + + {#if sourceMapsQuery.isLoading} + + {:else if sourceMapsQuery.isError} + + {:else if !sourceMapsQuery.data?.length} + No source maps have been discovered or uploaded yet. + {:else} + {#each sourceMapsQuery.data as sourceMap (sourceMap.id)} + + {sourceMap.generated_file_url} + + + {sourceMap.is_auto_downloaded ? 'Automatic' : 'Uploaded'} + + {#if sourceMap.file_name}
{sourceMap.file_name}
{/if} + + + + + + + + {/each} + {/if} + + +
+
+
+ + + + + Delete Source Map + + Delete the source map for {sourceMapToDelete?.generated_file_url}? Future events will use + automatic discovery or retain their generated stack frames. + + + + Cancel + Delete Source Map + + + diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/source-maps/source-maps-page.svelte.test.ts b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/source-maps/source-maps-page.svelte.test.ts new file mode 100644 index 0000000000..8c69efebf7 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/source-maps/source-maps-page.svelte.test.ts @@ -0,0 +1,27 @@ +import { render, screen } from '@testing-library/svelte'; +import { describe, expect, it, vi } from 'vitest'; + +import SourceMapsPage from './+page.svelte'; + +vi.mock('$app/paths', () => ({ + resolve: (path: string) => path +})); + +vi.mock('$app/state', () => ({ + page: { params: { projectId: 'project-id' } } +})); + +vi.mock('$features/projects/api.svelte', () => ({ + deleteSourceMapMutation: () => ({ isPending: false, mutateAsync: vi.fn() }), + getSourceMapsQuery: () => ({ data: [], isError: false, isLoading: false }), + postSourceMapMutation: () => ({ mutateAsync: vi.fn() }) +})); + +describe('Source Maps page', () => { + it('renders the source map upload form', () => { + render(SourceMapsPage); + + expect(screen.getByRole('heading', { name: 'Upload Source Map' })).toBeTruthy(); + expect(screen.getByLabelText('Source map').getAttribute('type')).toBe('file'); + }); +}); diff --git a/src/Exceptionless.Web/Extensions/HttpExtensions.cs b/src/Exceptionless.Web/Extensions/HttpExtensions.cs index a53cfe1fc2..c38281f3ae 100644 --- a/src/Exceptionless.Web/Extensions/HttpExtensions.cs +++ b/src/Exceptionless.Web/Extensions/HttpExtensions.cs @@ -140,6 +140,15 @@ public static ICollection GetAssociatedOrganizationIds(this HttpRequest return request.HttpContext.Connection.RemoteIpAddress?.ToString(); } + public static string? GetClientKeyHash(this HttpRequest request) + { + ArgumentNullException.ThrowIfNull(request); + + return request.HttpContext.Items.TryGetAndReturn("ApiKey") is string apiKey + ? apiKey.ToSHA256() + : null; + } + public static string? GetQueryString(this HttpRequest request, string key) { ArgumentNullException.ThrowIfNull(request); diff --git a/src/Exceptionless.Web/Program.cs b/src/Exceptionless.Web/Program.cs index 455d3d00c3..0faeebe0e0 100644 --- a/src/Exceptionless.Web/Program.cs +++ b/src/Exceptionless.Web/Program.cs @@ -157,6 +157,7 @@ public static async Task Main(string[] args) o.AddPolicy(AuthorizationRoles.StacksReadPolicy, policy => policy.RequireAssertion(context => context.User.IsInRole(AuthorizationRoles.User) || context.User.IsInRole(AuthorizationRoles.StacksRead))); o.AddPolicy(AuthorizationRoles.StacksWritePolicy, policy => policy.RequireAssertion(context => context.User.IsInRole(AuthorizationRoles.User) || context.User.IsInRole(AuthorizationRoles.StacksWrite))); o.AddPolicy(AuthorizationRoles.EventsReadPolicy, policy => policy.RequireAssertion(context => context.User.IsInRole(AuthorizationRoles.User) || context.User.IsInRole(AuthorizationRoles.EventsRead))); + o.AddPolicy(AuthorizationRoles.SourceMapsWritePolicy, policy => policy.RequireAssertion(context => context.User.IsInRole(AuthorizationRoles.User) || context.User.IsInRole(AuthorizationRoles.SourceMapsWrite))); }); builder.Services.AddRouting(r => diff --git a/src/Exceptionless.Web/Utility/OpenApi/RequestBodyContentOperationTransformer.cs b/src/Exceptionless.Web/Utility/OpenApi/RequestBodyContentOperationTransformer.cs index 54077a0486..ca35db63e4 100644 --- a/src/Exceptionless.Web/Utility/OpenApi/RequestBodyContentOperationTransformer.cs +++ b/src/Exceptionless.Web/Utility/OpenApi/RequestBodyContentOperationTransformer.cs @@ -36,20 +36,7 @@ public Task TransformAsync(OpenApiOperation operation, OpenApiOperationTransform { ["multipart/form-data"] = new() { - Schema = new OpenApiSchema - { - Type = JsonSchemaType.Object, - Required = new HashSet { multipartFileUploadAttribute.FileParameterName }, - Properties = new Dictionary - { - [multipartFileUploadAttribute.FileParameterName] = new OpenApiSchema - { - Type = JsonSchemaType.String, - Format = "binary", - Description = "The image file to upload." - } - } - } + Schema = CreateMultipartSchema(multipartFileUploadAttribute) } } }; @@ -87,6 +74,33 @@ public Task TransformAsync(OpenApiOperation operation, OpenApiOperationTransform return Task.CompletedTask; } + + private static OpenApiSchema CreateMultipartSchema(MultipartFileUploadAttribute attribute) + { + var required = new HashSet { attribute.FileParameterName }; + var properties = new Dictionary + { + [attribute.FileParameterName] = new OpenApiSchema + { + Type = JsonSchemaType.String, + Format = "binary", + Description = attribute.FileDescription + } + }; + + foreach (string parameterName in attribute.RequiredStringParameterNames) + { + required.Add(parameterName); + properties[parameterName] = new OpenApiSchema { Type = JsonSchemaType.String }; + } + + return new OpenApiSchema + { + Type = JsonSchemaType.Object, + Required = required, + Properties = properties + }; + } } /// @@ -110,9 +124,12 @@ public RequestBodyContentAttribute(params string[] contentTypes) public class MultipartFileUploadAttribute : Attribute { public string FileParameterName { get; } + public string FileDescription { get; init; } = "The image file to upload."; + public IReadOnlyCollection RequiredStringParameterNames { get; } - public MultipartFileUploadAttribute(string fileParameterName = "file") + public MultipartFileUploadAttribute(string fileParameterName = "file", params string[] requiredStringParameterNames) { FileParameterName = fileParameterName; + RequiredStringParameterNames = requiredStringParameterNames; } } diff --git a/tests/Exceptionless.Tests/Api/Data/endpoint-manifest.json b/tests/Exceptionless.Tests/Api/Data/endpoint-manifest.json index 65f2578aba..b2ea4461e4 100644 --- a/tests/Exceptionless.Tests/Api/Data/endpoint-manifest.json +++ b/tests/Exceptionless.Tests/Api/Data/endpoint-manifest.json @@ -1979,6 +1979,48 @@ "authorizationRoles": [], "authenticationSchemes": [] }, + { + "method": "GET", + "route": "/api/v2/projects/{projectId:objectid}/source-maps", + "displayName": "HTTP: GET api/v2/projects/{projectId:objectid}/source-maps =\u003E GetAsync", + "tags": [ + "Source Map" + ], + "allowAnonymous": false, + "authorizationPolicies": [ + "UserPolicy" + ], + "authorizationRoles": [], + "authenticationSchemes": [] + }, + { + "method": "POST", + "route": "/api/v2/projects/{projectId:objectid}/source-maps", + "displayName": "HTTP: POST api/v2/projects/{projectId:objectid}/source-maps =\u003E PostAsync", + "tags": [ + "Source Map" + ], + "allowAnonymous": false, + "authorizationPolicies": [ + "SourceMapsWritePolicy" + ], + "authorizationRoles": [], + "authenticationSchemes": [] + }, + { + "method": "DELETE", + "route": "/api/v2/projects/{projectId:objectid}/source-maps/{sourceMapId}", + "displayName": "HTTP: DELETE api/v2/projects/{projectId:objectid}/source-maps/{sourceMapId} =\u003E DeleteAsync", + "tags": [ + "Source Map" + ], + "allowAnonymous": false, + "authorizationPolicies": [ + "UserPolicy" + ], + "authorizationRoles": [], + "authenticationSchemes": [] + }, { "method": "GET", "route": "/api/v2/projects/{projectId:objectid}/stacks", diff --git a/tests/Exceptionless.Tests/Api/Data/openapi.json b/tests/Exceptionless.Tests/Api/Data/openapi.json index 68ec2364ce..1b41d2f877 100644 --- a/tests/Exceptionless.Tests/Api/Data/openapi.json +++ b/tests/Exceptionless.Tests/Api/Data/openapi.json @@ -1848,7 +1848,7 @@ "Token" ], "summary": "Create for project", - "description": "This is a helper action that makes it easier to create a token for a specific project. You may also specify a scope when creating a token. There are three valid scopes: client, user and admin.", + "description": "This is a helper action that makes it easier to create a token for a specific project. Valid scopes are client, user, admin, and source-maps:write.", "parameters": [ { "name": "projectId", @@ -2157,7 +2157,7 @@ "Token" ], "summary": "Create", - "description": "To create a new token, you must specify an organization_id. There are three valid scopes: client, user and admin.", + "description": "To create a new token, you must specify an organization_id and project_id. Valid scopes are client, user, admin, and source-maps:write. The source-maps:write scope requires a project-scoped token.", "requestBody": { "description": "The token.", "content": { @@ -10741,6 +10741,169 @@ } } } + }, + "/api/v2/projects/{projectId}/source-maps": { + "get": { + "tags": [ + "Source Map" + ], + "summary": "Get source maps for a project", + "parameters": [ + { + "name": "projectId", + "in": "path", + "description": "The identifier of the project.", + "required": true, + "schema": { + "pattern": "^[a-zA-Z\\d]{24,36}$", + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "OK", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/SourceMapArtifact" + } + } + } + } + }, + "404": { + "description": "The project could not be found.", + "content": { + "application/problem\u002Bjson": { + "schema": { + "$ref": "#/components/schemas/ProblemDetails" + } + } + } + } + } + }, + "post": { + "tags": [ + "Source Map" + ], + "summary": "Upload a source map for a generated JavaScript file", + "parameters": [ + { + "name": "projectId", + "in": "path", + "description": "The identifier of the project.", + "required": true, + "schema": { + "pattern": "^[a-zA-Z\\d]{24,36}$", + "type": "string" + } + } + ], + "requestBody": { + "content": { + "multipart/form-data": { + "schema": { + "required": [ + "file", + "generated_file_url" + ], + "type": "object", + "properties": { + "file": { + "type": "string", + "description": "The source map file to upload.", + "format": "binary" + }, + "generated_file_url": { + "type": "string" + } + } + } + } + }, + "required": true + }, + "responses": { + "201": { + "description": "Created", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SourceMapArtifact" + } + } + } + }, + "404": { + "description": "The project could not be found.", + "content": { + "application/problem\u002Bjson": { + "schema": { + "$ref": "#/components/schemas/ProblemDetails" + } + } + } + }, + "422": { + "description": "The source map file or generated file URL is invalid.", + "content": { + "application/problem\u002Bjson": { + "schema": { + "$ref": "#/components/schemas/ProblemDetails" + } + } + } + } + } + } + }, + "/api/v2/projects/{projectId}/source-maps/{sourceMapId}": { + "delete": { + "tags": [ + "Source Map" + ], + "summary": "Delete a source map from a project", + "parameters": [ + { + "name": "projectId", + "in": "path", + "description": "The identifier of the project.", + "required": true, + "schema": { + "pattern": "^[a-zA-Z\\d]{24,36}$", + "type": "string" + } + }, + { + "name": "sourceMapId", + "in": "path", + "description": "The identifier of the source map.", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "204": { + "description": "No Content" + }, + "404": { + "description": "The project or source map could not be found.", + "content": { + "application/problem\u002Bjson": { + "schema": { + "$ref": "#/components/schemas/ProblemDetails" + } + } + } + } + } + } } }, "components": { @@ -12124,6 +12287,47 @@ } } }, + "SourceMapArtifact": { + "required": [ + "id", + "generated_file_url", + "size", + "is_auto_downloaded", + "created_utc" + ], + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "generated_file_url": { + "type": "string" + }, + "source_map_url": { + "type": [ + "null", + "string" + ] + }, + "file_name": { + "type": [ + "null", + "string" + ] + }, + "size": { + "type": "integer", + "format": "int64" + }, + "is_auto_downloaded": { + "type": "boolean" + }, + "created_utc": { + "type": "string", + "format": "date-time" + } + } + }, "Stack": { "required": [ "organization_id", @@ -13554,6 +13758,9 @@ }, { "name": "Stack" + }, + { + "name": "Source Map" } ] } \ No newline at end of file diff --git a/tests/Exceptionless.Tests/Api/Endpoints/SourceMapEndpointTests.cs b/tests/Exceptionless.Tests/Api/Endpoints/SourceMapEndpointTests.cs new file mode 100644 index 0000000000..b284ec26f6 --- /dev/null +++ b/tests/Exceptionless.Tests/Api/Endpoints/SourceMapEndpointTests.cs @@ -0,0 +1,263 @@ +using System.Net; +using System.Text; +using Exceptionless.Core.Authorization; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services.SourceMaps; +using Exceptionless.Core.Utility; +using Exceptionless.Tests.Extensions; +using Exceptionless.Tests.Utility; +using Exceptionless.Web.Api.Endpoints; +using Exceptionless.Web.Models; +using FluentRest; +using Foundatio.Repositories; +using Xunit; + +namespace Exceptionless.Tests.Api.Endpoints; + +public sealed class SourceMapEndpointTests : IntegrationTestsBase +{ + private const string GeneratedFileUrl = "https://cdn.example.com/assets/app.min.js"; + private static readonly byte[] SourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["src/app.ts"],"names":["meaningfulFunction"],"mappings":"AAAAA"}"""); + + public SourceMapEndpointTests(ITestOutputHelper output, AppWebHostFactory factory) : base(output, factory) + { + } + + protected override async Task ResetDataAsync() + { + await base.ResetDataAsync(); + await GetService().CreateDataAsync(); + } + + [Fact] + public async Task PostAsync_WithValidSourceMap_CanListAndDeleteArtifact() + { + using var content = CreateSourceMapContent(SourceMap); + + var uploaded = await SendRequestAsAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeCreated()); + + Assert.NotNull(uploaded); + Assert.Equal(GeneratedFileUrl, uploaded.GeneratedFileUrl); + Assert.False(uploaded.IsAutoDownloaded); + + var artifacts = await SendRequestAsAsync>(request => request + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .StatusCodeShouldBeOk()); + Assert.NotNull(artifacts); + Assert.Single(artifacts); + + await SendRequestAsync(request => request + .Delete() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps", uploaded.Id) + .StatusCodeShouldBeNoContent()); + } + + [Fact] + public async Task PostAsync_WithInvalidSourceMap_ReturnsUnprocessableEntity() + { + using var content = CreateSourceMapContent(Encoding.UTF8.GetBytes("not a source map")); + + await SendRequestAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeUnprocessableEntity()); + } + + [Fact] + public async Task PostAsync_WithMissingMappings_ReturnsUnprocessableEntity() + { + using var content = CreateSourceMapContent(Encoding.UTF8.GetBytes("""{"version":3,"sources":[],"names":[]}""")); + + await SendRequestAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeUnprocessableEntity()); + } + + [Fact] + public async Task PostAsync_WithOversizedVersion_ReturnsUnprocessableEntity() + { + using var content = CreateSourceMapContent(Encoding.UTF8.GetBytes("""{"version":999999999999,"sources":[],"names":[],"mappings":""}""")); + + await SendRequestAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeUnprocessableEntity()); + } + + [Fact] + public async Task PostAsync_WithProjectSourceMapsWriteToken_CreatesArtifact() + { + var token = await CreateSourceMapUploadTokenAsync(SampleDataService.TEST_PROJECT_ID); + using var content = CreateSourceMapContent(SourceMap); + + var uploaded = await SendRequestAsAsync(request => request + .Post() + .BearerToken(token.Id) + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeCreated()); + + Assert.NotNull(uploaded); + Assert.Equal(GeneratedFileUrl, uploaded.GeneratedFileUrl); + } + + [Fact] + public async Task PostAsync_WithClientToken_ReturnsForbidden() + { + using var content = CreateSourceMapContent(SourceMap); + + await SendRequestAsync(request => request + .Post() + .AsTestOrganizationClientUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeForbidden()); + } + + [Fact] + public async Task PostAsync_WithSourceMapsWriteTokenForDifferentProject_ReturnsNotFound() + { + var otherProject = await GetService().AddAsync(new Project + { + Name = "Other project in the same organization", + OrganizationId = SampleDataService.TEST_ORG_ID, + NextSummaryEndOfDayTicks = TimeProvider.GetUtcNow().UtcDateTime.Ticks + }, options => options.ImmediateConsistency()); + var token = await CreateSourceMapUploadTokenAsync(SampleDataService.TEST_PROJECT_ID); + using var content = CreateSourceMapContent(SourceMap); + + await SendRequestAsync(request => request + .Post() + .BearerToken(token.Id) + .AppendPaths("projects", otherProject.Id, "source-maps") + .Content(content) + .StatusCodeShouldBeNotFound()); + } + + [Fact] + public async Task PostAsync_WithOrganizationScopedSourceMapsWriteToken_ReturnsNotFound() + { + var utcNow = TimeProvider.GetUtcNow().UtcDateTime; + var token = await GetService().AddAsync(new Token + { + Id = StringExtensions.GetNewToken(), + OrganizationId = SampleDataService.TEST_ORG_ID, + Scopes = [AuthorizationRoles.SourceMapsWrite], + Type = TokenType.Access, + CreatedBy = TestConstants.UserId, + CreatedUtc = utcNow, + UpdatedUtc = utcNow + }, options => options.ImmediateConsistency()); + using var content = CreateSourceMapContent(SourceMap); + + await SendRequestAsync(request => request + .Post() + .BearerToken(token.Id) + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .Content(content) + .StatusCodeShouldBeNotFound()); + } + + [Fact] + public async Task GetAsync_WithSourceMapsWriteToken_ReturnsForbidden() + { + var token = await CreateSourceMapUploadTokenAsync(SampleDataService.TEST_PROJECT_ID); + + await SendRequestAsync(request => request + .BearerToken(token.Id) + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "source-maps") + .StatusCodeShouldBeForbidden()); + } + + [Fact] + public Task GetAsync_ProjectOutsideUserOrganization_ReturnsNotFound() + { + return SendRequestAsync(request => request + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.FREE_PROJECT_ID, "source-maps") + .StatusCodeShouldBeNotFound()); + } + + private static MultipartFormDataContent CreateSourceMapContent(byte[] sourceMap) + { + var fileContent = new ByteArrayContent(sourceMap); + fileContent.Headers.ContentType = new("application/json"); + + var content = new MultipartFormDataContent(); + content.Add(new StringContent(GeneratedFileUrl), "generated_file_url"); + content.Add(fileContent, "file", "app.min.js.map"); + return content; + } + + private async Task CreateSourceMapUploadTokenAsync(string projectId) + { + var token = await SendRequestAsAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", projectId, "tokens") + .Content(new NewToken + { + OrganizationId = SampleDataService.TEST_ORG_ID, + ProjectId = projectId, + Scopes = [AuthorizationRoles.SourceMapsWrite], + Notes = "Source map deployment" + }) + .StatusCodeShouldBeCreated()); + + Assert.NotNull(token); + await RefreshDataAsync(); + return token; + } +} + +public sealed class SourceMapEndpointValidationTests +{ + [Fact] + public async Task ReadFormAsync_WithoutMultipartContent_ReturnsUnprocessableEntity() + { + var context = new DefaultHttpContext(); + context.Request.ContentType = "text/plain"; + + var (form, result) = await SourceMapEndpoints.ReadFormAsync(context.Request, TestContext.Current.CancellationToken); + + Assert.Null(form); + var statusCodeResult = Assert.IsAssignableFrom(result); + var valueResult = Assert.IsAssignableFrom(result); + var problemDetails = Assert.IsType(valueResult.Value); + Assert.Equal(StatusCodes.Status422UnprocessableEntity, statusCodeResult.StatusCode); + Assert.Equal(["The request must use multipart/form-data."], problemDetails.Errors["file"]); + } + + [Fact] + public async Task ReadFormAsync_WithMalformedMultipartContent_ReturnsUnprocessableEntity() + { + var context = new DefaultHttpContext(); + context.Request.ContentType = "multipart/form-data"; + context.Request.Body = new MemoryStream(Encoding.UTF8.GetBytes("not multipart")); + + var (form, result) = await SourceMapEndpoints.ReadFormAsync(context.Request, TestContext.Current.CancellationToken); + + Assert.Null(form); + var statusCodeResult = Assert.IsAssignableFrom(result); + var valueResult = Assert.IsAssignableFrom(result); + var problemDetails = Assert.IsType(valueResult.Value); + Assert.Equal(StatusCodes.Status422UnprocessableEntity, statusCodeResult.StatusCode); + Assert.Equal(["The multipart form data is invalid."], problemDetails.Errors["file"]); + } +} diff --git a/tests/Exceptionless.Tests/Api/Endpoints/TokenEndpointTests.cs b/tests/Exceptionless.Tests/Api/Endpoints/TokenEndpointTests.cs index 0f061d78f2..9756b5ddf0 100644 --- a/tests/Exceptionless.Tests/Api/Endpoints/TokenEndpointTests.cs +++ b/tests/Exceptionless.Tests/Api/Endpoints/TokenEndpointTests.cs @@ -486,6 +486,46 @@ public async Task PostByProjectAsync_WithValidProject_CreatesProjectScopedToken( Assert.Equal(SampleDataService.TEST_PROJECT_ID, token.ProjectId); } + [Fact] + public async Task PostByProjectAsync_WithSourceMapsWriteScope_CreatesProjectScopedToken() + { + var viewToken = await SendRequestAsAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("projects", SampleDataService.TEST_PROJECT_ID, "tokens") + .Content(new NewToken + { + OrganizationId = SampleDataService.TEST_ORG_ID, + ProjectId = SampleDataService.TEST_PROJECT_ID, + Scopes = [AuthorizationRoles.SourceMapsWrite], + Notes = "Source map deployment" + }) + .StatusCodeShouldBeCreated()); + + Assert.NotNull(viewToken); + Assert.Equal(SampleDataService.TEST_ORG_ID, viewToken.OrganizationId); + Assert.Equal(SampleDataService.TEST_PROJECT_ID, viewToken.ProjectId); + Assert.Equal([AuthorizationRoles.SourceMapsWrite], viewToken.Scopes); + } + + [Fact] + public async Task PostByOrganizationAsync_WithSourceMapsWriteScope_ReturnsValidationError() + { + var problemDetails = await SendRequestAsAsync(request => request + .Post() + .AsTestOrganizationUser() + .AppendPaths("organizations", SampleDataService.TEST_ORG_ID, "tokens") + .Content(new NewToken + { + OrganizationId = SampleDataService.TEST_ORG_ID, + Scopes = [AuthorizationRoles.SourceMapsWrite] + }) + .StatusCodeShouldBeUnprocessableEntity()); + + Assert.NotNull(problemDetails); + Assert.Contains(problemDetails.Errors, error => error.Key.Equals("project_id", StringComparison.OrdinalIgnoreCase)); + } + [Fact] public async Task PostAsync_WithProjectFromUnauthorizedOrganization_ReturnsValidationError() { diff --git a/tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs b/tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs index c7300641a2..c2aff78bd7 100644 --- a/tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs +++ b/tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs @@ -143,19 +143,31 @@ public async Task CanCleanupSoftDeletedOrganization() await _organizationRepository.AddAsync(organization, o => o.ImmediateConsistency()); var project = await _projectRepository.AddAsync(_projectData.GenerateSampleProject(), o => o.ImmediateConsistency()); + var deletedProject = _projectData.GenerateProject(generateId: true, organizationId: organization.Id, name: "Deleted project"); + deletedProject.IsDeleted = true; + await _projectRepository.AddAsync(deletedProject, o => o.ImmediateConsistency()); var stack = await _stackRepository.AddAsync(_stackData.GenerateSampleStack(), o => o.ImmediateConsistency()); var persistentEvent = await _eventRepository.AddAsync(_eventData.GenerateEvent(organization.Id, project.Id, stack.Id), o => o.ImmediateConsistency()); string iconPath = OrganizationStoragePaths.GetProfileImagePath(organization.Id, "icon.png"); using var stream = new MemoryStream([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); await _fileStorage.SaveFileAsync(iconPath, stream, TestCancellationToken); + string sourceMapPath = $"source-maps/{project.Id}/app.map"; + await using (var sourceMap = new MemoryStream([0x7B, 0x7D])) + await _fileStorage.SaveFileAsync(sourceMapPath, sourceMap, TestCancellationToken); + string deletedProjectSourceMapPath = $"source-maps/{deletedProject.Id}/app.map"; + await using (var sourceMap = new MemoryStream([0x7B, 0x7D])) + await _fileStorage.SaveFileAsync(deletedProjectSourceMapPath, sourceMap, TestCancellationToken); await _job.RunAsync(TestCancellationToken); Assert.Null(await _organizationRepository.GetByIdAsync(organization.Id, o => o.IncludeSoftDeletes())); Assert.Null(await _projectRepository.GetByIdAsync(project.Id, o => o.IncludeSoftDeletes())); + Assert.Null(await _projectRepository.GetByIdAsync(deletedProject.Id, o => o.IncludeSoftDeletes())); Assert.Null(await _stackRepository.GetByIdAsync(stack.Id, o => o.IncludeSoftDeletes())); Assert.Null(await _eventRepository.GetByIdAsync(persistentEvent.Id, o => o.IncludeSoftDeletes())); Assert.False(await _fileStorage.ExistsAsync(iconPath)); + Assert.False(await _fileStorage.ExistsAsync(sourceMapPath)); + Assert.False(await _fileStorage.ExistsAsync(deletedProjectSourceMapPath)); } [Fact] @@ -248,6 +260,9 @@ public async Task CanCleanupSoftDeletedProject() var stack = await _stackRepository.AddAsync(_stackData.GenerateSampleStack(), o => o.ImmediateConsistency()); var persistentEvent = await _eventRepository.AddAsync(_eventData.GenerateEvent(organization.Id, project.Id, stack.Id), o => o.ImmediateConsistency()); + string sourceMapPath = $"source-maps/{project.Id}/app.map"; + await using (var sourceMap = new MemoryStream([0x7B, 0x7D])) + await _fileStorage.SaveFileAsync(sourceMapPath, sourceMap, TestCancellationToken); await _job.RunAsync(TestCancellationToken); @@ -255,6 +270,7 @@ public async Task CanCleanupSoftDeletedProject() Assert.Null(await _projectRepository.GetByIdAsync(project.Id, o => o.IncludeSoftDeletes())); Assert.Null(await _stackRepository.GetByIdAsync(stack.Id, o => o.IncludeSoftDeletes())); Assert.Null(await _eventRepository.GetByIdAsync(persistentEvent.Id, o => o.IncludeSoftDeletes())); + Assert.False(await _fileStorage.ExistsAsync(sourceMapPath)); } [Fact] diff --git a/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapDocumentTests.cs b/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapDocumentTests.cs new file mode 100644 index 0000000000..a4831b22a4 --- /dev/null +++ b/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapDocumentTests.cs @@ -0,0 +1,118 @@ +using System.Text; +using System.Text.Json; +using Exceptionless.Core.Services.SourceMaps; +using Xunit; + +namespace Exceptionless.Tests.Services.SourceMaps; + +public sealed class SourceMapDocumentTests +{ + [Fact] + public void FindOriginalLocation_MappedSegment_ReturnsOriginalSourceAndName() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["src/app.ts"],"names":["meaningfulFunction"],"mappings":"AAAAA"}"""); + var document = SourceMapDocument.Parse(sourceMap); + + var location = document.FindOriginalLocation(0, 12); + + Assert.NotNull(location); + Assert.Equal("src/app.ts", location.Source); + Assert.Equal(0, location.Line); + Assert.Equal(0, location.Column); + Assert.Equal("meaningfulFunction", location.Name); + } + + [Fact] + public void FindOriginalLocation_WithSourceRoot_CombinesRelativeSource() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sourceRoot":"https://cdn.example.com/source/","sources":["app.ts"],"names":[],"mappings":"AAAA"}"""); + var document = SourceMapDocument.Parse(sourceMap); + + var location = document.FindOriginalLocation(0, 0); + + Assert.NotNull(location); + Assert.Equal("https://cdn.example.com/source/app.ts", location.Source); + } + + [Fact] + public void FindOriginalLocation_AfterUnmappedSegment_ReturnsNull() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["src/app.ts"],"names":[],"mappings":"AAAA,U"}"""); + var document = SourceMapDocument.Parse(sourceMap); + + var location = document.FindOriginalLocation(0, 12); + + Assert.Null(location); + } + + [Fact] + public void Parse_IndexedSourceMap_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sections":[],"sources":[],"names":[],"mappings":""}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap)); + + Assert.Contains("Indexed source maps", exception.Message); + } + + [Fact] + public void Parse_MissingMappings_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":[],"names":[]}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap)); + + Assert.Contains("mappings", exception.Message); + } + + [Fact] + public void Parse_OversizedVersion_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":999999999999,"sources":[],"names":[],"mappings":""}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap)); + + Assert.Contains("version 3", exception.Message); + } + + [Fact] + public void Parse_OversizedVlqValue_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":[],"names":[],"mappings":"gggggggA"}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap)); + + Assert.Contains("too large", exception.Message); + } + + [Fact] + public void Parse_TooManyOneLineMappingSegments_ThrowsJsonException() + { + string mappings = String.Join(',', Enumerable.Repeat("A", 10_000)); + byte[] sourceMap = Encoding.UTF8.GetBytes($$"""{"version":3,"sources":[],"names":[],"mappings":"{{mappings}}"}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap, maximumSegments: 1)); + + Assert.Contains("too many mapping segments", exception.Message); + } + + [Fact] + public void Parse_TooManyGeneratedLines_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":[],"names":[],"mappings":";"}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap, maximumSegments: 1_000_000, maximumLines: 1)); + + Assert.Contains("too many generated lines", exception.Message); + } + + [Fact] + public void Parse_TooManySourceEntries_ThrowsJsonException() + { + byte[] sourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["one.ts","two.ts"],"names":[],"mappings":""}"""); + + var exception = Assert.Throws(() => SourceMapDocument.Parse(sourceMap, maximumSegments: 1, maximumLines: 1, maximumSourceEntries: 1)); + + Assert.Contains("sources array contains too many entries", exception.Message); + } +} diff --git a/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapServiceTests.cs b/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapServiceTests.cs new file mode 100644 index 0000000000..eedd95f944 --- /dev/null +++ b/tests/Exceptionless.Tests/Services/SourceMaps/SourceMapServiceTests.cs @@ -0,0 +1,604 @@ +using System.Net; +using System.Text; +using System.Text.Json; +using Exceptionless.Core; +using Exceptionless.Core.Billing; +using Exceptionless.Core.Configuration; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.Data; +using Exceptionless.Core.Plugins.EventProcessor; +using Exceptionless.Core.Plugins.EventProcessor.Default; +using Exceptionless.Core.Services.SourceMaps; +using Foundatio.Caching; +using Foundatio.Lock; +using Foundatio.Serializer; +using Foundatio.Storage; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Logging; +using Xunit; + +namespace Exceptionless.Tests.Services.SourceMaps; + +public sealed class SourceMapServiceTests : TestWithServices +{ + private const string ProjectId = "507f1f77bcf86cd799439011"; + private const string GeneratedFileUrl = "https://cdn.example.com/assets/app.min.js"; + private static readonly byte[] SourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["src/app.ts"],"names":["meaningfulFunction"],"mappings":"AAAAA"}"""); + private static readonly byte[] UpdatedSourceMap = Encoding.UTF8.GetBytes("""{"version":3,"sources":["src/updated.ts"],"names":["updatedFunction"],"mappings":"AAAAA"}"""); + + public SourceMapServiceTests(ITestOutputHelper output) : base(output) + { + } + + [Fact] + public async Task SymbolicateAsync_WithUploadedSourceMap_RewritesFrameAndPreservesGeneratedLocation() + { + var service = GetService(); + await using var sourceMapStream = new MemoryStream(SourceMap); + var artifact = await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + var error = CreateError(); + + bool changed = await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken); + + Assert.True(changed); + var frame = Assert.Single(error.StackTrace!); + Assert.Equal("meaningfulFunction", frame.Name); + Assert.Equal("src/app.ts", frame.FileName); + Assert.Equal(1, frame.LineNumber); + Assert.Equal(1, frame.Column); + var generatedLocation = Assert.IsType(frame.Data!["@source_map"]); + Assert.Equal(GeneratedFileUrl, generatedLocation["generated_file_name"]); + Assert.Equal(artifact.Id, generatedLocation["source_map_id"]); + } + + [Fact] + public async Task SymbolicateAsync_WithoutGeneratedColumn_LeavesFrameUnchanged() + { + var service = GetService(); + await using var sourceMapStream = new MemoryStream(SourceMap); + await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + var error = CreateError(); + var frame = Assert.Single(error.StackTrace!); + frame.Column = null; + var serializer = GetService(); + string originalFrame = serializer.SerializeToString(frame); + + bool changed = await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken); + + Assert.False(changed); + Assert.Equal(originalFrame, serializer.SerializeToString(frame)); + } + + [Fact] + public async Task GetArtifactsAsync_AfterUploadAndDelete_ReflectsStoredArtifact() + { + var service = GetService(); + await using var sourceMapStream = new MemoryStream(SourceMap); + var uploaded = await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + + var artifacts = await service.GetArtifactsAsync(ProjectId, TestContext.Current.CancellationToken); + bool deleted = await service.DeleteArtifactAsync(ProjectId, uploaded.Id, TestContext.Current.CancellationToken); + + var artifact = Assert.Single(artifacts); + Assert.Equal(GeneratedFileUrl, artifact.GeneratedFileUrl); + Assert.False(artifact.IsAutoDownloaded); + Assert.True(deleted); + Assert.Empty(await service.GetArtifactsAsync(ProjectId, TestContext.Current.CancellationToken)); + } + + [Fact] + public async Task SymbolicateAsync_WithPublicSourceMapHeader_DownloadsAndStoresSourceMap() + { + var options = GetService(); + var requestedUris = new List(); + var handler = new DelegateHandler(request => + { + requestedUris.Add(request.RequestUri!); + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + Assert.Equal("bytes=-65536", request.Headers.Range?.ToString()); + Assert.Equal("identity", Assert.Single(request.Headers.AcceptEncoding).Value); + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Content.Headers.ContentLength = options.SourceMapOptions.MaximumGeneratedFileSize + 1; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + var service = new SourceMapService( + new TestHttpClientFactory(httpClient), + GetService(), + GetService(), + GetService(), + GetService(), + GetService(), + options, + GetService(), + GetService>()); + var error = CreateError(); + + bool changed = await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken); + + Assert.True(changed); + Assert.Equal([new Uri(GeneratedFileUrl), new Uri("https://cdn.example.com/assets/app.min.js.map")], requestedUris); + var artifact = Assert.Single(await service.GetArtifactsAsync(ProjectId, TestContext.Current.CancellationToken)); + Assert.True(artifact.IsAutoDownloaded); + Assert.Equal("https://cdn.example.com/assets/app.min.js.map", artifact.SourceMapUrl); + } + + [Fact] + public async Task SymbolicateAsync_WithOversizedGeneratedFile_UsesConventionalSourceMapUrl() + { + var options = GetService(); + var requestedUris = new List(); + var handler = new DelegateHandler(request => + { + requestedUris.Add(request.RequestUri!); + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + Assert.Equal("bytes=-65536", request.Headers.Range?.ToString()); + Assert.Equal("identity", Assert.Single(request.Headers.AcceptEncoding).Value); + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Content.Headers.ContentLength = options.SourceMapOptions.MaximumGeneratedFileSize + 1; + return response; + } + + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + var error = CreateError(); + + bool changed = await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken); + + Assert.True(changed); + Assert.Equal([new Uri(GeneratedFileUrl), new Uri("https://cdn.example.com/assets/app.min.js.map")], requestedUris); + var artifact = Assert.Single(await service.GetArtifactsAsync(ProjectId, TestContext.Current.CancellationToken)); + Assert.True(artifact.IsAutoDownloaded); + Assert.Equal("https://cdn.example.com/assets/app.min.js.map", artifact.SourceMapUrl); + } + + [Fact] + public async Task SymbolicateAsync_WithExpiredAutoDownloadedSourceMap_RefreshesStoredMap() + { + int sourceMapDownloadCount = 0; + var handler = new DelegateHandler(request => + { + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + sourceMapDownloadCount++; + byte[] content = sourceMapDownloadCount == 1 ? SourceMap : UpdatedSourceMap; + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(content) }; + }); + using var httpClient = new HttpClient(handler); + using (var initialService = CreateService(httpClient)) + { + var initialError = CreateError(); + Assert.True(await initialService.SymbolicateAsync(ProjectId, initialError, TestContext.Current.CancellationToken)); + Assert.Equal("meaningfulFunction", Assert.Single(initialError.StackTrace!).Name); + } + + TimeProvider.Advance(TimeSpan.FromMinutes(61)); + + using var refreshedService = CreateService(httpClient); + var refreshedError = CreateError(); + Assert.True(await refreshedService.SymbolicateAsync(ProjectId, refreshedError, TestContext.Current.CancellationToken)); + var refreshedFrame = Assert.Single(refreshedError.StackTrace!); + Assert.Equal("updatedFunction", refreshedFrame.Name); + Assert.Equal("src/updated.ts", refreshedFrame.FileName); + Assert.Equal(2, sourceMapDownloadCount); + } + + [Fact] + public async Task SymbolicateAsync_WhenExpiredSourceMapCannotRefresh_DoesNotUseStaleMap() + { + int sourceMapDownloadCount = 0; + var handler = new DelegateHandler(request => + { + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + sourceMapDownloadCount++; + return sourceMapDownloadCount == 1 + ? new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) } + : new HttpResponseMessage(HttpStatusCode.ServiceUnavailable); + }); + using var httpClient = new HttpClient(handler); + using (var initialService = CreateService(httpClient)) + Assert.True(await initialService.SymbolicateAsync(ProjectId, CreateError(), TestContext.Current.CancellationToken)); + + TimeProvider.Advance(TimeSpan.FromMinutes(61)); + + using var refreshedService = CreateService(httpClient); + var error = CreateError(); + Assert.False(await refreshedService.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken)); + var frame = Assert.Single(error.StackTrace!); + Assert.Equal("a", frame.Name); + Assert.Equal(GeneratedFileUrl, frame.FileName); + Assert.Equal(2, sourceMapDownloadCount); + } + + [Fact] + public async Task SaveUploadedAsync_WhenReplacingMap_UsesNewMapAndRemovesOldContent() + { + var service = GetService(); + await using (var sourceMapStream = new MemoryStream(SourceMap)) + await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + await using (var sourceMapStream = new MemoryStream(UpdatedSourceMap)) + await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + + var error = CreateError(); + Assert.True(await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken)); + Assert.Equal("updatedFunction", Assert.Single(error.StackTrace!).Name); + + var mapFiles = await GetService().GetFileListAsync($"source-maps/{ProjectId}/*.map", cancellationToken: TestContext.Current.CancellationToken); + Assert.Single(mapFiles); + } + + [Fact] + public async Task SymbolicateAsync_WhenMapExceedsParsedCacheLimit_DoesNotRetainMapInMemory() + { + var options = GetService(); + options.SourceMapOptions.MaximumParsedSourceMapCacheSize = 1; + int requestCount = 0; + var handler = new DelegateHandler(request => + { + requestCount++; + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + Assert.True(await service.SymbolicateAsync(ProjectId, CreateError(), TestContext.Current.CancellationToken)); + + await GetService().DeleteFilesAsync($"source-maps/{ProjectId}/*", TestContext.Current.CancellationToken); + + Assert.True(await service.SymbolicateAsync(ProjectId, CreateError(), TestContext.Current.CancellationToken)); + Assert.Equal(4, requestCount); + } + + [Fact] + public async Task SymbolicateAsync_WhenFirstCallerCancels_KeepsSharedDownloadForSecondCaller() + { + int requestCount = 0; + var mapRequestStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var releaseMapRequest = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var handler = new AsyncDelegateHandler(async request => + { + requestCount++; + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + mapRequestStarted.TrySetResult(); + await releaseMapRequest.Task; + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + using var cancellationTokenSource = new CancellationTokenSource(); + Task canceledCaller = service.SymbolicateAsync(ProjectId, CreateError(), cancellationTokenSource.Token); + await mapRequestStarted.Task; + + cancellationTokenSource.Cancel(); + await Assert.ThrowsAnyAsync(() => canceledCaller); + Task secondCaller = service.SymbolicateAsync(ProjectId, CreateError(), TestContext.Current.CancellationToken); + releaseMapRequest.TrySetResult(); + + Assert.True(await secondCaller); + Assert.Equal(2, requestCount); + } + + [Fact] + public async Task SaveUploadedAsync_DuringAutomaticDownload_UploadRemainsAuthoritative() + { + var mapRequestStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var releaseMapRequest = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var handler = new AsyncDelegateHandler(async request => + { + if (request.RequestUri == new Uri(GeneratedFileUrl)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", "app.min.js.map"); + return response; + } + + mapRequestStarted.TrySetResult(); + await releaseMapRequest.Task; + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + Task automaticDownload = service.SymbolicateAsync(ProjectId, CreateError(), TestContext.Current.CancellationToken); + await mapRequestStarted.Task; + + await using var uploadedStream = new MemoryStream(UpdatedSourceMap); + Task upload = service.SaveUploadedAsync( + ProjectId, + GeneratedFileUrl, + "app.min.js.map", + uploadedStream, + TestContext.Current.CancellationToken); + releaseMapRequest.TrySetResult(); + + Assert.True(await automaticDownload); + var uploaded = await upload; + Assert.False(uploaded.IsAutoDownloaded); + var error = CreateError(); + Assert.True(await service.SymbolicateAsync(ProjectId, error, TestContext.Current.CancellationToken)); + Assert.Equal("updatedFunction", Assert.Single(error.StackTrace!).Name); + Assert.False(Assert.Single(await service.GetArtifactsAsync(ProjectId, TestContext.Current.CancellationToken)).IsAutoDownloaded); + } + + [Fact] + public async Task SymbolicateAsync_FreeClientKeyExceedsDiscoveryLimit_StopsDownloadingNewUrls() + { + var options = GetService(); + options.SourceMapOptions.MaximumAutoDiscoveriesPerFreeClientKey = 1; + options.SourceMapOptions.MaximumAutoDiscoveriesPerFreeProject = 10; + options.SourceMapOptions.MaximumAutoDiscoveriesPerFreeOrganization = 10; + int requestCount = 0; + var handler = new DelegateHandler(request => + { + requestCount++; + if (!request.RequestUri!.AbsolutePath.EndsWith(".map", StringComparison.Ordinal)) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("minified") }; + response.Headers.TryAddWithoutValidation("SourceMap", request.RequestUri.AbsolutePath + ".map"); + return response; + } + + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(SourceMap) }; + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + string suffix = Guid.NewGuid().ToString("N"); + var request = new SourceMapRequest($"organization-{suffix}", $"project-{suffix}", $"key-{suffix}", true); + + Assert.True(await service.SymbolicateAsync(request, CreateError($"https://cdn.example.com/{suffix}/first.js"), TestContext.Current.CancellationToken)); + Assert.False(await service.SymbolicateAsync(request, CreateError($"https://cdn.example.com/{suffix}/second.js"), TestContext.Current.CancellationToken)); + Assert.Equal(2, requestCount); + } + + [Fact] + public async Task TryReserveDiscoveryAsync_MultipleClientKeysExceedProjectLimit_BlocksProject() + { + var options = GetService(); + options.SourceMapOptions.MaximumAutoDiscoveriesPerClientKey = 10; + options.SourceMapOptions.MaximumAutoDiscoveriesPerProject = 2; + options.SourceMapOptions.MaximumAutoDiscoveriesPerOrganization = 10; + var throttle = GetService(); + string suffix = Guid.NewGuid().ToString("N"); + + Assert.True(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest($"organization-{suffix}", $"project-{suffix}", $"key-1-{suffix}", false))); + Assert.True(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest($"organization-{suffix}", $"project-{suffix}", $"key-2-{suffix}", false))); + Assert.False(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest($"organization-{suffix}", $"project-{suffix}", $"key-3-{suffix}", false))); + } + + [Fact] + public async Task TryReserveDiscoveryAsync_SameClientKeyAcrossProjects_UsesSeparateKeyBudgets() + { + var options = GetService(); + options.SourceMapOptions.MaximumAutoDiscoveriesPerClientKey = 1; + options.SourceMapOptions.MaximumAutoDiscoveriesPerProject = 10; + options.SourceMapOptions.MaximumAutoDiscoveriesPerOrganization = 10; + var throttle = GetService(); + string suffix = Guid.NewGuid().ToString("N"); + string organizationId = $"organization-{suffix}"; + string clientKeyHash = $"key-{suffix}"; + + Assert.True(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest(organizationId, $"project-1-{suffix}", clientKeyHash, false))); + Assert.False(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest(organizationId, $"project-1-{suffix}", clientKeyHash, false))); + Assert.True(await throttle.TryReserveDiscoveryAsync(new SourceMapRequest(organizationId, $"project-2-{suffix}", clientKeyHash, false))); + } + + [Fact] + public async Task TryReserveOutboundRequestAsync_DestinationExceedsLimit_BlocksDestinationOnly() + { + var options = GetService(); + options.SourceMapOptions.MaximumAutoDownloadRequestsPerDestination = 1; + options.SourceMapOptions.MaximumAutoDownloadRequestsGlobally = 10; + var throttle = GetService(); + string suffix = Guid.NewGuid().ToString("N"); + + Assert.True(await throttle.TryReserveOutboundRequestAsync(new Uri($"https://{suffix}.example.com/first.js"))); + Assert.False(await throttle.TryReserveOutboundRequestAsync(new Uri($"https://{suffix}.example.com/second.js"))); + Assert.True(await throttle.TryReserveOutboundRequestAsync(new Uri($"https://other-{suffix}.example.com/app.js"))); + } + + [Fact] + public async Task TryReserveOutboundRequestAsync_RefreshUsesSeparateBudget() + { + var options = GetService(); + options.SourceMapOptions.MaximumAutoDownloadRequestsPerDestination = 1; + options.SourceMapOptions.MaximumAutoDownloadRequestsGlobally = 10; + options.SourceMapOptions.MaximumAutoRefreshRequestsPerDestination = 1; + options.SourceMapOptions.MaximumAutoRefreshRequestsGlobally = 10; + var throttle = GetService(); + string suffix = Guid.NewGuid().ToString("N"); + var uri = new Uri($"https://{suffix}.example.com/app.js"); + + Assert.True(await throttle.TryReserveOutboundRequestAsync(uri)); + Assert.False(await throttle.TryReserveOutboundRequestAsync(uri)); + Assert.True(await throttle.TryReserveOutboundRequestAsync(uri, isRefresh: true)); + Assert.False(await throttle.TryReserveOutboundRequestAsync(uri, isRefresh: true)); + } + + [Fact] + public async Task SymbolicateAsync_RepeatedFailedUrl_UsesFailureCacheWithoutAnotherRequest() + { + int requestCount = 0; + var handler = new DelegateHandler(_ => + { + requestCount++; + return new HttpResponseMessage(HttpStatusCode.NotFound); + }); + using var httpClient = new HttpClient(handler); + using var service = CreateService(httpClient); + string suffix = Guid.NewGuid().ToString("N"); + string generatedFileUrl = $"https://cdn.example.com/{suffix}/missing.js"; + + Assert.False(await service.SymbolicateAsync($"project-{suffix}", CreateError(generatedFileUrl), TestContext.Current.CancellationToken)); + Assert.False(await service.SymbolicateAsync($"project-{suffix}", CreateError(generatedFileUrl), TestContext.Current.CancellationToken)); + Assert.Equal(1, requestCount); + } + + [Fact] + public void ReadFromConfiguration_WithUnsafeLimits_NormalizesLimits() + { + var values = new Dictionary + { + ["SourceMaps:RequestTimeoutMilliseconds"] = "0", + ["SourceMaps:MaximumGeneratedFileSize"] = "0", + ["SourceMaps:MaximumSourceMapSize"] = "-1", + ["SourceMaps:MaximumMappingSegments"] = "0", + ["SourceMaps:MaximumRedirects"] = "-1", + ["SourceMaps:MaximumConcurrentDownloads"] = "0", + ["SourceMaps:MaximumConcurrentDownloadsGlobally"] = "0", + ["SourceMaps:AutoDownloadRateLimitPeriodMinutes"] = "0", + ["SourceMaps:MaximumAutoDiscoveriesPerFreeClientKey"] = "-1", + ["SourceMaps:MaximumAutoDiscoveriesPerClientKey"] = "-1", + ["SourceMaps:MaximumAutoDiscoveriesPerFreeProject"] = "-1", + ["SourceMaps:MaximumAutoDiscoveriesPerProject"] = "-1", + ["SourceMaps:MaximumAutoDiscoveriesPerFreeOrganization"] = "-1", + ["SourceMaps:MaximumAutoDiscoveriesPerOrganization"] = "-1", + ["SourceMaps:MaximumAutoDownloadRequestsPerDestination"] = "-1", + ["SourceMaps:MaximumAutoDownloadConnectionsPerIpAddress"] = "-1", + ["SourceMaps:MaximumAutoDownloadRequestsGlobally"] = "-1", + ["SourceMaps:MaximumAutoRefreshRequestsPerDestination"] = "-1", + ["SourceMaps:MaximumAutoRefreshRequestsGlobally"] = "-1", + ["SourceMaps:MaximumFramesPerError"] = "0", + ["SourceMaps:MaximumProcessingTimeMilliseconds"] = "0", + ["SourceMaps:AutoDownloadRefreshIntervalMinutes"] = "0", + ["SourceMaps:ParsedSourceMapCacheLifetimeMinutes"] = "0", + ["SourceMaps:MaximumParsedSourceMapCacheSize"] = "0" + }; + var configuration = new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + + var options = SourceMapOptions.ReadFromConfiguration(configuration); + + Assert.Equal(1, options.RequestTimeoutMilliseconds); + Assert.Equal(1, options.MaximumGeneratedFileSize); + Assert.Equal(1, options.MaximumSourceMapSize); + Assert.Equal(1, options.MaximumMappingSegments); + Assert.Equal(0, options.MaximumRedirects); + Assert.Equal(1, options.MaximumConcurrentDownloads); + Assert.Equal(1, options.MaximumConcurrentDownloadsGlobally); + Assert.Equal(1, options.AutoDownloadRateLimitPeriodMinutes); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerFreeClientKey); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerClientKey); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerFreeProject); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerProject); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerFreeOrganization); + Assert.Equal(0, options.MaximumAutoDiscoveriesPerOrganization); + Assert.Equal(0, options.MaximumAutoDownloadRequestsPerDestination); + Assert.Equal(0, options.MaximumAutoDownloadConnectionsPerIpAddress); + Assert.Equal(0, options.MaximumAutoDownloadRequestsGlobally); + Assert.Equal(0, options.MaximumAutoRefreshRequestsPerDestination); + Assert.Equal(0, options.MaximumAutoRefreshRequestsGlobally); + Assert.Equal(1, options.MaximumFramesPerError); + Assert.Equal(1, options.MaximumProcessingTimeMilliseconds); + Assert.Equal(1, options.AutoDownloadRefreshIntervalMinutes); + Assert.Equal(1, options.ParsedSourceMapCacheLifetimeMinutes); + Assert.Equal(1, options.MaximumParsedSourceMapCacheSize); + } + + [Fact] + public async Task EventProcessingAsync_WithUploadedSourceMap_UsesOriginalNameForStackSignature() + { + var service = GetService(); + await using var sourceMapStream = new MemoryStream(SourceMap); + await service.SaveUploadedAsync(ProjectId, GeneratedFileUrl, "app.min.js.map", sourceMapStream, TestContext.Current.CancellationToken); + var serializer = GetService(); + var options = GetService(); + var loggerFactory = GetService(); + var sourceMapPlugin = new SourceMapPlugin(service, serializer, GetService(), options, loggerFactory); + var errorPlugin = new ErrorPlugin(serializer, options, loggerFactory); + var persistentEvent = new PersistentEvent { Type = Event.KnownTypes.Error }; + persistentEvent.SetError(CreateError()); + var context = new EventContext( + persistentEvent, + new Organization { Id = "507f1f77bcf86cd799439012" }, + new Project { Id = ProjectId, OrganizationId = "507f1f77bcf86cd799439012" }); + + await sourceMapPlugin.EventProcessingAsync(context); + await errorPlugin.EventProcessingAsync(context); + + Assert.Equal("meaningfulFunction()", context.StackSignatureData["Method"]); + Assert.True(sourceMapPlugin.HandleError(new IOException("Unavailable"), context)); + } + + private static Error CreateError(string generatedFileUrl = GeneratedFileUrl) + { + return new Error + { + Message = "Test error", + Type = "TypeError", + StackTrace = + [ + new StackFrame + { + Name = "a", + FileName = generatedFileUrl, + LineNumber = 1, + Column = 1 + } + ] + }; + } + + private SourceMapService CreateService(HttpClient httpClient) + { + return new SourceMapService( + new TestHttpClientFactory(httpClient), + GetService(), + GetService(), + GetService(), + GetService(), + GetService(), + GetService(), + GetService(), + GetService>()); + } + + private sealed class TestHttpClientFactory(HttpClient httpClient) : IHttpClientFactory + { + public HttpClient CreateClient(string name) => httpClient; + } + + private sealed class DelegateHandler(Func handler) : HttpMessageHandler + { + protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + return Task.FromResult(handler(request)); + } + } + + private sealed class AsyncDelegateHandler(Func> handler) : HttpMessageHandler + { + protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + return handler(request); + } + } +} diff --git a/tests/http/source-maps.http b/tests/http/source-maps.http new file mode 100644 index 0000000000..80d14bafd8 --- /dev/null +++ b/tests/http/source-maps.http @@ -0,0 +1,64 @@ +@apiUrl = http://localhost:7110/api/v2 +@email = admin@exceptionless.test +@password = tester +@organizationId = 537650f3b77efe23a47914f3 +@projectId = 537650f3b77efe23a47914f4 +@generatedFileUrl = https://cdn.example.com/assets/app.min.js +@sourceMapFile = ./app.min.js.map + +### Login to test account +# @name login +POST {{apiUrl}}/auth/login +Content-Type: application/json + +{ + "email": "{{email}}", + "password": "{{password}}" +} + +### + +@userToken = {{login.response.body.$.token}} + +### Create a project-scoped source-map upload token +# @name sourceMapToken +POST {{apiUrl}}/projects/{{projectId}}/tokens +Authorization: Bearer {{userToken}} +Content-Type: application/json + +{ + "organization_id": "{{organizationId}}", + "project_id": "{{projectId}}", + "scopes": ["source-maps:write"], + "notes": "HTTP sample source map uploads" +} + +### + +@uploadToken = {{sourceMapToken.response.body.$.id}} + +### Upload source map +# Replace {{sourceMapFile}} with a local version 3 source map. +# @name uploadedSourceMap +POST {{apiUrl}}/projects/{{projectId}}/source-maps +Authorization: Bearer {{uploadToken}} +Content-Type: multipart/form-data; boundary=source-map-boundary + +--source-map-boundary +Content-Disposition: form-data; name="generated_file_url" + +{{generatedFileUrl}} +--source-map-boundary +Content-Disposition: form-data; name="file"; filename="app.min.js.map" +Content-Type: application/json + +< {{sourceMapFile}} +--source-map-boundary-- + +### List source maps +GET {{apiUrl}}/projects/{{projectId}}/source-maps +Authorization: Bearer {{userToken}} + +### Delete source map +DELETE {{apiUrl}}/projects/{{projectId}}/source-maps/{{uploadedSourceMap.response.body.$.id}} +Authorization: Bearer {{userToken}}