Skip to content

Replace JSON.NET with System.Text.Json across the codebase#2135

Open
niemyjski wants to merge 141 commits into
mainfrom
feature/system-text-json-v2
Open

Replace JSON.NET with System.Text.Json across the codebase#2135
niemyjski wants to merge 141 commits into
mainfrom
feature/system-text-json-v2

Conversation

@niemyjski
Copy link
Copy Markdown
Member

@niemyjski niemyjski commented Feb 28, 2026

Replace JSON.NET with System.Text.Json

Replaces all Newtonsoft.Json serialization with System.Text.Json (STJ) and migrates from NEST to Elastic.Clients.Elasticsearch.

Key changes

Serialization

  • All JSON serialization now uses STJ with SnakeCaseLower naming policy
  • ObjectToInferredTypesConverter handles object-typed properties (replaces DataObjectConverter)
  • [JsonExtensionData] on Event merges root-level @error, @request, etc. into Data dict
  • [JsonPropertyName] overrides for legacy property names (o_s_name, o_s_version)
  • V1 webhook models use [JsonPropertyName] for PascalCase backward compatibility

Elasticsearch

  • Migrated from NEST to Elastic.Clients.Elasticsearch throughout
  • All index configurations use new fluent API
  • Migrations and jobs updated for new ES client
  • Repository queries migrated to Foundatio expression-based API

Event processing

  • All event upgraders migrated from JObject to JsonObject (System.Text.Json.Nodes)
  • All plugins migrated from Newtonsoft serializer to STJ
  • GetValue<T>() simplified — removed unnecessary TryDeserializeWithFallback (PascalCase data never existed in ES)
  • Error write-back fix in ErrorPlugin / SimpleErrorPlugin after pipeline mutation

Removed

  • DataObjectConverter, ElasticJsonNetSerializer, all Newtonsoft classes
  • Foundatio.JsonNet, NEST.JsonNetSerializer, FluentRest.NewtonsoftJson packages
  • Unnecessary PascalCase fallback machinery

Test results

  • 1549 total, 0 failed, 1547 passed, 2 skipped
  • New: 27 serializer tests, model serialization tests, query migration tests, integration tests

Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread tests/Exceptionless.Tests/Serializer/SerializerTests.cs Fixed
Comment thread tests/Exceptionless.Tests/Serializer/SerializerTests.cs Fixed
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Fixed
Comment thread src/Exceptionless.Web/Controllers/EventController.cs Fixed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Migrates application-layer JSON usage from Newtonsoft.Json to System.Text.Json (STJ), aligning core models/plugins/tests and adding STJ-based infrastructure for Elasticsearch/NEST while keeping Newtonsoft only as a transitive dependency for the NEST wire protocol.

Changes:

  • Introduces STJ-based IElasticsearchSerializer implementation and STJ type metadata modifiers to match prior Newtonsoft serialization behavior (e.g., omit empty collections).
  • Updates core pipeline/plugins/controllers/jobs to use ITextSerializer/JsonSerializerOptions instead of Newtonsoft abstractions.
  • Refactors tests and fixtures to validate STJ semantics (including semantic JSON comparison).

Reviewed changes

Copilot reviewed 87 out of 87 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/Exceptionless.Tests/Utility/DataBuilder.cs Updates test builder to use ITextSerializer for manual stacking info.
tests/Exceptionless.Tests/Serializer/SerializerTests.cs Reworks serializer tests away from Newtonsoft-specific behavior toward STJ round-trips.
tests/Exceptionless.Tests/Serializer/Models/PersistentEventSerializerTests.cs Uses ITextSerializer for typed data retrieval in PersistentEvent tests.
tests/Exceptionless.Tests/Serializer/Models/DataDictionaryTests.cs Updates GetValue<T> tests to use ITextSerializer pathway.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs Compares serialized JSON via ITextSerializer instead of ToJson().
tests/Exceptionless.Tests/Repositories/ProjectRepositoryTests.cs Updates Slack token access to use serializer-aware accessor.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs Uses serializer-based typed accessors in repository tests.
tests/Exceptionless.Tests/Plugins/WebHookDataTests.cs Switches expected/actual comparisons to semantic JSON equivalence using STJ.
tests/Exceptionless.Tests/Plugins/SummaryDataTests.cs Moves summary comparisons to semantic JSON using STJ.
tests/Exceptionless.Tests/Plugins/GeoTests.cs Updates Geo plugin/tests to pass ITextSerializer.
tests/Exceptionless.Tests/Plugins/EventUpgraderTests.cs Uses JsonNode formatting + semantic compare for upgrader outputs.
tests/Exceptionless.Tests/Plugins/EventParserTests.cs Uses ITextSerializer for round-trip verification; removes Newtonsoft formatting asserts.
tests/Exceptionless.Tests/Pipeline/EventPipelineTests.cs Updates pipeline tests to use serializer-based typed accessors.
tests/Exceptionless.Tests/Mail/MailerTests.cs Updates Mailer construction to accept ITextSerializer.
tests/Exceptionless.Tests/IntegrationTestsBase.cs Replaces FluentRest Newtonsoft serializer with STJ JsonContentSerializer.
tests/Exceptionless.Tests/Exceptionless.Tests.csproj Removes FluentRest.NewtonsoftJson, adds STJ-based FluentRest.
tests/Exceptionless.Tests/Controllers/EventControllerTests.cs Uses ITextSerializer for typed accessors in controller tests.
src/Exceptionless.Web/Exceptionless.Web.csproj Removes NEST.JsonNetSerializer package reference.
src/Exceptionless.Web/Controllers/ProjectController.cs Injects ITextSerializer for Slack token access.
src/Exceptionless.Web/Controllers/EventController.cs Injects ITextSerializer and uses it for event payload byte generation.
src/Exceptionless.Core/Utility/TypeHelper.cs Updates equality handling for STJ JsonElement.
src/Exceptionless.Core/Utility/ExtensibleObject.cs Adds JsonElement handling to generic property retrieval.
src/Exceptionless.Core/Utility/ErrorSignature.cs Switches error signature extraction to serializer-based GetValue<T>.
src/Exceptionless.Core/Services/SlackService.cs Uses serializer-aware Slack token access.
src/Exceptionless.Core/Serialization/ObjectToInferredTypesConverter.cs Refines number inference logic for STJ dynamic object conversion.
src/Exceptionless.Core/Serialization/LowerCaseUnderscorePropertyNamesContractResolver.cs Deletes Newtonsoft contract resolver (obsolete after STJ migration).
src/Exceptionless.Core/Serialization/JsonSerializerOptionsExtensions.cs Defines Exceptionless STJ defaults (snake_case, safe encoder, converters, empty-collection skipping).
src/Exceptionless.Core/Serialization/ExceptionlessNamingStrategy.cs Deletes Newtonsoft naming strategy (replaced by STJ naming policy).
src/Exceptionless.Core/Serialization/EmptyCollectionModifier.cs Adds STJ type info modifier to omit empty collections.
src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Adds STJ IElasticsearchSerializer for NEST with custom converters.
src/Exceptionless.Core/Serialization/ElasticJsonNetSerializer.cs Deletes Newtonsoft-based NEST serializer.
src/Exceptionless.Core/Serialization/ElasticConnectionSettingsAwareContractResolver.cs Deletes Newtonsoft resolver used by old NEST serializer.
src/Exceptionless.Core/Serialization/DynamicTypeContractResolver.cs Deletes Newtonsoft dynamic contract resolver.
src/Exceptionless.Core/Serialization/DataObjectConverter.cs Deletes Newtonsoft-based model/data converter.
src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs Switches Elasticsearch client wiring to STJ serializer.
src/Exceptionless.Core/Plugins/WebHook/Default/010_VersionOnePlugin.cs Uses serializer-based typed accessors; adds [JsonPropertyName] for v1 webhook compatibility.
src/Exceptionless.Core/Plugins/WebHook/Default/005_SlackPlugin.cs Uses serializer-based typed accessors for Slack webhook creation.
src/Exceptionless.Core/Plugins/Formatting/FormattingPluginBase.cs Converts formatting plugins to depend on ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/99_DefaultFormattingPlugin.cs Updates default formatting/slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/60_LogFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/50_SessionFormattingPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/40_UsageFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/30_NotFoundFormattingPlugin.cs Updates typed accessors + IP retrieval to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/20_ErrorFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/10_SimpleErrorFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/05_ManualStackingFormattingPlugin.cs Updates manual stacking info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventUpgrader/EventUpgraderContext.cs Migrates upgrader DOM from JObject/JArray to JsonObject/JsonArray.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V2_EventUpgrade.cs Rewrites upgrader logic from Newtonsoft DOM to JsonNode DOM.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R850_EventUpgrade.cs Rewrites upgrader logic to JsonObject.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R844_EventUpgrade.cs Rewrites upgrader logic to JsonObject.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R500_EventUpgrade.cs Rewrites upgrader logic to JsonObject and normalizes date string formatting.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/GetVersion.cs Rewrites version detection to JsonNode APIs.
src/Exceptionless.Core/Plugins/EventProcessor/Default/90_RemovePrivateInformationPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/80_AngularPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs Updates typed accessors + session start creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/50_GeoPlugin.cs Updates IP extraction to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/45_EnvironmentInfoPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/40_RequestInfoPlugin.cs Updates typed accessors and request exclusion processing to accept ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/30_SimpleErrorPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/20_ErrorPlugin.cs Updates typed accessors + ErrorSignature to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/10_NotFoundPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/0_ThrottleBotsPlugin.cs Updates request-info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/03_ManualStackingPlugin.cs Updates manual stacking info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventParser/Default/LegacyErrorParserPlugin.cs Migrates legacy parsing to JsonNode + STJ options.
src/Exceptionless.Core/Plugins/EventParser/Default/JsonEventParserPlugin.cs Uses ITextSerializer for parsing event payloads instead of Newtonsoft.
src/Exceptionless.Core/Models/SummaryData.cs Adjusts nullability/required-ness to align with STJ behaviors.
src/Exceptionless.Core/Models/StackSummaryModel.cs Adjusts nullability/required-ness to align with STJ behaviors.
src/Exceptionless.Core/Models/Stack.cs Removes Newtonsoft enum converter attribute; keeps STJ converter.
src/Exceptionless.Core/Models/SlackToken.cs Replaces Newtonsoft [JsonProperty] with STJ [JsonPropertyName]; updates SlackAttachment ctor to use ITextSerializer.
src/Exceptionless.Core/Models/Messaging/ReleaseNotification.cs Removes required modifiers (likely to avoid STJ required-member enforcement).
src/Exceptionless.Core/Models/Event.cs Adds [JsonExtensionData] + IJsonOnDeserialized merge into Data.
src/Exceptionless.Core/Mail/Mailer.cs Switches user-info extraction to serializer-based accessors.
src/Exceptionless.Core/Jobs/WebHooksJob.cs Switches webhook POST payload handling to STJ PostAsJsonAsync + options.
src/Exceptionless.Core/Jobs/EventPostsJob.cs Uses serializer-based event bytes for retries.
src/Exceptionless.Core/Jobs/EventNotificationsJob.cs Updates request-info access to use ITextSerializer.
src/Exceptionless.Core/Jobs/CloseInactiveSessionsJob.cs Updates identity extraction to use ITextSerializer.
src/Exceptionless.Core/Extensions/RequestInfoExtensions.cs Requires serializer for post-data exclusion parsing (STJ).
src/Exceptionless.Core/Extensions/ProjectExtensions.cs Makes Slack token extraction serializer-aware via GetValue<T>.
src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs Updates helpers to accept ITextSerializer for typed accessors.
src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Adds STJ JsonNode helpers used by upgraders/tests (formatting, mutation helpers).
src/Exceptionless.Core/Extensions/JsonExtensions.cs Removes Newtonsoft helpers, retains JSON-type detection helpers.
src/Exceptionless.Core/Extensions/EventExtensions.cs Updates typed accessor APIs to accept ITextSerializer; switches GetBytes to serializer bytes.
src/Exceptionless.Core/Extensions/ErrorExtensions.cs Updates stacking target helper to use serializer-based error access.
src/Exceptionless.Core/Extensions/DataDictionaryExtensions.cs Reworks GetValue<T> around STJ/ITextSerializer, adds case-insensitive JsonElement handling.
src/Exceptionless.Core/Exceptionless.Core.csproj Removes Newtonsoft/JsonNet packages.
src/Exceptionless.Core/Bootstrapper.cs Removes Newtonsoft setup; registers STJ options + SystemTextJsonSerializer in DI.
AGENTS.md Documents the new STJ-only serialization architecture and security posture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 87 out of 87 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Comment thread src/Exceptionless.Core/Serialization/ObjectToInferredTypesConverter.cs Outdated
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/DataDictionaryExtensions.cs Fixed
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch from e6a3b35 to c292e13 Compare March 1, 2026 00:52
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/ProjectExtensions.cs Fixed
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch 2 times, most recently from 4d880ad to 09134ce Compare March 1, 2026 01:19
Copilot AI review requested due to automatic review settings March 22, 2026 00:51
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch from 09134ce to 13aa277 Compare March 22, 2026 00:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 86 out of 86 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/Exceptionless.Tests/IntegrationTestsBase.cs Outdated
Comment thread tests/Exceptionless.Tests/Exceptionless.Tests.csproj Outdated
Comment thread src/Exceptionless.Core/Exceptionless.Core.csproj Outdated
Comment thread src/Exceptionless.Core/Utility/TypeHelper.cs Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 117 out of 117 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Repositories/OrganizationRepository.cs
Comment thread src/Exceptionless.Core/Jobs/Elastic/DataMigrationJob.cs Fixed
Copilot AI review requested due to automatic review settings March 22, 2026 19:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Repositories/OrganizationRepository.cs
Comment thread src/Exceptionless.Core/Serialization/EmptyCollectionModifier.cs
Comment thread tests/Exceptionless.Tests/IntegrationTestsBase.cs
Comment thread src/Exceptionless.Core/Utility/ExtensibleObject.cs Fixed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@niemyjski niemyjski requested a review from Copilot March 22, 2026 21:14
niemyjski added 5 commits May 22, 2026 07:26
…S container

The persistent Elasticsearch test container retains stale unversioned event
indices (e.g., test-events-2026.05.06) from previous runs. These conflict with
DailyIndex aliases of the same name, causing the migration tests to fail.

Added ResetDataAsync override that deletes only date-prefixed unversioned
indices ({baseName}-20*) before base cleanup, without touching versioned
indices ({baseName}-v1-*) used by parallel test classes.
Introduces a new AI agent and skill for extremely strict code quality audits. The `thermo-nuclear-code-quality-review` skill defines ambitious standards for maintainability, abstraction quality, file size management (1k-line rule), and prevention of spaghetti code growth, emphasizing structural simplification and "code judo" moves.
https://github.com/cursor/plugins/tree/main/cursor-team-kit/skills/thermo-nuclear-code-quality-review
Standardizes `Task.Delay` calls to consistently include both `TimeProvider` for
testability and `CancellationToken` for proper operation cancellation.

Also clarifies the behavior of `JavaScriptEncoder` for `System.Text.Json`
serialization and improves Elasticsearch test cleanup robustness by
adding `Conflicts.Proceed` to delete-by-query requests.
Comment thread src/Exceptionless.Web/Controllers/SavedViewController.cs Outdated
Comment thread src/Exceptionless.Web/Utility/OpenApi/DateTimeSchemaTransformer.cs Outdated
Comment thread tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs Outdated
Comment thread tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs Outdated
Comment thread tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs Outdated
Comment thread tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs Outdated
Comment thread tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs Outdated
Comment thread src/Exceptionless.Web/Controllers/SavedViewController.cs Outdated
Comment thread tests/Exceptionless.Tests/TestWithServices.cs Outdated
Comment thread src/Exceptionless.AppHost/Exceptionless.AppHost.csproj Outdated
}

[Fact]
public async Task CanBuildScopedRangeQueryAsync()
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is no longer needed due to FoundatioFx/Foundatio.Parsers#234 we need to look at the visitor and this nad remove them now... We just need to update to 8.0.0-beta1.11 for repositories and account for changes in this pr too FoundatioFx/Foundatio.Repositories#281

@github-actions
Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.Web 60% 47% 4025
Exceptionless.AppHost 20% 15% 88
Exceptionless.Insulation 24% 23% 203
Exceptionless.Core 69% 61% 7870
Summary 64% (13267 / 20845) 55% (6461 / 11720) 12186

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants