Skip to content

blehnen/DotNetWorkQueue

Repository files navigation

DotNetWorkQueue

License LGPLv2.1 CI Coverity status codecov

A producer / distributed consumer library for .NET applications. Targets .NET 8.0 and .NET 10.0.

High-level features:

  • Queue / de-queue POCOs for distributed processing
  • Queue / process compiled LINQ expressions
  • Re-occurring job scheduler

See the Wiki for in-depth documentation.


Installation

Package Description NuGet
DotNetWorkQueue Core library with all abstractions, interfaces, and default implementations NuGet

Transports

Package Description NuGet
DotNetWorkQueue.Transport.SqlServer SQL Server transport using Microsoft.Data.SqlClient NuGet
DotNetWorkQueue.Transport.PostgreSQL PostgreSQL transport using Npgsql NuGet
DotNetWorkQueue.Transport.Redis Redis transport using StackExchange.Redis NuGet
DotNetWorkQueue.Transport.SQLite SQLite transport using System.Data.SQLite NuGet
DotNetWorkQueue.Transport.LiteDb LiteDB embedded NoSQL transport NuGet
DotNetWorkQueue.Transport.Memory In-memory transport for testing and lightweight scenarios NuGet

Dashboard

Package Description NuGet
DotNetWorkQueue.Dashboard.Api REST API for monitoring and managing queues across all transports. Queue status, message counts, error tracking, consumer tracking, and admin actions. ASP.NET Core with Swagger/OpenAPI. NuGet
DotNetWorkQueue.Dashboard.Ui Blazor Server web UI (MudBlazor) for browsing queues, viewing messages, managing errors, and monitoring consumers. Connects to the Dashboard API. NuGet
DotNetWorkQueue.Dashboard.Client Typed API client and consumer registration client. Sends heartbeats with running metric counters (processed, errors, rollbacks, poison messages). No dependency on the core library — HttpClient and System.Text.Json only. NuGet

Docker

A pre-built Docker image runs both the Dashboard UI and API in a single container — no separate API container needed:

docker pull blehnen74/dotnetworkqueue-dashboard:latest
docker run -d -p 8080:8080 -v "$(pwd)/appsettings.json:/app/appsettings.json:ro" blehnen74/dotnetworkqueue-dashboard:latest

See Docker Hub for tags and docker/dashboard/README.md for full configuration (authentication, API keys, SQLite/LiteDB volume mounts, Docker Compose).


Metrics: Built-in metrics are provided via System.Diagnostics.Metrics in the core library. No additional package is needed. To collect and export metrics, configure OpenTelemetry.Metrics in your host application. A sample Grafana dashboard (Prometheus data source) is available — import it into Grafana to visualize queue metrics.


Usage — POCO

Transport Producer Consumer
SQL Server SQLServerProducer SQLServerConsumer
PostgreSQL PostgreSQLProducer PostgreSQLConsumer
Redis RedisProducer RedisConsumer
SQLite SQLiteProducer SQLiteConsumer
LiteDb LiteDbProducer LiteDbConsumer

Usage — LINQ Expressions

You can queue LINQ expressions to be executed instead of POCOs. This makes producers and consumers generic — they no longer need to be message-specific. The examples below are not transport-specific and assume any queue creation steps have already been performed.

Note: It is possible for a producer to queue work that a consumer cannot process. For a consumer to execute a LINQ expression, all referenced types must be resolvable.

SQLiteProducerLinq/Program.cs

Consumer

The consumer is generic and can process any LINQ expression, but it must be able to resolve all types the expression references.


Usage — Job Scheduler

Jobs are scheduled using standard cron expressions (5-field or 6-field with seconds), parsed by Cronos. The scheduler and consumers are separate; schedulers only queue work, while standard LINQ consumers handle processing.

Any LINQ statement supported by a LINQ producer can be scheduled via the scheduler.

Multiple schedulers sharing the same schedule can be run for redundancy, but it is important that the clocks on all machines are in sync, or that the same time provider is injected into both schedulers and consumers. See the Wiki for more details.

Note: Using multiple machines with out-of-sync clocks may produce unexpected results. Server-based transports often provide solutions for this. See the Wiki.

See crontab.guru for the cron expression format. 6-field expressions (with seconds) are also supported.

Scheduler

The scheduler and its container must remain in scope for as long as you are scheduling work. Disposing or allowing the scheduler to go out of scope will stop all queuing.

SQliteScheduler/Program.cs

To consume and process scheduled jobs, use a LINQ Consumer:

SQLiteSchedulerConsumer/Program.cs


Samples & Examples


Building the Source

You'll need Visual Studio 2022/2026 (any edition) and the .NET 8.0 / 10.0 SDKs installed.

All references are on NuGet. Building from Visual Studio should restore all required packages automatically.


License

Copyright © 2015–2026 Brian Lehnen

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 2.1 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/.


Third-Party Libraries

Core (DotNetWorkQueue): SimpleInjector, Polly, Newtonsoft.Json, OpenTelemetry, Microsoft.Extensions.Caching.Memory, Microsoft.IO.RecyclableMemoryStream, GuerrillaNtp

Scheduling: Cronos (cron parsing), CronExpressionDescriptor (human-readable descriptions)

Package Dependencies
Transport.SqlServer Microsoft.Data.SqlClient
Transport.PostgreSQL Npgsql
Transport.Redis StackExchange.Redis, MsgPack-CLI
Transport.SQLite System.Data.SQLite
Transport.LiteDb LiteDb
Dashboard.Api Swashbuckle.AspNetCore

Tests: MSTest, NSubstitute, AutoFixture, FluentAssertions, Tynamix.ObjectFiller


Developed with:

About

A work queue for dot.net with SQL server, SQLite, Redis and PostGreSQL backends

Topics

Resources

License

Stars

Watchers

Forks

Contributors

Languages