Distributed Job Scheduler

A production-inspired distributed job scheduling platform — users authenticate, create projects, each project owns queues, queues hold jobs (immediate, delayed, scheduled-cron, batch). A pool of independent worker processes atomically claims jobs from queues, executes them concurrently under per-queue concurrency limits, retries failures with configurable backoff, and dead-letters permanent failures. Every step is observable via structured logs, Prometheus metrics, and Grafana dashboards.

Stack: Rust (Axum / Tokio / SQLx) · TypeScript + Vite (React) · PostgreSQL 16 · Docker Compose

---

Table of Contents

• Quick Start
• Seed Data
• Architecture
• Database Design
• Job Lifecycle
• API Overview
• Frontend
• Observability
• Project Structure
• Configuration
• Design Decisions

---

Quick Start

Prerequisites

• Docker and Docker Compose v2+
• OR Rust 1.75+ and Node.js 20+ for local development

Run with Docker Compose (recommended)

# Bring up the entire stack
docker compose up --build

# Scale workers independently
docker compose up --build --scale worker=3

The stack will be available at:

Service	URL	Auth
Frontend dashboard	http://localhost:8080	admin@example.com / password123
API server	http://localhost:3000	JWT Bearer token
Prometheus	http://localhost:9090	—
Grafana	http://localhost:3001	admin / admin

Run locally (development)

# 1. Start PostgreSQL
docker run -d --name pg-scheduler \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=job_scheduler \
  -p 5432:5432 postgres:16-alpine

# 2. Backend (env vars use APP_ prefix — see .env.example)
export APP_DATABASE_URL=postgres://postgres:postgres@localhost:5432/job_scheduler
cargo run --bin api-server        # API on :3000
cargo run --bin worker             # Worker process
cargo run --bin scheduler-daemon   # Scheduler singleton

# 3. Frontend
cd frontend
npm install
npm run dev                        # Vite dev server on :5173, proxies /api → :3000

---

Seed Data

On first deployment, three database migrations are applied automatically:

1. Initial schema — all 14 tables, 6 enum types, indexes, triggers
2. Idempotency fix — replaces UNIQUE NULLS NOT DISTINCT with a partial unique index (so multiple jobs can have a NULL idempotency key)
3. Seed data — a complete set of sample data for exploration

Accounts (password: password123)

Email	Full Name	Role in Acme Corp
admin@example.com	Alice Admin	owner
dev@example.com	Bob Developer	admin
viewer@example.com	Carol Viewer	viewer

Projects & Queues

Org	Project	Queues
Acme Corp	Web App	default (×5), high-priority (×10, p=10), email (×3)
Acme Corp	Background Jobs	default (persistent retry, 10 attempts)
Startup Inc	Analytics Pipeline	analytics (×2)

Jobs (10 across 7 statuses)

Status	Count	Examples
completed	3	noop, sleep_100
queued	1	sleep_1000 — waiting for a worker
scheduled	1	welcome email — runs in 2 hours
running	1	sleep_5000 — in-flight
failed	2	Connection refused, Payment timeout
dead_letter	1	Exhausted 5 attempts: downstream 500
cancelled	1	Cancelled by user

Plus: 11 execution records, 7 log lines, 2 DLQ entries (one from max attempts, one from lease expiration), 4 cron scheduled jobs (daily cleanup, hourly digest, weekly report, inactive job), 3 job batches (100-job nightly, 500-job import, 50-job migration).

---

Architecture

System Topology

graph TB
    subgraph "Client Layer"
        Browser["Browser<br/>(React SPA)"]
        API_Client["External API Client<br/>(JWT / API Key)"]
    end

    subgraph "Gateway"
        Nginx["Nginx<br/>(SPA + reverse proxy)"]
    end

    subgraph "Application Layer"
        API["api-server<br/>(Axum HTTP + WS)<br/>N replicas, stateless"]
        Worker["worker<br/>(poll + claim + execute)<br/>N replicas"]
        Scheduler["scheduler-daemon<br/>(advisory-lock singleton)<br/>1 active + 1 standby"]
    end

    subgraph "Data Layer"
        PG[("PostgreSQL 16<br/>jobs · queues · workers<br/>executions · logs · DLQ")]
    end

    subgraph "Observability"
        Prom["Prometheus<br/>/metrics scraping"]
        Grafana["Grafana<br/>dashboards"]
    end

    Browser --> Nginx
    API_Client --> Nginx
    Nginx -->|"REST + WebSocket"| API
    Worker -->|"poll + claim<br/>LISTEN/NOTIFY"| PG
    Scheduler -->|"promote · cron · reap"| PG
    API -->|"queries"| PG
    PG --> Prom
    API --> Prom
    Worker --> Prom
    Scheduler --> Prom
    Prom --> Grafana

Loading

Hexagonal (Ports & Adapters) Architecture

The backend is a modular monolith compiled as three independent binaries from a single Cargo workspace. Each binary scales independently in production without being separate repos.

graph TB
    subgraph "bin/"
        API_BIN["api-server / main.rs"]
        WORKER_BIN["worker / main.rs"]
        SCHED_BIN["scheduler-daemon / main.rs"]
    end

    subgraph "crates/ — Application"
        API_CRATE["api<br/>Axum routes · middleware · DTOs<br/>error.rs · validation.rs"]
        WORKER_RT["worker-runtime<br/>executor · heartbeat · shutdown"]
    end

    subgraph "crates/ — Domain Logic"
        SCHED_CORE["scheduler-core<br/>claim · retry · cron · reaper"]
    end

    subgraph "crates/ — Domain (zero I/O)"
        DOMAIN["domain<br/>entities · value objects · ports (traits)"]
    end

    subgraph "crates/ — Infrastructure"
        INFRA["infra-postgres<br/>SQLx repository adapters"]
        COMMON["common<br/>config · telemetry · errors"]
    end

    subgraph "External"
        PG[("PostgreSQL")]
    end

    API_BIN --> API_CRATE
    WORKER_BIN --> WORKER_RT
    SCHED_BIN --> SCHED_CORE

    API_CRATE --> DOMAIN
    API_CRATE --> INFRA
    WORKER_RT --> DOMAIN
    WORKER_RT --> INFRA
    SCHED_CORE --> DOMAIN

    DOMAIN -. "defines traits" .-> INFRA
    INFRA --> PG
    COMMON --> DOMAIN

Loading

Dependency rule (enforced by Cargo.toml):

• domain depends on nothing in the workspace — zero I/O dependencies
• infra-postgres and scheduler-core depend on domain only
• api and worker-runtime depend on domain + scheduler-core + infra-postgres — never on each other
• Binaries are thin entry points wiring everything together

Claim Flow (the core reliability guarantee)

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant PG as PostgreSQL

    W1->>PG: BEGIN
    W2->>PG: BEGIN
    Note over W1,PG: CTE: SELECT id FROM jobs<br/>WHERE status='queued'<br/>ORDER BY priority DESC, run_at ASC<br/>FOR UPDATE SKIP LOCKED<br/>LIMIT free_permits
    W1->>PG: Locks rows 1,2,3 (SKIP LOCKED)
    W2->>PG: Locks rows 4,5 (skips 1,2,3)
    W1->>PG: UPDATE SET status='claimed', locked_by=W1<br/>RETURNING *
    W2->>PG: UPDATE SET status='claimed', locked_by=W2<br/>RETURNING *
    W1->>PG: COMMIT
    W2->>PG: COMMIT
    Note over W1,W2: Zero contention, zero duplicate execution

Loading

Workers never SELECT then UPDATE separately — that's a race condition. The CTE with FOR UPDATE SKIP LOCKED is a single atomic operation.

---

Database Design

Entity-Relationship Diagram

erDiagram
    users ||--o{ organization_members : "belongs to"
    organizations ||--o{ organization_members : "has"
    organizations ||--o{ projects : "owns"
    projects ||--o{ queues : "contains"
    projects ||--o{ retry_policies : "defines"
    projects ||--o{ job_batches : "groups"
    queues ||--o{ jobs : "holds"
    queues ||--o{ scheduled_jobs : "templates"
    queues ||--o{ dead_letter_queue : "failed"
    retry_policies ||--o| queues : "default for"
    retry_policies ||--o| jobs : "overrides"
    retry_policies ||--o| scheduled_jobs : "applies to"
    workers ||--o{ worker_heartbeats : "emits"
    workers ||--o{ jobs : "locks"
    workers ||--o{ job_executions : "executes"
    jobs ||--o{ job_executions : "has attempts"
    jobs ||--o{ job_logs : "produces"
    jobs ||--o{ job_dependencies : "depends on"
    jobs ||--o| dead_letter_queue : "dead-letters to"
    job_executions ||--o{ job_logs : "contains"
    job_batches ||--o{ jobs : "batches"
    scheduled_jobs ||--o{ jobs : "spawns"

    users {
        uuid id PK
        text email UK
        text password_hash
        text full_name
        boolean is_active
        timestamptz created_at
        timestamptz updated_at
    }

    organizations {
        uuid id PK
        text name
        text slug UK
        timestamptz created_at
    }

    organization_members {
        uuid org_id PK_FK
        uuid user_id PK_FK
        org_role role
    }

    projects {
        uuid id PK
        uuid org_id FK
        text name
        text api_key_hash UK
        uuid created_by FK
        timestamptz created_at
        timestamptz updated_at
    }

    queues {
        uuid id PK
        uuid project_id FK
        text name
        int priority
        int max_concurrency
        boolean is_paused
        uuid default_retry_policy_id FK
        int visibility_timeout_secs
        timestamptz created_at
        timestamptz updated_at
    }

    retry_policies {
        uuid id PK
        uuid project_id FK
        text name
        retry_strategy strategy
        bigint base_delay_ms
        float multiplier
        bigint max_delay_ms
        int max_attempts
        boolean jitter
        timestamptz created_at
    }

    jobs {
        uuid id PK
        uuid queue_id FK
        uuid scheduled_job_id FK
        uuid batch_id FK
        text job_type
        jsonb payload
        int priority
        job_status status
        timestamptz run_at
        text idempotency_key
        uuid retry_policy_id FK
        int max_attempts_override
        int attempt_count
        uuid locked_by FK
        timestamptz locked_at
        timestamptz lease_expires_at
        jsonb result
        text error_message
        text trace_id
        timestamptz created_at
        timestamptz updated_at
        timestamptz completed_at
    }

    job_executions {
        uuid id PK
        uuid job_id FK
        uuid worker_id FK
        int attempt_number
        execution_status status
        timestamptz started_at
        timestamptz finished_at
        bigint duration_ms
        text error_message
        text error_stack
        jsonb result
        timestamptz created_at
    }

    job_logs {
        bigserial id PK
        uuid execution_id FK
        uuid job_id FK
        timestamptz ts
        text level
        text message
    }

    workers {
        uuid id PK
        text hostname
        int pid
        text version
        uuid[] subscribed_queues
        int max_concurrency
        worker_status status
        timestamptz started_at
        timestamptz last_seen_at
    }

    worker_heartbeats {
        bigserial id PK
        uuid worker_id FK
        timestamptz ts
        int active_job_count
        float cpu_percent
        float memory_mb
        jsonb metadata
    }

    dead_letter_queue {
        uuid id PK
        uuid job_id FK
        uuid queue_id FK
        jsonb payload_snapshot
        text failure_reason
        int attempt_count
        text last_error
        timestamptz moved_at
        boolean resolved
        uuid resolved_by FK
        timestamptz resolved_at
        text resolution_action
    }

    scheduled_jobs {
        uuid id PK
        uuid queue_id FK
        text name
        text cron_expression
        text timezone
        text job_type
        jsonb payload_template
        uuid retry_policy_id FK
        boolean is_active
        timestamptz next_run_at
        timestamptz last_run_at
        timestamptz created_at
        timestamptz updated_at
    }

    job_batches {
        uuid id PK
        uuid project_id FK
        text name
        int total_jobs
        int completed_jobs
        int failed_jobs
        batch_status status
        timestamptz created_at
        timestamptz updated_at
    }

    job_dependencies {
        uuid job_id PK_FK
        uuid depends_on_job_id PK_FK
    }

Loading

Key Indexes

Table	Index	Purpose
jobs	(queue_id, status, priority DESC, run_at) WHERE status = 'queued'	Claim query access path — the most important index in the schema
jobs	(status, run_at) WHERE status = 'scheduled'	Scheduler promotion sweep
jobs	(lease_expires_at) WHERE status IN ('claimed','running')	Reaper expired-lease sweep
jobs	(queue_id, idempotency_key) WHERE idempotency_key IS NOT NULL	Idempotent submission (partial unique)
job_executions	(job_id, attempt_number)	Retry history per job
job_logs	(job_id, ts)	Log tail per job
worker_heartbeats	(worker_id, ts DESC)	Staleness detection

Enums

graph LR
    subgraph "job_status"
        scheduled --> queued
        queued --> claimed
        claimed --> running
        running --> completed
        running --> failed
        failed --> scheduled
        failed --> dead_letter
        scheduled --> dead_letter
        claimed --> dead_letter
        running --> dead_letter
        queued --> cancelled
        scheduled --> cancelled
    end

Loading

Six enum types: job_status, execution_status, retry_strategy, worker_status, batch_status, org_role.

---

Job Lifecycle

Full State Machine

stateDiagram-v2
    [*] --> scheduled : Job created with<br/>run_at in the future
    [*] --> queued : Job created with<br/>run_at ≤ now()
    scheduled --> queued : scheduler-daemon<br/>promotion sweep
    queued --> claimed : worker atomically claims<br/>(FOR UPDATE SKIP LOCKED)
    claimed --> running : worker begins execution
    running --> completed : handler returns Ok
    running --> failed : handler returns Err
    failed --> scheduled : attempt_count < max_attempts<br/>(retry with backoff)
    failed --> dead_letter : attempt_count ≥ max_attempts
    scheduled --> dead_letter : lease expired + attempts exhausted
    claimed --> scheduled : lease expired + attempts remain
    running --> scheduled : lease expired + attempts remain
    queued --> cancelled : user cancels
    scheduled --> cancelled : user cancels
    completed --> [*]
    dead_letter --> [*]
    cancelled --> [*]

    note right of failed
        Retry backoff reuses the
        "scheduled" state.
        attempt_count > 0
        distinguishes retries.
    end note

Loading

Retry Strategies

Strategy	Formula	Jitter
fixed	delay = base_delay_ms	delay × uniform(0.5, 1.5)
linear	delay = base_delay_ms × attempt_count	delay × uniform(0.5, 1.5)
exponential	delay = min(base_delay_ms × multiplier^(attempt_count−1), max_delay_ms)	delay × uniform(0.5, 1.5)

Lease & Reaper

• Every claimed job gets a lease_expires_at = now() + visibility_timeout_secs
• Healthy workers renew leases via the heartbeat ticker (every 10s)
• scheduler-daemon's reaper sweeps expired leases every 15s:
  • attempt_count < max_attempts → reschedule for retry
  • attempt_count ≥ max_attempts → dead letter

---

API Overview

Authentication

Method	Path	Auth	Description
POST	/api/auth/register	Public	Register + get JWT tokens
POST	/api/auth/login	Public	Login + get JWT tokens
POST	/api/auth/refresh	Public	Rotate tokens with refresh token

Protected routes require Authorization: Bearer <token> header. JWT claims contain sub (user UUID), email, exp, iat.

Organizations & Projects

Method	Path	Description
POST	/api/organizations	Create org (user becomes owner)
GET	/api/organizations	List orgs for authenticated user
POST	/api/projects	Create project (returns API key once)
GET	/api/projects?org_id=	List projects
GET	/api/projects/{id}	Get project
DELETE	/api/projects/{id}	Delete project

Queues

Method	Path	Description
POST	/api/queues	Create queue
GET	/api/queues?project_id=	List queues with depth
GET	/api/queues/{id}	Get queue details + depth
DELETE	/api/queues/{id}	Delete queue
POST	/api/queues/{id}/pause	Pause processing
POST	/api/queues/{id}/resume	Resume processing

Jobs

Method	Path	Description
POST	/api/jobs	Submit job (supports idempotency keys, delayed run_at, dependencies)
GET	/api/jobs?queue_id=&status=&job_type=&limit=&offset=	List jobs with filtering & pagination
GET	/api/jobs/{id}	Get job details
POST	/api/jobs/{id}/cancel	Cancel a job
GET	/api/jobs/{id}/executions	Get execution history
GET	/api/jobs/{id}/logs	Get job logs

Dead Letter Queue

Method	Path	Description
GET	/api/dead-letter?queue_id=&resolved=	List DLQ entries (no queue_id = all queues)
POST	/api/dead-letter/{id}/resolve	Resolve entry (requeued or discarded)

Retry Policies

Method	Path	Description
POST	/api/retry-policies	Create policy
GET	/api/retry-policies?project_id=	List policies for project
GET	/api/retry-policies/{id}	Get policy
DELETE	/api/retry-policies/{id}	Delete policy

Scheduled Jobs (Cron)

Method	Path	Description
POST	/api/scheduled-jobs	Schedule a cron job
GET	/api/scheduled-jobs	List active scheduled jobs
GET	/api/scheduled-jobs/{id}	Get scheduled job
POST	/api/scheduled-jobs/{id}/deactivate	Deactivate

Batches, Workers, Health & Metrics

Method	Path	Auth	Description
GET	/api/batches/{id}	JWT	Get batch progress
GET	/api/workers	JWT	List active workers
GET	/healthz	Public	Liveness (always 200)
GET	/readyz	Public	Readiness (checks DB)
GET	/metrics	Public	Prometheus metrics
WS	/ws/events	Public	WebSocket for live events

Error Response Format

All errors return a consistent JSON structure:

{
  "error": "validation_error",
  "message": "Validation failed",
  "details": [
    {"field": "email", "message": "Invalid email format"},
    {"field": "password", "message": "Password must be at least 8 characters"}
  ]
}

Error codes: bad_request, unauthorized, forbidden, not_found, conflict, validation_error, too_many_requests, internal_error.

---

Frontend

A React 18 SPA served by nginx, with TanStack Query (server state), Zustand (auth store), Recharts (dashboard charts), and Tailwind CSS.

Pages

Page	Route	Features
Dashboard	/	Stats cards (queues, queued, running, DLQ, workers), Recharts bar chart (queue depth), pie chart (status distribution), cron job list, org selector
Queues	/queues	Project selector, create queue form, pause/resume, per-queue depth breakdown (queued/scheduled/running/DLQ), link to jobs
Jobs	/jobs	Status filter, paginated data table with status badges, execution history, job log viewer, cancel action
Workers	/workers	Live grid with status dots, auto-refresh every 5s, host/version/concurrency details
Dead Letter	/dead-letter	DLQ entries with requeue/discard, payload snapshot expand, failure reason + last error
Retry Policies	/retry-policies	Creator form, strategy badges (fixed/linear/exponential), delay configs
Scheduled Jobs	/scheduled-jobs	Cron job cards with next-run display, create/deactivate, queue selector
Login	/login	Login/register toggle, Zustand auth store integration, protected route redirect

Authentication Flow

1. User logs in → JWT access + refresh tokens stored in localStorage
2. All API calls attach Authorization: Bearer <token> header automatically
3. Protected pages redirect to /login when unauthenticated
4. Auth state managed via Zustand store (useAuthStore)

---

Observability

Metrics Catalogue

Metric	Type	Labels	Purpose
jobs_created_total	Counter	queue, job_type	Intake rate
jobs_claimed_total	Counter	queue, worker_id	Claim throughput
jobs_completed_total	Counter	queue, job_type	Success throughput
jobs_failed_total	Counter	queue, job_type, terminal	Failure rate
job_execution_duration_seconds	Histogram	queue, job_type	p50/p95/p99 latency
queue_depth	Gauge	queue, status	Backlog per status
worker_active_jobs	Gauge	worker_id	Concurrency utilization
worker_up	Gauge	worker_id	Worker health
dlq_size	Gauge	queue	DLQ growth
lease_expirations_total	Counter	queue	Crashed worker rate

Logging

All three binaries emit structured JSON logs (configurable to pretty-format for development). Every log line carries: timestamp, level, target (module path), span fields (job_id, queue_id, worker_id, attempt), thread.id, thread.name.

Example:

{
  "timestamp": "2026-07-04T12:00:00.000Z",
  "level": "INFO",
  "target": "worker_runtime::executor",
  "fields": {
    "job_id": "a1b2c3d4-...",
    "queue_id": "e5f6a7b8-...",
    "attempt": 1,
    "duration_ms": 234
  },
  "message": "Job completed successfully"
}

---

Project Structure

├── Cargo.toml                          # Workspace root
├── .env.example                        # All configuration (APP_ prefixed)
├── docker-compose.yml                  # Full stack: PG + 3 binaries + frontend + Prom + Grafana
│
├── bin/
│   ├── api-server/main.rs              # Thin: metrics init + axum::serve
│   ├── worker/main.rs                  # Thin: queue discovery + poll loop + graceful shutdown
│   └── scheduler-daemon/main.rs        # Thin: advisory lock + promotion/cron/reaper loops
│
├── crates/
│   ├── domain/                         # Entities, value objects, ports (ZERO I/O)
│   │   └── src/
│   │       ├── entities.rs             # 14 entity structs
│   │       ├── value_objects.rs        # 6 enums (JobStatus, RetryStrategy, …)
│   │       ├── ports.rs                # 11 async repository traits
│   │       └── errors.rs               # DomainError enum
│   │
│   ├── common/                         # Shared infrastructure
│   │   └── src/
│   │       ├── config.rs               # Env-var config (APP_ prefix via figment)
│   │       ├── telemetry.rs            # tracing-subscriber + Prometheus init
│   │       └── error.rs                # CommonError type
│   │
│   ├── infra-postgres/                 # SQLx adapters (implements domain::ports)
│   │   └── src/
│   │       ├── repositories/           # 11 repository impls
│   │       │   ├── job.rs              # ★ Claim query (SKIP LOCKED CTE)
│   │       │   ├── queue.rs            # CRUD + depth (Uuid::nil = all)
│   │       │   ├── worker.rs           # Registration + heartbeats
│   │       │   ├── execution.rs        # Execution records + logs
│   │       │   ├── dead_letter.rs      # DLQ (Option<Uuid> = all queues)
│   │       │   ├── scheduled_job.rs    # Cron template management
│   │       │   ├── user.rs             # Registration + lookup
│   │       │   ├── organization.rs     # Org membership + RBAC
│   │       │   ├── project.rs          # Project CRUD
│   │       │   ├── retry_policy.rs     # Retry policy CRUD
│   │       │   └── batch.rs            # Batch progress tracking
│   │       └── migrations.rs           # sqlx::migrate! runner
│   │
│   ├── scheduler-core/                 # Pure business logic
│   │   └── src/
│   │       ├── claim.rs                # JobClaimer
│   │       ├── retry.rs                # RetryCalculator (pure fn, unit tested)
│   │       ├── cron_eval.rs            # CronEvaluator (parse + next run)
│   │       └── lease_reaper.rs         # LeaseReaper (sweep + DLQ insert)
│   │
│   ├── api/                            # Axum HTTP layer
│   │   └── src/
│   │       ├── lib.rs                  # AppState + build_router()
│   │       ├── dto.rs                  # Request/response DTOs
│   │       ├── error.rs                # AppError (8 variants → HTTP status)
│   │       ├── validation.rs           # Validator (email, password, bounds, enums)
│   │       ├── middleware/
│   │       │   ├── auth.rs             # JWT decode + Claims → Extension
│   │       │   ├── request_id.rs       # UUID X-Request-Id
│   │       │   ├── rate_limit.rs       # Token bucket limiter (bonus)
│   │       │   ├── api_key.rs          # X-API-Key auth (bonus)
│   │       │   └── rbac.rs             # Org role checking (bonus)
│   │       ├── routes/
│   │       │   ├── auth.rs             # register, login, refresh
│   │       │   ├── health.rs           # GET /healthz, /readyz
│   │       │   ├── metrics.rs          # GET /metrics (global handle)
│   │       │   ├── projects.rs         # CRUD /api/projects
│   │       │   ├── queues.rs           # CRUD + pause/resume
│   │       │   ├── jobs.rs             # CRUD + cancel + executions + logs
│   │       │   ├── workers.rs          # GET /api/workers
│   │       │   ├── organizations.rs    # CRUD /api/organizations (real JWT user)
│   │       │   ├── dead_letter.rs      # List + resolve
│   │       │   ├── retry_policies.rs   # CRUD /api/retry-policies
│   │       │   ├── scheduled_jobs.rs   # CRUD + deactivate
│   │       │   └── batches.rs          # GET /api/batches/{id}
│   │       └── websocket.rs            # WS /ws/events
│   │
│   └── worker-runtime/                 # Job execution engine
│       └── src/
│           ├── lib.rs                  # WorkerConfig + QueueRuntime
│           ├── executor.rs             # poll + claim + execute + retry/DLQ
│           ├── heartbeat.rs            # HeartbeatService (ticker + lease renew)
│           ├── shutdown.rs             # GracefulShutdown (SIGTERM → drain)
│           └── job_handlers.rs         # HandlerRegistry + noop/sleep/failing handlers
│
├── migrations/
│   ├── 20240704000001_initial_schema.up.sql    # Full schema
│   ├── 20240704000002_fix_idempotency.up.sql   # Partial unique index
│   └── 20240704000003_seed_data.up.sql         # Sample users/orgs/jobs/DLQ/cron
│
├── docker/
│   ├── Dockerfile.backend              # Multi-stage (cargo-chef, --release)
│   ├── Dockerfile.frontend             # Multi-stage (Vite → nginx)
│   ├── nginx.conf                      # SPA fallback + /api + /ws proxy
│   ├── prometheus.yml                  # Scrape config for all 3 binaries
│   └── grafana/provisioning/           # Datasource auto-provisioning
│
└── frontend/
    ├── package.json
    ├── vite.config.ts
    ├── tailwind.config.js
    └── src/
        ├── main.tsx                    # React root + QueryClient + Router
        ├── App.tsx                     # Navigation + 7 routes + sign out
        ├── api-client/index.ts         # Typed API client (all endpoints)
        ├── stores/auth.ts              # Zustand auth store
        ├── types/index.ts              # TypeScript interfaces
        └── pages/
            ├── Dashboard.tsx           # Recharts + stats + cron list
            ├── Queues.tsx               # Create/pause/resume + depth cards
            ├── Jobs.tsx                 # Filters + table + executions + logs
            ├── Workers.tsx              # Live grid with status dots
            ├── DeadLetter.tsx           # Requeue/discard + payload view
            ├── RetryPolicies.tsx        # Strategy badges + create form
            ├── ScheduledJobs.tsx        # Cron cards + create/deactivate
            └── Login.tsx                # Login/register toggle

---

Configuration

All configuration via environment variables with APP_ prefix (see .env.example):

Variable	Default	Description
APP_DATABASE_URL	postgres://postgres:postgres@localhost:5432/job_scheduler	Connection string
APP_DATABASE_MAX_CONNECTIONS	10	Pool max (per binary)
APP_JWT_SECRET	change-me-in-production	HMAC signing key
APP_JWT_ACCESS_EXPIRY_SECS	900	Access token TTL
APP_JWT_REFRESH_EXPIRY_SECS	604800	Refresh token TTL
APP_API_HOST	0.0.0.0	Bind address
APP_API_PORT	3000	HTTP port
APP_METRICS_PORT	9001	Prometheus port
APP_WORKER_POLL_INTERVAL_MS	1000	Poll interval
APP_WORKER_MAX_CONCURRENCY	10	Per-worker cap
APP_WORKER_HEARTBEAT_INTERVAL_SECS	10	Heartbeat interval
APP_WORKER_GRACEFUL_SHUTDOWN_TIMEOUT_SECS	30	Drain timeout
APP_SCHEDULER_POLL_INTERVAL_MS	2000	Promotion interval
APP_SCHEDULER_LEASE_REAPER_INTERVAL_SECS	15	Reaper interval
APP_SCHEDULER_ADVISORY_LOCK_ID	42	Singleton lock key
APP_LOG_LEVEL	info	Tracing filter
APP_LOG_FORMAT	json	json or pretty

---

Design Decisions

Postgres as the Queue (not Kafka/RabbitMQ)

Pro	Con
One fewer moving part	Lower raw throughput than dedicated broker
SKIP LOCKED for lock-free consumers	Queue depth growth increases claim cost
LISTEN/NOTIFY for event-driven wake-up	Polling is the safety net
Transactional create + enqueue in one write	Hot-table vacuum needed at scale

For throughput up to ~10k jobs/sec, Postgres-as-queue is more than sufficient and operationally simpler.

Retry Backoff Reuses 'scheduled' State

Rather than introducing a separate retrying state, a job waiting for its initial run_at and a job waiting for its next retry are the same: "not ready yet." attempt_count > 0 distinguishes retries.

Idempotency: At-Least-Once + Idempotent Handlers

The system guarantees at-least-once delivery. Exactly-once is provably impossible in a distributed system. The correct target is at-least-once with idempotent job handlers.

Denormalization Trade-offs

1. job_logs.job_id — avoids a JOIN on the highest-write table for "tail this job's logs"
2. dead_letter_queue.payload_snapshot — DLQ entries survive job purges/archives

Known Future Work

• job_logs partitioning — monthly range with retention-based DROP
• Soft deletes on projects and queues
• Job archiving — move old completed/dead_letter jobs to jobs_archive
• Redis rate limiting — cross-replica token buckets
• Workflow dependencies — job_dependencies table exists; promotion sweep should check satisfaction

---