diff --git a/README.md b/README.md index a0b9a2ea..ef10ed8d 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite, - [Performance considerations](#performance-considerations) - [Failed jobs and retries](#failed-jobs-and-retries) - [Error reporting on jobs](#error-reporting-on-jobs) + - [Jobs interrupted by non-graceful process death](#jobs-interrupted-by-non-graceful-process-death) - [Puma plugin](#puma-plugin) - [Jobs and transactional integrity](#jobs-and-transactional-integrity) - [Recurring tasks](#recurring-tasks) @@ -664,6 +665,27 @@ class ApplicationMailer < ActionMailer::Base end ``` +### Jobs interrupted by non-graceful process death + +When a process dies without a clean shutdown (for example, `SIGKILL`ed by the OS or the container runtime because of memory limits), the jobs it was running can't be released back to their queues. Once another process notices the missing heartbeats and prunes the dead process's registration, its in-flight jobs are marked as failed with `SolidQueue::Processes::ProcessPrunedError`. Solid Queue deliberately doesn't retry these automatically: the job itself might be what's killing the process (for example, a job that exhausts the container's memory), and retrying it blindly would just kill the next worker too. + +If you know your jobs are idempotent and want to implement your own recovery policy, you can subscribe to the `fail_many_claimed.solid_queue` event, which includes the error and the affected job IDs in its payload: + +```ruby +# config/initializers/solid_queue_recovery.rb +ActiveSupport::Notifications.subscribe("fail_many_claimed.solid_queue") do |event| + if event.payload[:error].is_a?(SolidQueue::Processes::ProcessPrunedError) + SolidQueue::FailedExecution.where(job_id: event.payload[:job_ids]).each do |failed_execution| + # Apply your own safeguard against retrying the same job in a loop, + # e.g. a counter in the job's arguments or a cap stored elsewhere. + failed_execution.retry + end + end +end +``` + +The event is emitted in the process that performs the pruning (or the supervisor when it reaps a crashed fork, with `SolidQueue::Processes::ProcessExitError`), so make sure the subscription is set up in an initializer, where all Solid Queue processes will load it. + ## Puma plugin We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add diff --git a/app/models/solid_queue/claimed_execution.rb b/app/models/solid_queue/claimed_execution.rb index 26569269..9a606e0b 100644 --- a/app/models/solid_queue/claimed_execution.rb +++ b/app/models/solid_queue/claimed_execution.rb @@ -48,6 +48,7 @@ def fail_all_with(error) payload[:process_ids] = executions.map(&:process_id).uniq payload[:job_ids] = executions.map(&:job_id).uniq payload[:size] = executions.size + payload[:error] = error end end end diff --git a/lib/solid_queue/log_subscriber.rb b/lib/solid_queue/log_subscriber.rb index bb6b67da..4bb52baf 100644 --- a/lib/solid_queue/log_subscriber.rb +++ b/lib/solid_queue/log_subscriber.rb @@ -16,7 +16,10 @@ def release_many_claimed(event) end def fail_many_claimed(event) - warn formatted_event(event, action: "Fail claimed jobs", **event.payload.slice(:job_ids, :process_ids)) + attributes = event.payload.slice(:job_ids, :process_ids) + attributes[:error] = formatted_error(event.payload[:error]) if event.payload[:error] + + warn formatted_event(event, action: "Fail claimed jobs", **attributes) end def release_claimed(event) diff --git a/test/integration/instrumentation_test.rb b/test/integration/instrumentation_test.rb index 3ff0b13f..4fd8a295 100644 --- a/test/integration/instrumentation_test.rb +++ b/test/integration/instrumentation_test.rb @@ -164,6 +164,9 @@ class InstrumentationTest < ActiveSupport::TestCase assert_equal 1, events.count assert_event events.first, "fail_many_claimed", process_ids: [ process.id ], job_ids: jobs.map(&:id), size: 3 + + error = events.first.last[:error] + assert_instance_of SolidQueue::Processes::ProcessPrunedError, error end test "errors when deregistering processes are included in deregister_process events" do