The job failed. The retry succeeded. The dashboard is green again.
That sounds like a successful recovery, but one more question is worth asking: did the business operation run once, or did we run it twice?
Consider a hypothetical background job that creates a record in an external system. The external system accepts the request, but the worker loses its connection before receiving confirmation. When the job runs again, our application still has no record of success.
Retrying the method is easy. Deciding whether to repeat the external action is the real architecture problem.
A Job Attempt Is Not a Business Operation
Microsoft's October 12 discussion of Durable Task use cases is a timely reminder that workflow recovery and external outcomes are different concerns. A framework can coordinate execution without making every integration safe to repeat.
This matters even if you never adopt durable orchestration. Hangfire's own guidance recommends background methods that can safely run again after interruption.
Start by separating the work's identity from the attempt's identity. One requested export might have several execution attempts. Those attempts should not become several unrelated exports simply because each worker generated a new identifier.
Create and persist the business-operation identity when accepting the work. Reuse it during recovery. If the requested content changes, explicitly decide whether that is a new operation rather than silently reusing the old identity for different instructions.
A Completion Flag Does Not Close the Failure Window
A common design checks a local completion flag, calls the provider, and then marks the operation complete. That helps skip work whose completion is already recorded. It does not solve a crash between the external commit and the local update.
It also doesn't prevent two workers from reading an incomplete operation before either writes its result.
A database uniqueness constraint can protect local admission. An atomic claim can coordinate local workers. Neither, by itself, makes a separate provider participate in your SQL transaction.
For an integration review, I would write down the contract before writing another retry loop:
Business operation: one requested export
Operation ID: persisted before dispatch; reused across attempts
Payload: immutable snapshot or validated payload fingerprint
Local admission: unique operation ID; atomic worker claim
Provider protection: documented idempotency or unique request identity
Unknown outcome: query by original identity before resubmission
Unresolvable outcome: stop automatic submission and request review
This is a design template, not executable code or a guarantee that your provider supports these capabilities. Its value is making the missing pieces visible.
Unknown Is a Useful Business State
A timeout should not automatically become “nothing happened.” It means the caller didn't get a definitive answer within its waiting period.
Use a state that preserves that uncertainty. Your operations team should be able to distinguish a confirmed rejection from an outcome that needs investigation.
If the provider supports idempotency keys, verify the contract: key scope, retention period, concurrent requests, payload-mismatch behavior, and how to retrieve the original result. A header with a promising name is not enough.
If the provider cannot deduplicate or expose a reliable lookup, you may not be able to guarantee automatic duplicate-free recovery. That limitation belongs in the design and support procedure, not hidden inside an exception handler.
Reliable Dispatch Is a Separate Boundary
Another gap appears before execution even starts: saving a request and enqueuing its job are often separate writes.
For database-backed applications, consider recording the operation and an outbox entry in the same local transaction. A dispatcher can then deliver the durable intent. Because delivery can repeat, the consumer must still recognize the original operation identity.
The outbox prevents intent from being lost between local acceptance and dispatch. It does not make the eventual external action exactly once. Keep those responsibilities separate when reviewing the implementation.
Test the Awkward Moment
A normal exception test is not enough. In a controlled test environment, inject failures at the boundaries:
- Stop the worker after the provider accepts the operation but before local success is saved.
- Deliver the same operation to two workers concurrently.
- Drop the provider response while letting the provider action finish.
- Replay the dispatch after the local request transaction commits.
- Try reusing the same identity with different content and verify rejection.
Inspect both systems afterward. Count the externally visible effects, check the stored receipt, and confirm that unresolved outcomes become visible to an operator. A green job status is not the only assertion.
Choose the Simplest Recovery Model That Works
A short, independent job may need only durable admission, safe retries, and good diagnostics. A process with long waits and several dependent stages may benefit from orchestration. Replay-based orchestrators also have determinism constraints; external I/O belongs in activities rather than arbitrary orchestrator code.
Undo is not automatically easier. Compensation is business-specific, can fail, and may require human intervention. Do not delete or reverse uncertain work just to make a dashboard look clean.
Pick one important background operation this week. Trace its acceptance, dispatch, external effect, and local completion. Then ask what happens if the process disappears between each pair of steps.
The goal is not more infrastructure. It is a recovery path that understands what the business operation actually did.
Which failure window has been hardest to test in your background jobs: duplicate delivery, a lost response, or recovery after the external action already succeeded?