mirror of
https://github.com/d3vyce/taskiq-deduplication.git
synced 2026-08-04 19:14:07 +00:00
feat: add lock heartbeat to prevent mid-execution expiry (#64)
This commit is contained in:
@@ -53,6 +53,7 @@ except DuplicateTaskError:
|
||||
- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis lock, before they reach the broker.
|
||||
- **Atomic lock release** — lock is released on completion or error via a Lua check-and-delete; only the owning task can release its lock.
|
||||
- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
|
||||
- **Lock heartbeat** — a background task re-extends the lock TTL while the task runs, so long-running tasks keep their lock instead of expiring mid-execution and admitting a duplicate.
|
||||
- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
|
||||
- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments.
|
||||
- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
|
||||
|
||||
@@ -23,6 +23,8 @@ broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
| `key_prefix` | `str` | `"taskiq:deduplication"` | Prefix for all Redis lock keys. |
|
||||
| `startup_retries` | `int` | `3` | Number of connection attempts during broker startup. |
|
||||
| `startup_retry_delay` | `float` | `1.0` | Base delay in seconds between retries (exponential backoff: delay × 2^n). |
|
||||
| `heartbeat` | `bool` | `True` | Whether to periodically re-extend the lock TTL while the task runs (see [Long-running tasks](#long-running-tasks-and-the-heartbeat)). |
|
||||
| `heartbeat_interval` | `float \| None` | `None` | Seconds between heartbeat refreshes. When `None`, defaults to a third of the task's TTL (1s floor). |
|
||||
|
||||
```python
|
||||
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
@@ -37,6 +39,31 @@ broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
)
|
||||
```
|
||||
|
||||
## Long-running tasks and the heartbeat
|
||||
|
||||
The lock is created with a TTL so a crashed worker cannot leak it forever. Without
|
||||
any refresh, a task that runs longer than its TTL would let the lock expire
|
||||
**mid-execution**, allowing a duplicate to be dispatched.
|
||||
|
||||
To prevent this, the middleware starts a background **heartbeat** in `pre_execute`
|
||||
that re-extends the lock TTL while the task runs (atomically, only if the lock is
|
||||
still owned by the running task). It is cancelled when the task completes or fails.
|
||||
This means you do **not** need to size `default_ttl` to your slowest task — the TTL
|
||||
only needs to outlive a single heartbeat interval; it acts purely as a safety net
|
||||
for worker crashes.
|
||||
|
||||
```python
|
||||
RedisDeduplicationMiddleware(
|
||||
redis_url="redis://localhost:6379",
|
||||
default_ttl=60, # safety-net TTL; refreshed every ~20s while running
|
||||
heartbeat_interval=20, # optional; defaults to default_ttl / 3
|
||||
)
|
||||
```
|
||||
|
||||
If you disable the heartbeat (`heartbeat=False`), the invariant **TTL must exceed
|
||||
the slowest task** applies: set `default_ttl` (or the per-task `deduplication_ttl`
|
||||
label) above your worst-case task duration, or duplicates may slip through.
|
||||
|
||||
## Startup resilience
|
||||
|
||||
On startup the middleware verifies the Redis connection with a `PING`. If Redis is
|
||||
|
||||
Reference in New Issue
Block a user