mirror of
https://github.com/d3vyce/taskiq-deduplication.git
synced 2026-08-04 19:14:07 +00:00
Version 1.0.0 (#2)
* feat: add taskiq deduplication * doc: rework class comment + update README * fix: make build_deduplication_key private * fix: raise_on_duplicate is now False by default * chore: remove pre_execute * chore: add documentation
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# Taskiq Deduplication
|
||||
|
||||
Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently.
|
||||
|
||||
[](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml)
|
||||
[](https://codecov.io/gh/d3vyce/taskiq-deduplication)
|
||||
[](https://github.com/astral-sh/ty)
|
||||
[](https://github.com/astral-sh/uv)
|
||||
[](https://github.com/astral-sh/ruff)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
|
||||
---
|
||||
|
||||
**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr)
|
||||
|
||||
**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
uv add "taskiq-deduplication"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from taskiq_redis import ListQueueBroker
|
||||
from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError
|
||||
|
||||
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
|
||||
)
|
||||
|
||||
@broker.task
|
||||
async def send_report(user_id: int) -> None:
|
||||
...
|
||||
|
||||
# First dispatch acquires the lock — succeeds.
|
||||
await send_report.kiq(user_id=42)
|
||||
|
||||
# Second dispatch while the first is queued or running — raises.
|
||||
try:
|
||||
await send_report.kiq(user_id=42)
|
||||
except DuplicateTaskError:
|
||||
pass # already queued or running
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis queue lock, before they reach the broker.
|
||||
- **Worker-side detection** — logs concurrent duplicate executions without raising, keeping `SmartRetryMiddleware` safe from retry storms.
|
||||
- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
|
||||
- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
|
||||
- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments.
|
||||
- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see [LICENSE](LICENSE) for details.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please feel free to submit issues and pull requests.
|
||||
@@ -0,0 +1,13 @@
|
||||
# API Reference
|
||||
|
||||
## Middleware
|
||||
|
||||
::: taskiq_deduplication.RedisDeduplicationMiddleware
|
||||
options:
|
||||
show_source: false
|
||||
|
||||
## Exceptions
|
||||
|
||||
::: taskiq_deduplication.DuplicateTaskError
|
||||
options:
|
||||
show_source: false
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
# Usage
|
||||
|
||||
## Setup
|
||||
|
||||
Register `RedisDeduplicationMiddleware` on your broker before the application starts:
|
||||
|
||||
```python
|
||||
from taskiq_redis import ListQueueBroker
|
||||
from taskiq_deduplication import RedisDeduplicationMiddleware
|
||||
|
||||
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
|
||||
)
|
||||
```
|
||||
|
||||
## Middleware options
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `redis_url` | `str` | — | Redis connection URL passed to `Redis.from_url`. |
|
||||
| `default_deduplication` | `bool` | `True` | Whether deduplication is enabled for all tasks by default. Set `False` to opt-in per task instead of opting out. |
|
||||
| `default_ttl` | `int` | `300` | Default lock TTL in seconds. Overridden per task with the `deduplication_ttl` label. |
|
||||
| `key_prefix` | `str` | `"taskiq:deduplication"` | Prefix for all Redis lock keys. |
|
||||
|
||||
```python
|
||||
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
||||
RedisDeduplicationMiddleware(
|
||||
redis_url="redis://localhost:6379",
|
||||
default_deduplication=True,
|
||||
default_ttl=60,
|
||||
key_prefix="myapp:dedup",
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
When a task is dispatched, the middleware acquires a Redis lock keyed on the task's
|
||||
fingerprint. Any subsequent dispatch with the same fingerprint raises
|
||||
`DuplicateTaskError` while the lock is held. The lock is released automatically when
|
||||
the task completes or fails.
|
||||
|
||||
## Handling duplicates
|
||||
|
||||
When a duplicate is detected, the middleware logs a warning and raises
|
||||
`DuplicateTaskError`, which prevents the task from reaching the broker.
|
||||
Catch it at the call site if you need to handle it explicitly:
|
||||
|
||||
```python
|
||||
from taskiq_deduplication import DuplicateTaskError
|
||||
|
||||
try:
|
||||
await my_task.kiq(user_id=42)
|
||||
except DuplicateTaskError:
|
||||
pass # task is already queued or running
|
||||
```
|
||||
|
||||
## Per-task label overrides
|
||||
|
||||
Labels can be set at the task level (applied to every call) or at call time.
|
||||
|
||||
### Task-level (decorator)
|
||||
|
||||
```python
|
||||
@broker.task(deduplication_ttl=60)
|
||||
async def my_task(user_id: int) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
### Call-level (kicker)
|
||||
|
||||
```python
|
||||
await my_task.kicker().with_labels(deduplication_ttl=60).kiq(user_id=42)
|
||||
```
|
||||
|
||||
### Available labels
|
||||
|
||||
| Label | Type | Description |
|
||||
|---|---|---|
|
||||
| `deduplication` | `bool` | Set `False` to opt out of deduplication entirely for this task. |
|
||||
| `deduplication_ttl` | `int` | Lock TTL in seconds. Overrides the middleware `default_ttl`. |
|
||||
| `deduplication_key` | `str` | Explicit lock key. Skips fingerprint computation entirely. |
|
||||
| `deduplication_key_fields` | `list[str]` | Subset of kwargs to include in the fingerprint. Ignored if `deduplication_key` is set. |
|
||||
|
||||
## Fingerprint and key customisation
|
||||
|
||||
By default the lock key is a SHA-256 fingerprint of the task name and all kwargs.
|
||||
|
||||
### Explicit key
|
||||
|
||||
Use `deduplication_key` when you want full control over the lock key, regardless of
|
||||
the kwargs:
|
||||
|
||||
```python
|
||||
@broker.task(deduplication_key="send-welcome-email")
|
||||
async def send_welcome_email(user_id: int, locale: str) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
All calls to this task share a single lock, no matter what arguments are passed.
|
||||
|
||||
### Partial key (key fields)
|
||||
|
||||
Use `deduplication_key_fields` to deduplicate only on a subset of kwargs.
|
||||
Here, two calls with the same `user_id` but different `locale` are treated as
|
||||
duplicates:
|
||||
|
||||
```python
|
||||
@broker.task(deduplication_key_fields=["user_id"])
|
||||
async def send_welcome_email(user_id: int, locale: str) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
## Opting out per task
|
||||
|
||||
```python
|
||||
@broker.task(deduplication=False)
|
||||
async def always_run(payload: str) -> None:
|
||||
...
|
||||
```
|
||||
Reference in New Issue
Block a user