Version 1.0.0 (#2)

* feat: add taskiq deduplication

* doc: rework class comment + update README

* fix: make build_deduplication_key private

* fix: raise_on_duplicate is now False by default

* chore: remove pre_execute

* chore: add documentation
This commit is contained in:
d3vyce
2026-05-02 14:49:42 +02:00
committed by GitHub
parent 413c4c42f3
commit b4ee3fce45
21 changed files with 3184 additions and 1 deletions
+66
View File
@@ -0,0 +1,66 @@
# Taskiq Deduplication
Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently.
[![CI](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml/badge.svg)](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/d3vyce/taskiq-deduplication/graph/badge.svg)](https://codecov.io/gh/d3vyce/taskiq-deduplication)
[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
---
**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr)
**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication)
---
## Installation
```bash
uv add "taskiq-deduplication"
```
## Quick Start
```python
from taskiq_redis import ListQueueBroker
from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
)
@broker.task
async def send_report(user_id: int) -> None:
...
# First dispatch acquires the lock — succeeds.
await send_report.kiq(user_id=42)
# Second dispatch while the first is queued or running — raises.
try:
await send_report.kiq(user_id=42)
except DuplicateTaskError:
pass # already queued or running
```
## Features
- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis queue lock, before they reach the broker.
- **Worker-side detection** — logs concurrent duplicate executions without raising, keeping `SmartRetryMiddleware` safe from retry storms.
- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments.
- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
## License
MIT License - see [LICENSE](LICENSE) for details.
## Contributing
Contributions are welcome! Please feel free to submit issues and pull requests.
+13
View File
@@ -0,0 +1,13 @@
# API Reference
## Middleware
::: taskiq_deduplication.RedisDeduplicationMiddleware
options:
show_source: false
## Exceptions
::: taskiq_deduplication.DuplicateTaskError
options:
show_source: false
+120
View File
@@ -0,0 +1,120 @@
# Usage
## Setup
Register `RedisDeduplicationMiddleware` on your broker before the application starts:
```python
from taskiq_redis import ListQueueBroker
from taskiq_deduplication import RedisDeduplicationMiddleware
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
)
```
## Middleware options
| Parameter | Type | Default | Description |
|---|---|---|---|
| `redis_url` | `str` | — | Redis connection URL passed to `Redis.from_url`. |
| `default_deduplication` | `bool` | `True` | Whether deduplication is enabled for all tasks by default. Set `False` to opt-in per task instead of opting out. |
| `default_ttl` | `int` | `300` | Default lock TTL in seconds. Overridden per task with the `deduplication_ttl` label. |
| `key_prefix` | `str` | `"taskiq:deduplication"` | Prefix for all Redis lock keys. |
```python
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
RedisDeduplicationMiddleware(
redis_url="redis://localhost:6379",
default_deduplication=True,
default_ttl=60,
key_prefix="myapp:dedup",
),
)
```
## How it works
When a task is dispatched, the middleware acquires a Redis lock keyed on the task's
fingerprint. Any subsequent dispatch with the same fingerprint raises
`DuplicateTaskError` while the lock is held. The lock is released automatically when
the task completes or fails.
## Handling duplicates
When a duplicate is detected, the middleware logs a warning and raises
`DuplicateTaskError`, which prevents the task from reaching the broker.
Catch it at the call site if you need to handle it explicitly:
```python
from taskiq_deduplication import DuplicateTaskError
try:
await my_task.kiq(user_id=42)
except DuplicateTaskError:
pass # task is already queued or running
```
## Per-task label overrides
Labels can be set at the task level (applied to every call) or at call time.
### Task-level (decorator)
```python
@broker.task(deduplication_ttl=60)
async def my_task(user_id: int) -> None:
...
```
### Call-level (kicker)
```python
await my_task.kicker().with_labels(deduplication_ttl=60).kiq(user_id=42)
```
### Available labels
| Label | Type | Description |
|---|---|---|
| `deduplication` | `bool` | Set `False` to opt out of deduplication entirely for this task. |
| `deduplication_ttl` | `int` | Lock TTL in seconds. Overrides the middleware `default_ttl`. |
| `deduplication_key` | `str` | Explicit lock key. Skips fingerprint computation entirely. |
| `deduplication_key_fields` | `list[str]` | Subset of kwargs to include in the fingerprint. Ignored if `deduplication_key` is set. |
## Fingerprint and key customisation
By default the lock key is a SHA-256 fingerprint of the task name and all kwargs.
### Explicit key
Use `deduplication_key` when you want full control over the lock key, regardless of
the kwargs:
```python
@broker.task(deduplication_key="send-welcome-email")
async def send_welcome_email(user_id: int, locale: str) -> None:
...
```
All calls to this task share a single lock, no matter what arguments are passed.
### Partial key (key fields)
Use `deduplication_key_fields` to deduplicate only on a subset of kwargs.
Here, two calls with the same `user_id` but different `locale` are treated as
duplicates:
```python
@broker.task(deduplication_key_fields=["user_id"])
async def send_welcome_email(user_id: int, locale: str) -> None:
...
```
## Opting out per task
```python
@broker.task(deduplication=False)
async def always_run(payload: str) -> None:
...
```