mirror of
https://github.com/d3vyce/taskiq-deduplication.git
synced 2026-08-04 19:14:07 +00:00
d3vyceandGitHub
b4ee3fce45
* feat: add taskiq deduplication * doc: rework class comment + update README * fix: make build_deduplication_key private * fix: raise_on_duplicate is now False by default * chore: remove pre_execute * chore: add documentation
67 lines
2.9 KiB
Markdown
67 lines
2.9 KiB
Markdown
# Taskiq Deduplication
|
|
|
|
Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently.
|
|
|
|
[](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml)
|
|
[](https://codecov.io/gh/d3vyce/taskiq-deduplication)
|
|
[](https://github.com/astral-sh/ty)
|
|
[](https://github.com/astral-sh/uv)
|
|
[](https://github.com/astral-sh/ruff)
|
|
[](https://www.python.org/downloads/)
|
|
[](https://opensource.org/licenses/MIT)
|
|
|
|
---
|
|
|
|
**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr)
|
|
|
|
**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication)
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
uv add "taskiq-deduplication"
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from taskiq_redis import ListQueueBroker
|
|
from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError
|
|
|
|
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
|
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
|
|
)
|
|
|
|
@broker.task
|
|
async def send_report(user_id: int) -> None:
|
|
...
|
|
|
|
# First dispatch acquires the lock — succeeds.
|
|
await send_report.kiq(user_id=42)
|
|
|
|
# Second dispatch while the first is queued or running — raises.
|
|
try:
|
|
await send_report.kiq(user_id=42)
|
|
except DuplicateTaskError:
|
|
pass # already queued or running
|
|
```
|
|
|
|
## Features
|
|
|
|
- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis queue lock, before they reach the broker.
|
|
- **Worker-side detection** — logs concurrent duplicate executions without raising, keeping `SmartRetryMiddleware` safe from retry storms.
|
|
- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
|
|
- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
|
|
- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments.
|
|
- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
|
|
|
|
## License
|
|
|
|
MIT License - see [LICENSE](LICENSE) for details.
|
|
|
|
## Contributing
|
|
|
|
Contributions are welcome! Please feel free to submit issues and pull requests.
|