Version 1.0.0 (#2)

* feat: add taskiq deduplication * doc: rework class comment + update README * fix: make build_deduplication_key private * fix: raise_on_duplicate is now False by default * chore: remove pre_execute * chore: add documentation
2026-08-04 19:14:07 +00:00 · 2026-05-02 14:49:42 +02:00
parent 413c4c42f3
commit b4ee3fce45
21 changed files with 3184 additions and 1 deletions
@@ -1,2 +1,66 @@
-# Taskiq deduplication
+# Taskiq Deduplication

+Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently.
+
+[![CI](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml/badge.svg)](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/d3vyce/taskiq-deduplication/graph/badge.svg)](https://codecov.io/gh/d3vyce/taskiq-deduplication)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+---
+
+**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr)
+
+**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication)
+
+---
+
+## Installation
+
+```bash
+uv add taskiq-deduplication
+```
+
+## Quick Start
+
+```python
+from taskiq_redis import ListQueueBroker
+from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError
+
+broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
+    RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
+)
+
+@broker.task
+async def send_report(user_id: int) -> None:
+    ...
+
+# First dispatch acquires the lock — succeeds.
+await send_report.kiq(user_id=42)
+
+# Second dispatch while the first is queued or running — raises.
+try:
+    await send_report.kiq(user_id=42)
+except DuplicateTaskError:
+    pass  # already queued or running
+```
+
+## Features
+
+- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis queue lock, before they reach the broker.
+- **Worker-side detection** — logs concurrent duplicate executions without raising, keeping `SmartRetryMiddleware` safe from retry storms.
+- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
+- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
+- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments.
+- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.