mirror of
https://github.com/d3vyce/taskiq-deduplication.git
synced 2026-08-04 19:14:07 +00:00
6b1b745d5f
* ⬆ bump ruff from 0.15.17 to 0.16.0 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.17 to 0.16.0. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.15.17...0.16.0) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.16.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix: ruff warnings --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: d3vyce <nicolas.sudres@proton.me>
72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# Taskiq Deduplication
|
|
|
|
Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently.
|
|
|
|
[](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml)
|
|
[](https://codecov.io/gh/d3vyce/taskiq-deduplication)
|
|
[](https://github.com/astral-sh/ty)
|
|
[](https://github.com/astral-sh/uv)
|
|
[](https://github.com/astral-sh/ruff)
|
|
[](https://www.python.org/downloads/)
|
|
[](https://opensource.org/licenses/MIT)
|
|
|
|
---
|
|
|
|
**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr)
|
|
|
|
**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication)
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
uv add "taskiq-deduplication"
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from taskiq_redis import ListQueueBroker
|
|
from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError
|
|
|
|
broker = ListQueueBroker("redis://localhost:6379").with_middlewares(
|
|
RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"),
|
|
)
|
|
|
|
|
|
@broker.task
|
|
async def send_report(user_id: int) -> None: ...
|
|
|
|
|
|
# First dispatch acquires the lock — succeeds.
|
|
await send_report.kiq(user_id=42)
|
|
|
|
# Second dispatch while the first is queued or running — raises.
|
|
try:
|
|
await send_report.kiq(user_id=42)
|
|
except DuplicateTaskError:
|
|
pass # already queued or running
|
|
```
|
|
|
|
## Features
|
|
|
|
- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis lock, before they reach the broker.
|
|
- **Handle to the winning task** — a rejected caller gets the winner's `task_id` on the error, so it can await the winner's result instead of re-kicking.
|
|
- **Atomic lock release** — lock is released on completion or error via a Lua check-and-delete; only the owning task can release its lock.
|
|
- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label.
|
|
- **Lock heartbeat** — a background task re-extends the lock TTL while the task runs, so long-running tasks keep their lock instead of expiring mid-execution and admitting a duplicate.
|
|
- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely.
|
|
- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments (positional arguments are excluded).
|
|
- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label.
|
|
- **Startup resilience** — automatic reconnection with exponential backoff if Redis is unavailable at broker startup.
|
|
- **Fail-open mode** — opt in with `fail_open` to keep dispatching tasks when Redis is unreachable at runtime, trading deduplication for availability.
|
|
|
|
## License
|
|
|
|
MIT License - see [LICENSE](LICENSE) for details.
|
|
|
|
## Contributing
|
|
|
|
Contributions are welcome! Please feel free to submit issues and pull requests.
|