diff --git a/content/projects/taskiq-deduplication/index.md b/content/projects/taskiq-deduplication/index.md new file mode 100644 index 0000000..a21f626 --- /dev/null +++ b/content/projects/taskiq-deduplication/index.md @@ -0,0 +1,80 @@ +--- +title: "Taskiq Deduplication" +date: 2026-01-25 +slug: "taskiq-deduplication" +showAuthor: false +showWordCount: false +showReadingTime: false +showRelatedContent: false +showPagination: false +tags: ["python", "taskiq", "package"] +--- + +![overview](featured.png) +{{< github repo="d3vyce/taskiq-deduplication" >}} + +> Redis-backed deduplication middleware for Taskiq that prevents duplicate tasks from being queued or executed concurrently + + +[![CI](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml/badge.svg)](https://github.com/d3vyce/taskiq-deduplication/actions/workflows/ci.yml) +[![codecov](https://codecov.io/gh/d3vyce/taskiq-deduplication/graph/badge.svg)](https://codecov.io/gh/d3vyce/taskiq-deduplication) +[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty) +[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv) +[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) +[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) + +--- + +**Documentation**: [https://taskiq-deduplication.d3vyce.fr](https://taskiq-deduplication.d3vyce.fr) + +**Source Code**: [https://github.com/d3vyce/taskiq-deduplication](https://github.com/d3vyce/taskiq-deduplication) + +--- + +## Installation + +```bash +uv add taskiq-deduplication +``` + +## Quick Start + +```python +from taskiq_redis import ListQueueBroker +from taskiq_deduplication import RedisDeduplicationMiddleware, DuplicateTaskError + +broker = ListQueueBroker("redis://localhost:6379").with_middlewares( + RedisDeduplicationMiddleware(redis_url="redis://localhost:6379"), +) + +@broker.task +async def send_report(user_id: int) -> None: + ... + +# First dispatch acquires the lock — succeeds. +await send_report.kiq(user_id=42) + +# Second dispatch while the first is queued or running — raises. +try: + await send_report.kiq(user_id=42) +except DuplicateTaskError: + pass # already queued or running +``` + +## Features + +- **Sender-side deduplication** — rejects duplicate tasks at dispatch time via a Redis queue lock, before they reach the broker. +- **Worker-side detection** — logs concurrent duplicate executions without raising, keeping `SmartRetryMiddleware` safe from retry storms. +- **Configurable TTL** — set a global default or override per task with the `deduplication_ttl` label. +- **Explicit lock key** — pin any task to a fixed Redis key with `deduplication_key`, bypassing fingerprint computation entirely. +- **Partial fingerprint** — deduplicate on a subset of kwargs with `deduplication_key_fields`, ignoring irrelevant arguments. +- **Per-task opt-out** — disable deduplication for individual tasks with the `deduplication` label. + +## License + +MIT License - see [LICENSE](LICENSE) for details. + +## Contributing + +Contributions are welcome! Please feel free to submit issues and pull requests.