The question
Can an exact deduplication structure use less memory while maintaining high throughput within a precisely defined retention window?
This is a proposed research direction. No novel algorithm, completed benchmark or performance gain is claimed.
Why it matters
Streaming and ingestion systems could reduce state costs while preserving their duplicate-handling contract.
Search space
Compact ID representations, partitioned expiry structures and exact membership indexes.
What would count as progress?
- Agreement with an exact reference
- bytes per retained ID
- throughput
- expiry latency
- recovery correctness.
A useful result improves a measured trade-off against strong baselines on held-out workloads, with reproducible code and documented limitations.
Baselines and boundaries
Compare a hash set with expiry bookkeeping and an embedded key-value store. Define event-time versus processing-time windows, late events and restart behavior. An approximate filter cannot be the sole authority for exact deduplication.
Starting reference
Where you could contribute
Help define representative, legally shareable workloads; critique the correctness contract; implement baselines; propose candidates; or reproduce measurements. Negative results and failure cases are useful contributions.
A detailed research brief is being prepared. Access requests will become available when that brief and the request service are ready.