The question
Can two large snapshots identify all missing, changed and duplicate records while minimizing scans and network transfer?
This is a proposed research direction. No novel algorithm, completed benchmark or performance gain is claimed.
Why it matters
Replication, migration, backup and financial reconciliation systems need trustworthy difference detection.
Search space
Hierarchical hashes, maintained partition summaries, adaptive partitioning and set-reconciliation protocols.
What would count as progress?
- Agreement with a full reference diff
- bytes scanned and transferred
- CPU time
- memory
- time to identify all differences.
A useful result improves a measured trade-off against strong baselines on held-out workloads, with reproducible code and documented limitations.
Baselines and boundaries
Compare full sort-merge or hash diffs and partition-hash approaches. Use consistent snapshots and explicit duplicate semantics. Count metadata construction and maintenance costs; hash equality alone is probabilistic evidence.
Where you could contribute
Help define representative, legally shareable workloads; critique the correctness contract; implement baselines; propose candidates; or reproduce measurements. Negative results and failure cases are useful contributions.
A detailed research brief is being prepared. Access requests will become available when that brief and the request service are ready.