Does storing each KV pair make sense? Especially when the model only queries a small portion of them in practice.

The idea behind KVzap is straightforward—by learning to identify which cache entries are unlikely to be used in subsequent queries and proactively deleting them. The result is that the cache size can be compressed to 1/2 to 1/4 of the original, with almost no impact on performance.

This intelligent, dynamic dependency-based KV cache pruning method has practical significance for improving model inference efficiency and reducing storage costs. Especially in large-scale deployment scenarios, the potential for optimization is quite substantial.