Memory Optimization: Advanced KV Cache Management

Implement a basic KV cache for autoregressive decoding and measure its memory growth over the generation sequence.

Implement and compare different KV cache eviction strategies (e.g., sliding window, token dropping) and evaluate their impact on generation quality.

Implement prefix caching to reuse KV cache entries for requests that share a common prompt.

Explain the memory fragmentation problem in naive KV cache allocation and the core concept of paged memory management.

Implement a paged KV cache allocator that manages non-contiguous physical memory blocks for storing logical token sequences.

Integrate the paged KV cache into an inference loop and benchmark the reduction in memory waste under a simulated concurrent workload.