- Published on
What You Need to Know About AI Caching
- Authors
- Name
- DeepHorus
The cost of using large models is mainly determined by the price of input and output tokens, but one item—the "cached hit price"—is often overlooked. Taking the DeepSeek V4 Flash model as an example, the input price for a cache hit is only 2 cents, which is one-fiftieth of the miss price (1 yuan). This huge price difference has attracted widespread attention from developers, and how to use caching to reduce costs has become a hot topic.
When processing prompts, large models need to convert tokens into vectors and compute the attention relationships among all tokens. This process is extremely compute-intensive, so model companies charge for input tokens. In multi-turn conversations or long-horizon tasks, each turn requires the previous results to be resubmitted to the model, and these repeated "prefix" computations can be fully cached to avoid consuming compute power repeatedly. This is the core value of the caching mechanism: significantly saving compute power and speeding up processing.
Caching is cheap because cached-hit requests consume almost no compute power, and the fee charged is equivalent only to the cost of storage space. However, caches are not permanent. According to tests, cache retention times vary widely among providers: Anthropic is 5 minutes, DeepSeek is 10 minutes, OpenAI gradually expires in 10 to 30 minutes, and Google gradually expires within 1 hour. Once the time limit is exceeded, the cache is deleted, and the same input will be recalculated at full price.
Taking DeepSeek as an example, if the conversation interval exceeds 10 minutes, the previous prompt needs to be recalculated, and the cost jumps from 2 cents to 1 yuan. This huge cost difference has prompted AI Agent tools to try their best to extend the cache validity period in design. When users are inactive for a long time, most agents automatically send a request every 30 seconds to keep the cache active, but this practice also incurs additional request costs.
By calculation, if there is no input for 10 minutes, sending an activation request every 30 seconds requires 20 times in total, and the cumulative cost is not negligible. Considering that the shortest cache period among providers is also 5 minutes, the latest suggestion is to extend the activation interval to 4 minutes, which both saves costs and ensures the cache remains valid. This strategic adjustment is crucial for applications that frequently call large models, as it can significantly reduce long-term operating costs.
The rational use of the caching mechanism has become one of the key means to optimize the cost of AI applications. Developers need to design scientific activation strategies based on the caching characteristics of the model they use, balancing response speed and cost. As competition among large models intensifies, future caching technology may become more intelligent, dynamically adjusting retention time to further lower the barrier to entry for AI applications.
