- Published on
A Managed Semantic Cache That Cuts LLM API Costs by Up to 90% and Returns Cache Hits Up to 15x Faster
- Authors
- Name
- DeepHorus
Redis has launched LangCache, a semantic caching service cutting LLM API costs by up to 90% and returning hits up to 15x faster. In public preview on Redis Cloud, it matches prompts by meaning rather than exact text. It uses a REST API and Python/JavaScript SDKs.
Production LLM applications often receive the same question phrased differently. Support assistants and RAG pipelines field identical intents thousands of times daily, yet most stacks treat each paraphrase as a fresh, fully billed request. Without semantic caching, every variation triggers a complete generation.
Prefix caching only partially solves this. When requests share a system prompt, the engine reuses computed KV states, but the request still reaches the LLM and the full answer is decoded. A prefix-cache hit is a cheaper generation call, not an avoided one.
LangCache moves caching outside the model and stores generated responses. The app sends the prompt to a search endpoint, where LangCache generates an embedding and runs a vector search over stored entries. If a similar entry clears the threshold, the cached response is returned and no LLM call occurs. On a miss, the app calls its LLM and caches the prompt and response.
The service includes default or bring-your-own embedding models. Cache behavior is controlled through similarity thresholds, TTLs, and eviction policies. Built on Redisās vector database, it works with any LLM provider and language.
A cache hit eliminates input tokens, output tokens, and decoding latency. In a demo on a paraphrased question, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis notes that features may change during the preview. Hit rates and savings are monitored from the Redis Cloud console. As semantic caching matures, it could become a standard layer in LLM infrastructure, especially for high-volume support and RAG workloads.
