- Published on
Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID
- Authors
- Name
- DeepHorus
Netflix's engineering team has announced a breakthrough in handling wide partitions in Apache Cassandra, reducing read latency from seconds to milliseconds. The solution, called dynamic repartitioning, splits oversized partitions transparently without requiring application changes. This innovation addresses a critical performance bottleneck in Netflix's TimeSeries Abstraction, a platform that manages petabytes of temporal event data with millisecond latency requirements.
Apache Cassandra 4.x serves as the underlying storage for TimeSeries, chosen for its throughput, latency, cost, and operational maturity. Time-series data is organized into partitions that group events by identifier and time range. These partitions can grow 'wide' as events accumulate, causing tail read latencies to rise into seconds, leading to read timeouts, garbage collection pauses, high CPU utilization, and thread queueing.
Dynamic repartitioning detects oversized partitions on the read path via byte counting and a Kafka event, prioritizing immutable partitions for splitting. Bloom filters and a cached wide row metadata lookup route reads to smaller child partitions, achieving single-digit microsecond overhead. Checksums, retained original partitions, Data Bridge Spark checks, and shadow comparison guard correctness.
The impact is significant: reads improved from seconds to low double-digit milliseconds, tail latency fell to approximately 200 milliseconds, and partitions exceeding 500 MB remained available. This performance gain is crucial for Netflix's high-read-throughput workloads, where scaling up the cluster would be costly.
Netflix's approach combines automated monitoring with asynchronous splitting. A background worker monitors partition-size percentiles via Cassandra's virtual tables, computing adjustment factors to maintain target densities (typically 2β10 MiB). When partition sizes deviate, the worker updates future time slices with corrected strategies, such as adjusting time bucket intervals from 60 seconds to 604,800 seconds.
The solution handles three scenarios: unknown workload estimates, evolving traffic, and data outliers. For datasets where most partitions are oversized, time slice re-partitioning updates future time chunks. For cases where only a percentage of IDs are wide, dynamic repartitioning offers three options: do nothing, repartition, or use the new automated method.
Looking ahead, Netflix plans to extend dynamic repartitioning to handle more complex cases and integrate it tightly with operational monitoring. This innovation not only improves latency and reduces costs but also provides a scalable model for other companies dealing with wide Cassandra partitions. The engineering team emphasizes that automation is key to sustainable management of thousands of datasets.
