- Published on
Stripe Uses Graph Search and State Machines to Automate Database Remediation
- Authors
- Name
- DeepHorus
Stripe has automated database incident recovery by modeling its global MongoDB infrastructure as a graph and using search algorithms with state machines. The system computes and executes remediation plans automatically, adapting to different shard layouts. This reduced database-related pager alerts by about 30%, or 200 fewer pages per year, and eliminated an estimated 12 days of unhealthy shard states annually.
The previous hard-coded remediation system struggled with fragile dependencies and multi-failure scenarios. In six months, operators were paged 124 times for misconfigured shards and 32 times for single-node-down cases with extra health issues. Critical operations like index builds were blocked for an average of one hour per incident.
Stripe modeled infrastructure as a graph, with nodes representing components and edges capturing relationships. Remediation uses composable rules with explicit state transitions instead of fixed workflows. The planner moved from breadth-first search to Dijkstra's algorithm to prioritize lower-cost recovery plans.
Because Dijkstra explores all reachable states, the system can offer partial remediation when no complete path exists. It returns the path to the least misconfigured state. This lets the same logic adapt as infrastructure evolves, without manual runbook updates.
Stripe's engineers say runbooks encode known procedures, while a state machine discovers novel ones. Scott MacVicar, head of developer infrastructure at Stripe, said the challenge is fixing daily hardware degradation without burning out on-call engineers. Stripe plans to extend the framework to topology changes, blue-green deployments, and planned maintenance.
Stripe joins Uber and Meta in automating infrastructure operations, reflecting a broader industry trend. This pattern of state machine modeling and runtime pathfinding offers a compelling alternative to accumulating more runbooks. As distributed systems grow, such automation will become increasingly critical for reliability and engineer well-being.
