DevOps & monitoring

How Close cut CI test times from 12 minutes to under three

Close reduced its 58,000-test suite runtime to under three minutes by replacing static test splitting with a dynamic orchestrator and spot instances.

Illustration of dynamic task distribution in a CI pipeline
Illustration created for this article

The engineering team at Close has reduced the runtime of its 58,000 integration tests from approximately twelve minutes to under three minutes. This performance leap was achieved by replacing static test distribution with a dynamic orchestration model and leveraging cloud spot instances for worker nodes.

What happened

For over a decade, Close relied on deterministic test splitting to manage its growing test suite. The team used plugins like pytest-split to divide tests across multiple parallel runners based on historical timing data. While this approach initially kept runtimes manageable, it eventually hit a performance ceiling. As the application grew, fixed-cost setup steps such as pulling container images, starting datastores, and collecting the full test suite began to dominate the total runtime. Even with seventeen parallel runners, the wall-clock time stabilized around twelve minutes because every runner had to perform heavy setup work regardless of how many tests it actually executed.

To break through this barrier, the team adopted an architecture inspired by tools like currents.dev. Instead of pre-assigning tests to specific runners, they introduced a central orchestrator. This orchestrator collects test files quickly without loading the full application context. It then dynamically distributes individual test batches to a large pool of workers. This shift allowed the heavy setup tasks to scale more efficiently, as workers only collect and run the specific tests assigned to them. The team further optimized costs and capacity by using fifty-four spot instance workers alongside two on-demand runners, achieving massive parallelism without prohibitive expenses.

Key details

  • The test suite contains nearly 58,000 integration tests, most of which interact with real datastores.
  • Previous static splitting methods resulted in a twelve-minute runtime due to fixed setup overheads.
  • The new dynamic orchestrator collects test files in parallel with worker startup, reducing idle time.
  • Workers use spot instances for cost-effective scaling, with on-demand runners reserved for critical paths.
  • Test collection is optimized by skipping deep file descent and outputting discovered files as JSON.
  • Batches are sorted in descending order by estimated runtime to improve load balancing across workers.

Background

Continuous Integration (CI) pipelines often struggle with the "fixed cost" problem. In traditional setups, every parallel runner must initialize the entire application environment, including databases and dependencies, before running its subset of tests. As the number of tests increases, teams add more runners to maintain speed. However, if the setup time remains constant per runner, adding more machines yields diminishing returns. The total time becomes limited by the slowest runner and the initial setup phase rather than the actual test execution.

Dynamic orchestration solves this by decoupling test collection from execution. A lightweight coordinator identifies what needs to be tested and feeds work to available workers on demand. This allows workers to start up and begin testing sooner, as they do not need to parse the entire codebase upfront. It also enables better resource utilization, as idle workers can pick up remaining tasks while others finish, preventing the "straggler" effect where one slow runner holds up the entire pipeline.

Why it matters

For teams running their own software, CI speed directly impacts developer productivity and deployment frequency. Long feedback loops discourage frequent commits and make it harder to identify the cause of failures. By reducing test runtime from twelve minutes to under three, Close enables developers to iterate faster and merge code with greater confidence. This efficiency is crucial for maintaining high velocity in mid-sized companies where engineering resources are finite.

Additionally, the shift to spot instances demonstrates how architectural changes can reduce infrastructure costs while increasing capacity. Spot instances are significantly cheaper than on-demand servers but can be terminated unexpectedly. By designing the system to handle worker loss gracefully, teams can leverage these savings without sacrificing reliability. This approach is particularly relevant for self-hosted environments where budget constraints often limit the ability to simply throw more hardware at performance problems.

What you can do

  • Audit your CI pipeline to identify fixed-cost steps that do not scale with parallelization.
  • Consider implementing a dynamic test distributor if static splitting no longer reduces runtime.
  • Optimize test collection by avoiding full application loads during the discovery phase.
  • Sort test batches by estimated duration to ensure even workload distribution across runners.
  • Use spot instances for non-critical workers to increase parallelism within budget limits.
  • Instrument your CI pipeline with tracing tools to measure true end-to-end wall-clock time.

More news

All news