On October 19, 2026, between approximately 12:10 AM and 12:55 AM PDT, customers on Prod4 experienced errors and intermittent unavailability of Harness Pipelines services. Heavy pipeline executions drove a sharp rise in CPU and memory use on the pipeline service, and for a short period all service instances were unavailable. Harness engineering scaled the service both vertically and horizontally, which restored service. A fix for the underlying behavior is being rolled out.
On October 19th, high number of concurrent executions with very large execution graphs drove a sharp increase in cluster resources. The service stores and retrieves this full execution graph for visualization purposes and further progressing the pipelines to next states. Our auto-scalaers on Prod4 took longer than usual for scaling resources due to a sharp rise in CPU and memory.
As a part of the fix, the execution graph update will now process graph changes incrementally instead of the full graph. It is being rolled out in a phased manner and will be completely rolled out at the earliest.
Immediate
Harness engineering scaled the pipeline service both vertically (more resources per instance) and horizontally (more instances), and removed instances in an unhealthy state. The instance count stabilized and normal service was restored. The system will stay scaled up till the fix is completely rolled out.
Permanent fix and Action Items
To prevent such issues from happening again, Harness is