The One Feature Kubernetes Needs for Fleet-Scale Stateful Workloads

Kubernetes 1.37 can scale a deployment to zero. It still can't tell a pod the one thing a stateful workload needs to know at shutdown: am I coming back?

Kubernetes keeps getting better at scaling down. The latest release, 1.37, graduated HPA scale-to-zero to beta, on by default. The pod on the receiving end still can't tell that event apart from a routine restart. SIGTERM is SIGTERM: a node OS patch, a Kubernetes upgrade, a new image, and a permanent scale-down all look the same from inside the container. For stateless workloads that's fine. For stateful workloads, it's the most expensive missing bit in the API.

I opened kubernetes/kubernetes#131823 to ask for that bit: let a pod differentiate a scale-down from a shutdown.

Why it matters

At termination, a stateful system has one decision: what happens to the state? If the pod is coming back, nothing: flush, exit, the PVC is still there. If the replica is going away, the data has to be offloaded to a peer, a shared operator node, or a backup volume before exit.

Restart vs scale-down: the same SIGTERM, very different obligations Left: on a restart, broker-0 receives SIGTERM, shuts down cleanly, its volume pvc-0 persists, and the pod returns in seconds with the same PVC, no drain, ordering preserved. Right: on a scale-down from three replicas to two, broker-2 receives SIGTERM, its messages are pushed from pvc-2 to the live brokers, the replica is removed and pvc-2 is deleted. Both cases start with the same SIGTERM, so the pod cannot tell which one it is in. Restart: node patch, upgrade, new image broker-0 pvc-0 messages SIGTERM clean shutdown: state stays on the volume broker-0 pvc-0 persists pod returns broker-0 pvc-0 back in seconds, same PVC: no drain, ordering preserved Scale-down (replicas: 3 → 2) SIGTERM (scale-down) broker-0 broker-1 broker-2 pvc-0 pvc-1 pvc-2 going away messages pushed to the live brokers drain competes with production traffic; ordering at risk replica removed broker-0 broker-1 broker-2 pvc-0 pvc-1 pvc-2 deleted two brokers remain; the volume is gone for good Today both columns start with the same SIGTERM: the pod cannot tell which one it is in. kubernetes/kubernetes#131823 proposes the missing signal
Restart vs scale-down: the same SIGTERM, very different obligations

The offload is the expensive part. Message queues offload by draining: pumping every queued message to the surviving brokers, where drain traffic competes with production traffic and usually destroys message ordering. Databases offload by export: a consistent dump that has to be shipped to archive storage.

Because the pod can't ask "am I coming back?", a careful operator treats every termination as permanent and pays the offload cost every time. The alternative risks losing data on the one termination that wasn't a restart.

The maintenance-window math

A typical maintenance window stacks three layers of churn: node OS patching, the Kubernetes version upgrade, and the messaging platform's own image roll. Each layer terminates every broker pod at least once, and each termination triggers a full precautionary drain. One window can drain every broker cluster two or three times, scrambling ordering each time, to guard against a removal that isn't happening.

Each bounce also shuffles client connections onto the surviving brokers, and spreading them back evenly takes smart rebalancing. That part is unavoidable; clients have to go somewhere during a restart. The message churn is not. If the platform knew "this is a restart, your volume survives," the drains simply wouldn't happen, and the window would shrink to rolling restarts plus rebalancing.

The ask

The issue asks for a small, optional signal (a dedicated lifecycle hook, or a flag on the existing PreStop hook) that tells the pod which of three situations it's in:

Termination eventWhat the pod should do
Restart (same identity, coming back)Nothing: clean shutdown
Change (upgrade, reschedule)Backup / migrate if policy requires
Removal (scale-down, decommission)Full offload: drain or export
The optional signal kubernetes/kubernetes#131823 asks for · one bit of termination context, three different costs

No scheduler changes, no new controllers: just termination context, passed to the workload that has to act on it. The issue has been open since April 2025 and sits on the SIG Apps board; the proposed Eviction Request API has come up as a way to carry the intent, with maintenance-window signaling tracked in kubernetes/enhancements#5683. None of it has moved forward yet.

The gap

Scale-to-zero makes the gap easy to see. At the moment the HPA decides to remove a replica, Kubernetes knows this termination is a scale-down.

It just doesn't tell the pod.

The information is dropped one hop before the component that could save hours of maintenance window with it. Stateful workloads pay for that missing hop in drained queues, scrambled ordering, export jobs, and windows sized for the worst case. They don't need another scaling feature; they need the pod to be told the one bit the control plane already holds: restart, or goodbye.

If you run message queues or databases on Kubernetes and this cost looks familiar, add your drain numbers to kubernetes/kubernetes#131823. Real numbers are what move a SIG.

Further reading: kubernetes/kubernetes#131823: differentiate scale-down from shutdown · Kubernetes v1.36 release notes (scale-to-zero alpha) · HPA scale-to-zero in 1.37 · kubernetes/enhancements#5683: node maintenance signals


Originally published on LinkedIn on October 29, 2026.

Cover photo by Haris Illahi on Unsplash.

Matt Pavlovich

Matt Pavlovich, the Chief Technology Officer and Technical Practice Lead at HYTE Technologies, directs the HYTE Product Development Team. Matt serves on the Apache ActiveMQ Project Management Committee (PMC) and commits code upstream to the broker. He led the JMS 3.1 and 2.0 feature work and the Jakarta conversion that shipped in ActiveMQ v6, and he started the effort to bring virtual thread support to the broker. Matt has led large-scale ActiveMQ implementations across defense, financial services, insurance, retail, and distribution. Under his direction, HYTE's tools and services speed enterprise application development and make middleware easier to support.

Next
Next

How ActiveMQ v6.3.0 boosts topic throughput up to 93%