Kubernetes Pod SIGKILLed Before Graceful Shutdown in CI

By Daniel Zoghalchali·Latchkey

On termination Kubernetes sends SIGTERM, waits up to terminationGracePeriodSeconds, then sends SIGKILL. If the grace period is shorter than the app needs to drain connections or flush state, the container is force-killed mid-shutdown.

What this error means

During a rollout or scale-down, pods exit with in-flight requests dropped or partial writes, and events show the container was killed after the grace period. Lengthening the period makes clean shutdown work, proving the cut-off was the cause.

kubectl events

Killing container with a grace period of 30 seconds
Stopping container app
# app's drain takes ~45s; at 30s it receives SIGKILL mid-drain → dropped connections

Diagnose it: read events, not just status

A deployment that never becomes ready has the reason in its events and in the pod state, not in the deployment status. Read both before changing the manifest.

Terminal

kubectl rollout status deploy/<name> --timeout=120s
kubectl describe deploy/<name> | sed -n "/Events/,$p"
kubectl get pods -l app=<name> -o wide
kubectl describe pod <pod> | sed -n "/Events/,$p"
kubectl logs <pod> --previous --tail=50   # the crash before the restart

Common causes

Grace period shorter than the drain time

The default 30s terminationGracePeriodSeconds is shorter than the time the app needs to finish in-flight requests or flush buffers, so SIGKILL arrives mid-shutdown.

App ignores SIGTERM

If the process does not handle SIGTERM (e.g. it is not PID 1, or the signal is swallowed by a shell), it never starts draining and is killed at the end of the grace period regardless of length.

How to fix it

Set a grace period that covers real shutdown

Raise terminationGracePeriodSeconds to the worst-case drain time, optionally with a preStop hook to begin draining.

deployment.yaml

spec:
  terminationGracePeriodSeconds: 60   # was 30; covers connection drain
  containers:
    - name: app
      lifecycle:
        preStop:
          exec: { command: ["sh","-c","sleep 5"] }

Make the app handle SIGTERM as PID 1

Run the process as PID 1 (use exec form in entrypoints, or an init like tini).
Trap SIGTERM and start graceful shutdown immediately.
Stop accepting new work, finish in-flight work, then exit before the grace period ends.

How to prevent it

Size terminationGracePeriodSeconds to measured worst-case drain time.
Ensure the app receives and handles SIGTERM as PID 1.
Use a preStop hook to begin draining before SIGTERM where needed.

Frequently asked questions

What causes Kubernetes pod SIGKILLed before graceful shutdown in CI?

There are 2 common causes: grace period shorter than the drain time and app ignores sigterm. The default 30s terminationGracePeriodSeconds is shorter than the time the app needs to finish in-flight requests or flush buffers, so SIGKILL arrives mid-shutdown.

How do I fix Kubernetes pod SIGKILLed before graceful shutdown in CI?

There are 2 fixes depending on which cause you have: set a grace period that covers real shutdown and make the app handle sigterm as pid 1. Work through them in order, since the first is the most common.

What does Kubernetes pod SIGKILLed before graceful shutdown in CI actually mean?

During a rollout or scale-down, pods exit with in-flight requests dropped or partial writes, and events show the container was killed after the grace period.

How do I stop Kubernetes pod SIGKILLed before graceful shutdown in CI happening again?

Size terminationGracePeriodSeconds to measured worst-case drain time. The prevention section lists 3 changes that keep it from recurring.