k8s-safe-node-reboot

Safely drain, reboot, and uncordon Kubernetes worker nodes, one node at a time.

Problem

Rolling reboots of worker nodes are easy to get wrong: pods left running, SSH targeting the wrong hostname, reboots fired on the wrong cluster, or nodes left cordoned after a failed wait. This script sequences the operations and aborts on failure.

Requirements

• kubectl configured for the target cluster
• SSH access to nodes as the current user (sudo reboot must work)
• Bash 4+

Usage

chmod +x reboot-nodes.sh

# Dry run against one node
./reboot-nodes.sh --dry-run worker-01

# Explicit suffix for short Kubernetes node names
./reboot-nodes.sh --fqdn-suffix .prod.example.com worker-01 worker-02

# Suffix from context mapping file
./reboot-nodes.sh --fqdn-suffix-file ./fqdn-suffix.conf k8s-worker

# Prefix match (all nodes whose name starts with k8s-worker)
./reboot-nodes.sh --fqdn-suffix .prod.example.com k8s-worker

# All nodes in the cluster
./reboot-nodes.sh --fqdn-suffix .prod.example.com all

FQDN resolution

Kubernetes node names are often short (worker-01). SSH usually needs a FQDN.

Resolution order for the SSH target:

1. If the node name already contains ., use it as-is.
2. Else append --fqdn-suffix, or FQDN_SUFFIX env, or a match from --fqdn-suffix-file.
3. If no suffix is set, SSH uses the short node name.

Suffix mapping file

Copy fqdn-suffix.example.conf and edit for your environments:

production*|.prod.example.com
development*|.dev.example.com

Patterns are bash case globs matched against kubectl config current-context.

Workflow per node

1. Confirm kubectl context (interactive unless --dry-run)
2. Drain (with retries) if the node is schedulable
3. If cordoned and Ready: check remote uptime via SSH; reboot if uptime exceeds 30 minutes
4. Wait for bootID change (preferred) or Ready status
5. Uncordon
6. Pause 15 seconds before the next node

Options

Flag	Description
--dry-run	Print planned actions; no drain, SSH, or uncordon
--fqdn-suffix SUFFIX	Append suffix to short node names for SSH
--fqdn-suffix-file FILE	Context glob → suffix mappings
--help	Show usage

Limits

• Processes nodes sequentially, not in parallel
• Assumes kubectl drain with --ignore-daemonsets --delete-emptydir-data is acceptable
• Uptime skip threshold is fixed at 30 minutes (edit UPTIME_THRESHOLD_SEC in the script)
• No built-in SSH options (keys, StrictHostKeyChecking, jump hosts) — configure ~/.ssh/config
• Aborts the entire run if any node fails drain, ready wait, or uncordon

License

MIT — see LICENSE.