Use kubectl debug and ephemeral containers for live debugging

Introduction

When your LLM chat API pod is healthy in kubectl get pods but returning 502s to users, and you try kubectl exec only to discover the container is distroless with no shell, the usual debugging toolkit runs out fast. Hardened production images strip /bin/sh, run with readOnlyRootFilesystem: true, and drop all capabilities — exactly the postures that make exec impossible while the failure is still live. Skip live debugging and you fall back to redeploying with extra logging, which loses the in-memory state that caused the bug. By the end of this lesson you'll be able to inject an ephemeral debug container into a running pod, share its process and network namespace, and clone failing pods with overridden commands — all without touching the original workload.

Key Terminology

• Ephemeral container — a temporary container injected into a running pod by kubectl debug; it cannot be added at pod-creation time and never restarts, which is why it's safe to attach to production without changing the pod spec.
• Process namespace sharing — the --target=<container> flag wires the debug container into the target's PID namespace so ps, /proc/<pid>/environ, and /proc/<pid>/fd reveal the target's runtime state without any tooling installed in the target.
• Distroless image — a minimal base image (Google's gcr.io/distroless/*) containing only the app binary and its runtime; no shell, no package manager, which forces ephemeral-container debugging.
• --copy-to — a kubectl debug flag that creates a modifiable clone of a failing pod (override command, image, or probes) while leaving the original untouched for further inspection.
• Node debug pod — a privileged pod placed on a specific node via kubectl debug node/<name> with the host filesystem mounted at /host, used when the failure is node-scoped (DNS, kubelet, iptables) rather than pod-scoped.

Concepts

Why kubectl exec is not enough

Production containers ship without a shell on purpose. A distroless chat-api image has no /bin/sh, so kubectl exec -it ... -- /bin/sh returns exec format error or executable not found. Even with a shell, readOnlyRootFilesystem: true blocks installing tools. Ephemeral containers sidestep both: they run alongside the target in the same pod, with their own writable filesystem and whatever debug tools you pick.

Ephemeral containers and namespace sharing

kubectl debug -it <pod> --image=<debug-image> --target=<container> injects a new container into the running pod. With --target, the debug container shares the target's process namespace, so ps aux lists the target's processes and /proc/1/environ exposes its env vars. All containers in a pod already share the network namespace, so localhost reaches sidecar proxies (e.g. gemini-proxy on :8081) and DNS lookups resolve in the pod's cluster context. Pick busybox for lightweight checks; pick nicolaka/netshoot when you need dig, tcpdump, traceroute, or HTTP timing breakdowns (see Code Walkthrough).

Debug copies and node debugging

When the bug needs a config change to reproduce — a probe that keeps killing the container, a missing env var, a broken entrypoint — kubectl debug --copy-to=<new-name> clones the pod with overrides applied, leaving the original in place. When the symptom isn't pod-scoped (a single node's DNS broken, kubelet failing pulls), kubectl debug node/<name> schedules a privileged pod on that node with /host mounted, exposing kubelet logs, iptables rules, and disk usage.

Code Walkthrough

The snippets below demonstrate the three escalation steps from Concepts: attach an ephemeral container with process-namespace sharing for in-pod inspection, then clone the pod with --copy-to when you need to change its command.

Code snippetbash
1# 1. Attach netshoot to a running pod, sharing the chat-api PID namespace
2kubectl debug -it llm-chat-api-7d9f8b6c5-x2k4p \
3  -n llm-chat \
4  --image=nicolaka/netshoot \
5  --target=chat-api \
6  -- bash
7
8# Inside the debug container — pod-network + chat-api PID namespace:
9ps aux                                            # see chat-api's processes
10cat /proc/1/environ | tr '\0' '\n'                # chat-api env vars
11dig postgres-svc.llm-chat.svc.cluster.local       # DNS from inside the pod
12curl -w "DNS:%{time_namelookup}s total:%{time_total}s\n" \
13  http://localhost:8081/health                    # hit the sidecar via loopback
14nc -zv postgres-svc 5432                          # check NetworkPolicy + DB reachability

• --target=chat-api shares the PID namespace so /proc/1 is chat-api's main process — env vars and file descriptors are visible without any tools installed in chat-api itself.
• localhost:8081 reaches the sidecar proxy because every container in a pod shares the network namespace.
• dig and nc isolate whether the failure is DNS, NetworkPolicy, or the backend service itself.

Code snippetbash
1# 2. Clone the failing pod with the entrypoint replaced by sleep,
2#    so the container stays up for manual inspection.
3kubectl debug llm-chat-api-7d9f8b6c5-x2k4p \
4  -n llm-chat \
5  --copy-to=llm-chat-debug \
6  --container=chat-api \
7  --image=llm-chat-api:v1.0 \
8  -- sleep 3600
9
10kubectl exec -it llm-chat-debug -n llm-chat -c chat-api -- /bin/sh
11# env | sort; ls -la /app/; python -c "import app; print(app.__file__)"

• --copy-to creates llm-chat-debug as a sibling pod; the original keeps serving traffic (or keeps crashing for further forensics).
• Replacing the entrypoint with sleep 3600 defeats a crash-on-startup loop so you can inspect the filesystem, config, and Python import path interactively.
• When the symptom is node-scoped instead, swap the whole command for kubectl debug node/<node-name> -it --image=nicolaka/netshoot -- bash — /host is mounted with the node's resolv.conf, kubelet logs, and iptables rules.

You'll know it works when ps aux in the ephemeral container lists the target's PID 1, curl http://localhost:<sidecar-port>/health returns 200, and the --copy-to clone shows Running in kubectl get pods without restarting. Done when you've reproduced the failure inside the debug pod or ruled out pod-level causes.

Do's and Don'ts

Do's

1. ✓Do prefer kubectl debug --target=<container> — process-namespace sharing is the only way to inspect distroless or read-only containers without redeploying them.
2. ✓Do use --copy-to for config experiments — overriding command, image, or probes on a clone keeps the failing pod intact for repeat reproductions and post-mortem evidence.
3. ✓Do match the debug image to the question — busybox for quick filesystem and /proc checks, nicolaka/netshoot for DNS, packet capture, and HTTP timing.

Don'ts

1. ✗Don't kubectl exec into the live pod first — on distroless or hardened images it just fails with a confusing error; start with kubectl debug.
2. ✗Don't skip --target when you need the app's process state — without it, ps and /proc/1 show only the debug container itself, not the workload you're debugging.
3. ✗Don't reach for kubectl debug node/<name> before pod-level steps — node debug pods are privileged and noisy; escalate only after logs, describe, and ephemeral containers have ruled out pod-scoped causes.