Troubleshooting

# ===== The systematic loop =====
# 1. What is the state? (kubectl get)
# 2. Why is it that state? (kubectl describe + events)
# 3. What is the workload doing? (kubectl logs)
# 4. What is inside? (kubectl exec)
# 5. What changed recently? (rollout history)

# ===== State =====
kubectl get pods -A
kubectl get pods -l app=shop -o wide
kubectl get pods,svc,ingress,deploy -n myns

# Common state colours:
# Running        normal
# Pending        not yet scheduled (resource / image issues)
# CrashLoopBackOff   container exits repeatedly
# ImagePullBackOff   wrong image / no auth
# CreateContainerConfigError   bad config / secret missing
# Completed      pod finished (Job)

# ===== Why? =====
kubectl describe pod <name>
# Look at:
# - Events (bottom of output)
# - Status reason
# - Container restart count + last termination reason
# - Pod conditions (PodScheduled, ContainersReady, Ready)

kubectl get events --sort-by=.metadata.creationTimestamp -A | tail -20

# ===== Logs =====
kubectl logs <pod>
kubectl logs <pod> -c <container>           # multi-container pods
kubectl logs <pod> --previous                # previous container instance
kubectl logs -l app=shop --tail=200          # by label
kubectl logs -f <pod>                        # follow

# Multi-pod:
stern -l app=shop

# ===== Inside =====
kubectl exec -it <pod> -- sh
kubectl exec <pod> -- env
kubectl exec <pod> -- ls /app

# Debug containers (K8s 1.25+):
kubectl debug -it <pod> --image=busybox --target=<container>

# ===== Recent changes =====
kubectl rollout history deploy/shop
kubectl rollout history deploy/shop --revision=3
kubectl rollout undo deploy/shop

# ===== Resources + limits =====
kubectl top pod -n myns
kubectl top node

kubectl describe node <node>      # capacity vs allocated

# ===== Networking =====
# DNS:
kubectl run -it --rm debug --image=busybox -- nslookup my-svc

# Service endpoints:
kubectl get endpoints my-svc

# Test connectivity:
kubectl run -it --rm curl --image=curlimages/curl -- sh
> curl http://my-svc.my-ns/healthz

# ===== Common bugs + fixes =====
# 1. ImagePullBackOff
#    - typo in image
#    - private registry without imagePullSecrets
#    - region mismatch

# 2. CrashLoopBackOff
#    - missing env var or config file
#    - bad command in entrypoint
#    - resource limits too low (OOMKilled)

# 3. Pending
#    - insufficient node resources
#    - PVC not bound
#    - taints/affinity not matched

# 4. 503 from Service
#    - no pods matching selector
#    - readinessProbe failing -> not in endpoints
#    - kube-proxy / CNI issue

# 5. Slow / timeout
#    - resource throttling (top + describe)
#    - upstream DB slow
#    - probe timeouts too low

# ===== Patterns =====
# - Always check events FIRST
# - logs --previous after crashes
# - kubectl debug for distroless containers
# - kubectl top for resource pressure

# ===== Pitfalls =====
# - kubectl exec into a container that doesn't have a shell (distroless) -> use debug
# - Reading wrong namespace (check current context!)
# - Stale image cached on node (pullPolicy: IfNotPresent on a moving tag)
# - Healthchecks failing because they hit external services

kubectl describe pod web
kubectl logs web --previous
kubectl exec -it web -- sh