Service Mesh (Istio / Linkerd)

# 1) What a service mesh adds
# • Mutual TLS between services (zero-trust)
# • Retries, timeouts, circuit breakers — app-agnostic
# • Traffic splitting (canary, blue/green, A/B)
# • Distributed tracing + metrics + access logs
# • Authorization policies between services
# • Multi-cluster, multi-region routing

# The data plane = sidecar proxy (Envoy or linkerd2-proxy) injected into every pod.
# The control plane = manages config, certs, telemetry.

# 2) Picking a mesh
# Istio   — feature-rich, heavy, biggest learning curve
# Linkerd — minimal, very fast (Rust), security-focused, simpler
# Cilium  — eBPF-based; no sidecar; high performance; uses kernel networking
# Consul — HashiCorp; good with multi-DC
# AWS App Mesh — managed, AWS-only, deprecating in 2026 — avoid for new projects
# Open Service Mesh (OSM) — deprecated

# Most teams: Linkerd if simplicity wins, Istio if features needed.

# 3) Install Linkerd (simplest start)
linkerd install | kubectl apply -f -
linkerd check                                              # verifies install
linkerd viz install | kubectl apply -f -                   # web UI
linkerd viz dashboard

# 4) Inject the proxy into your namespace
kubectl annotate namespace prod linkerd.io/inject=enabled
kubectl rollout restart deploy -n prod                     # pods restart with sidecar

# Every pod now has 2 containers: app + linkerd-proxy.

# 5) mTLS — automatic
# Once both sender and receiver are meshed, traffic is mTLS-encrypted with rotating certs.
# Zero app changes; opt-in per namespace via injection annotation.

# Verify:
linkerd viz edges -n prod
# Shows TLS yes/no per src→dst pair

# 6) Traffic shift — canary release (Linkerd)
# Two deployments: app-v1 + app-v2
apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata: { name: app, namespace: prod }
spec:
    service: app
    backends:
        - { service: app-v1, weight: 90 }
        - { service: app-v2, weight: 10 }

# 90% to v1, 10% to v2. Adjust weight over time; watch error rate; flip when stable.

# 7) Istio basics — VirtualService + DestinationRule
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: app }
spec:
    hosts: [app]
    http:
        - match: [{ headers: { 'x-canary': { exact: 'true' } } }]
            route: [{ destination: { host: app, subset: v2 } }]
        - route:
            - { destination: { host: app, subset: v1 }, weight: 90 }
            - { destination: { host: app, subset: v2 }, weight: 10 }
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata: { name: app }
spec:
    host: app
    subsets:
        - { name: v1, labels: { version: v1 } }
        - { name: v2, labels: { version: v2 } }

# 8) Observability — out of the box
# • Linkerd viz: golden metrics (success rate, latency, RPS) per route
# • Istio + Kiali / Jaeger: full service graph + traces
# • All without modifying app code

# Add tracing headers (x-request-id, traceparent) at edge; sidecars propagate.

# 9) Authorization policies
# Linkerd:
apiVersion: policy.linkerd.io/v1beta1
kind: AuthorizationPolicy
metadata: { name: orders-from-web, namespace: prod }
spec:
    targetRef: { kind: Service, name: orders }
    requiredAuthenticationRefs:
        - { kind: ServiceAccount, name: web, namespace: prod }

# Only the 'web' service account can call 'orders'. Anywhere else → 403.

# 10) Circuit breakers + retries
# Istio:
spec:
    http:
        - retries:
                attempts: 3
                perTryTimeout: 2s
                retryOn: '5xx,reset,connect-failure,refused-stream'
        - fault:
                delay: { percent: 0.1, fixedDelay: 5s }   # chaos engineering: inject 5s delay for 0.1%

# 11) Multi-cluster
# Istio multi-primary / Linkerd multi-cluster:
# • Trust roots shared via shared CA
# • Cross-cluster services exposed via gateways
# • DNS for cross-cluster discovery
# Powerful but complex; only do when you actually need it.

# 12) Cost — sidecars aren't free
# • Each pod: 50-100 MB extra RAM + 0.1-0.2 CPU for proxy
# • Cilium with eBPF: no sidecar; kernel-level; lower overhead at scale
# • Plan for it: bump cluster resource requests by 10-20% post-mesh

# 13) When you might NOT need a mesh
# • Few services (< 10) — direct K8s Services + TLS at edge fine
# • No cross-team encryption mandate
# • Don't need traffic splitting (use simple deployment strategies)
# • Limited ops capacity — mesh adds operational complexity
#
# Service meshes solve real problems at moderate-to-large scale.

# 14) Operational concerns
# • Mesh upgrade = critical-path change; test in staging first
# • Sidecar injection failure → pod won't start; check ENV + admission webhook
# • Cert rotation — automatic in Linkerd/Istio; monitor for failures
# • Mesh-to-non-mesh traffic — works but mTLS not enforced; mark explicitly
# • Bypassing the proxy (debug) — kubectl exec port-forward → direct app port

# 15) Common bugs
# • Injecting sidecar on jobs/cronjobs without 'wait for proxy' → app starts before proxy; retries fail
# • Init containers needing network before sidecar ready — use linkerd-await or Istio's holdApplicationUntilProxyStarts
# • mTLS but workload uses HTTPS already → double TLS; can be OK but unnecessary overhead
# • Mesh-injected pod can't reach external API — add ServiceEntry (Istio) or egress allowlist
# • Linkerd viz shows nothing — proxy not injected; check annotation + restart
# • TrafficSplit ignored — service ports must match across backends
# • RetryBudget too aggressive → retry storms amplify outages
# • Circuit breaker thresholds misconfigured → false positives; tune in staging first
# • Upgrading mesh control plane without testing → can break sidecars on all pods
# • Multi-cluster trust mis-bootstrap → all cross-cluster calls fail; use cert-manager properly

# Istio / Linkerd add mTLS, traffic shaping, observability.
# Worth it once you have 10+ services.