DaemonSets
A DaemonSet runs exactly one pod per node — the right model for logging agents, monitoring exporters, networking plugins, and storage drivers. Unlike Deployments, scaling means “match node count,” not “match replicas.”
Manifest, tolerations, rolling update
EXAMPLE
# 1) When to use a DaemonSet
# • Cluster-wide infrastructure — log shippers (Fluent Bit), node exporters, CNI plugins
# • Storage drivers (CSI nodes), service mesh sidecars, security agents (Falco)
# • Anything that NEEDS to run on every node OR a specific subset (tagged GPU nodes)
# 2) Minimum manifest — Fluent Bit log shipper
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: kube-system
labels: { app: fluent-bit }
spec:
selector:
matchLabels: { app: fluent-bit }
template:
metadata:
labels: { app: fluent-bit }
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.0
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 200m, memory: 256Mi }
volumeMounts:
- name: varlog
mountPath: /var/log
readOnly: true
- name: dockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config
mountPath: /fluent-bit/etc/
tolerations:
- operator: Exists # run on every node, including tainted control plane
volumes:
- name: varlog
hostPath: { path: /var/log }
- name: dockercontainers
hostPath: { path: /var/lib/docker/containers }
- name: config
configMap: { name: fluent-bit-config }
# 3) Inspect
kubectl -n kube-system get daemonset,pod -l app=fluent-bit
# fluent-bit DESIRED 5 CURRENT 5 READY 5 UP-TO-DATE 5 AVAILABLE 5 NODE SELECTOR <none>
# 4) Tolerations — run on EVERY node, even tainted
# Control plane nodes typically have a taint:
# node-role.kubernetes.io/control-plane:NoSchedule
# Add a tolerate-all rule to ensure the agent runs everywhere:
tolerations:
- operator: Exists # tolerate any taint, any effect
# Or be specific:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
operator: Exists
# 5) Run ONLY on certain nodes — nodeSelector / affinity
spec:
template:
spec:
nodeSelector:
node.kubernetes.io/instance-type: g5.xlarge # GPU nodes only
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload.example.com/gpu
operator: In
values: ['nvidia', 'amd']
# 6) Rolling update — default is OnDelete BEFORE 1.6; RollingUpdate after
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # at most 1 pod down at a time
maxSurge: 0 # DaemonSets default to 0 surge (pin to one-per-node)
# OnDelete strategy — manual control
spec:
updateStrategy:
type: OnDelete
# You delete pods one by one; the controller creates new ones with the updated template.
# Useful for highly sensitive workloads (CNI plugins) where you choreograph upgrades.
# 7) Priority + preemption
# DaemonSets typically run system-critical workloads — give them a high PriorityClass:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: system-node-critical }
value: 2000000000
globalDefault: false
description: "Cluster-essential daemons"
---
spec:
template:
spec:
priorityClassName: system-node-critical
# Now under cluster pressure, your daemons aren't the first to be evicted.
# 8) Privileged DaemonSets — common but dangerous
# Many node-level daemons need hostPath, hostNetwork, hostPID, or privileged:true.
# Lock these down with Pod Security Admission, OPA Gatekeeper, or Kyverno.
spec:
template:
spec:
hostNetwork: true # share host's network namespace
hostPID: true # see host's processes
containers:
- name: node-exporter
securityContext:
privileged: true # ⚠ full root on the node
readOnlyRootFilesystem: true
runAsUser: 0
capabilities:
drop: ["ALL"]
add: ["SYS_TIME"]
# Use the minimum capabilities needed; full 'privileged: true' is a kernel-level grant.
# 9) Common DaemonSet workloads
# Logging: Fluent Bit, Fluentd, Vector, Promtail
# Metrics: Node Exporter, cAdvisor, Datadog Agent, New Relic Infra
# CNI: Calico, Cilium, Flannel — runs on every node
# Storage: OpenEBS NDM, Rook, CSI node plugins
# Security: Falco, Sysdig Inspect, Tetragon
# Service mesh: Istio's ztunnel (ambient mode), Linkerd2-proxy
# 10) DaemonSet vs DeploymentSet vs StatefulSet vs Job
# DaemonSet — one pod per node (or per matching node)
# Deployment — fixed # of pods (replicas), scheduler decides where
# StatefulSet — ordered, named, per-pod storage
# Job/CronJob — run-to-completion (one-off or scheduled)
# 11) Resource limits matter
# Set 'requests' on every container — without them, the daemon competes with workloads under pressure.
# Set 'limits' to prevent runaway agents from eating the node.
# 12) Drain + cordon respect DaemonSets
# 'kubectl drain' EXCLUDES DaemonSet pods by default unless you pass --ignore-daemonsets.
# DaemonSet pods aren't part of the workload; they belong to the node, conceptually.
# 13) Common bugs
# • Missing tolerations → daemon doesn't run on tainted nodes (especially control-plane and GPU nodes)
# • Wrong nodeSelector → daemon runs nowhere; check label exists on at least one node
# • Privileged container without securityContext → pod refused by Pod Security Admission
# • HostPath without readOnly → daemon can modify node files; set readOnly: true unless writing is intentional
# • Update rollout stuck on bad image → set maxUnavailable: 1, prerequisite probes
# • Forgot priorityClass → daemon evicted under pressure during exactly the wrong moment
# • Daemon log volume eats node disk — limit/rotate logs; use hostPath to a separate disk if needed
# • CrashLoopBackOff on initial deploy — check resources.requests; the daemon may be OOMKilled before init
Why it matters
Reach for a DaemonSet when you need exactly one pod per node — log agents, node exporters, CNI plugins. Always set tolerations for tainted control-plane and GPU nodes, give system-critical daemons a high priorityClass, and lock down privileged/hostPath permissions via Pod Security Admission so a compromised agent can’t take the whole node.
Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.
Discussion
Loading…