Kubernetes Administrator — Interview Questions.

kube-apiserver — the front door; every read/write goes through it (auth, validation, admission). etcd — the consistent key-value store holding all cluster state (the source of truth). kube-scheduler — assigns unscheduled Pods to nodes based on resources, affinity, taints. kube-controller-manager — runs the reconciliation loops (Deployment, ReplicaSet, Node, etc.) driving actual state toward desired. cloud-controller-manager — integrates cloud provider (LBs, volumes, node lifecycle). On each worker node: kubelet (starts/monitors Pods via the container runtime) and kube-proxy (Service networking rules).

Pod: smallest deployable unit — one or more containers sharing network/storage. You rarely create them directly. ReplicaSet: keeps N identical Pod replicas running. Deployment: manages ReplicaSets to give you declarative rolling updates and rollback for stateless apps (the everyday workload object). StatefulSet: for stateful apps — stable network identity (ordinal names), stable per-Pod persistent storage, and ordered rollout/scaling (Pod-0 before Pod-1). Use Deployment for stateless web apps; StatefulSet for databases, Kafka, etc.

A Service is a stable virtual IP + DNS name fronting a set of Pods (selected by labels), load-balancing across their endpoints — so clients don't care that Pods come and go. Types: ClusterIP (internal-only, default), NodePort (exposes a port on every node), LoadBalancer (provisions a cloud LB), and ExternalName (CNAME to an external host). For HTTP routing across many services you typically layer an Ingress (or Gateway API) on top of ClusterIP services rather than many LoadBalancers.

Both inject config into Pods (as env vars or mounted files), decoupling config from images. ConfigMap for non-sensitive config; Secret for sensitive data — but note a Secret is only base64-encoded, not encrypted, by default. So enable encryption at rest for Secrets in etcd, lock down RBAC on who can read them, and prefer an external store (Vault / External Secrets Operator / CSI driver) for real secret management. Mounted ConfigMaps/Secrets update in the Pod over time; env vars are fixed at start.

Requests are what the scheduler reserves — they determine which node a Pod fits on and guarantee that much CPU/memory. Limits are the hard ceiling — exceeding the memory limit gets the container OOMKilled; exceeding CPU just throttles it (CPU is compressible, memory isn't). The request/limit relationship sets the Pod's QoS class: Guaranteed (requests==limits), Burstable (requests<limits), BestEffort (none) — which decides eviction order under node pressure. Setting them right is core admin hygiene.

A virtual cluster partition scoping names + RBAC + quotas — used for multi-tenancy and environment separation.

Runs one Pod copy per node (or selected nodes) — for node-level agents like log shippers, CNI, monitoring.

Deployment for stateless apps (interchangeable Pods, rolling updates); StatefulSet for stateful (stable identity + per-Pod storage, ordered rollout).

Non-sensitive config injected as env vars or mounted files, decoupling config from the image.

HTTP(S) routing into the cluster (host/path rules, TLS) via an ingress controller, layered over ClusterIP Services.

The CLI that talks to the API server to create/inspect/manage cluster resources.

Key/value tags on objects; selectors match them so Services/Deployments/policies target the right Pods.

A Job runs a Pod to completion (batch); a CronJob runs Jobs on a schedule.

The node agent that starts/monitors Pods via the container runtime and reports status to the API server.

Keeps a specified number of identical Pod replicas running; usually managed by a Deployment.

Pending = not yet scheduled to a node. kubectl describe pod and read the Events — the scheduler tells you exactly why. Common causes: insufficient resources (no node has enough CPU/mem for the requests — scale up / cluster-autoscaler / lower requests), taints with no matching toleration, node affinity / nodeSelector that nothing satisfies, unbound PVC (no provisioner or no matching PV — the Pod waits for storage), or Pod anti-affinity / topology spread that can't be satisfied. Check kubectl get nodes for capacity and kubectl describe node for allocatable vs allocated.

K8s networking model: every Pod gets its own IP and any Pod can reach any other Pod without NAT — implemented by a CNI plugin (Calico, Cilium, etc.). Services give stable VIPs; kube-proxy (iptables/IPVS) or eBPF programs the routing to backend endpoints. Cluster DNS (CoreDNS) resolves svc.namespace.svc.cluster.local. By default all traffic is allowed. A NetworkPolicy is a label-selected firewall: once a policy selects a Pod, it switches to default-deny for the specified direction and only the listed ingress/egress is permitted. Requires a CNI that enforces them (Calico/Cilium; flannel doesn't).

kubectl cordon node first (marks it unschedulable so nothing new lands), then kubectl drain node --ignore-daemonsets --delete-emptydir-data to evict the Pods gracefully. Eviction respects PodDisruptionBudgets — a PDB (e.g. minAvailable: 2) makes drain block rather than violate availability, which is exactly the safety net you want. DaemonSet Pods are skipped. Pods get their terminationGracePeriod to shut down. After maintenance, kubectl uncordon node. For stateful workloads, make sure the PDB and replica count actually tolerate losing one Pod.

Readiness: should this Pod get traffic? Fail → removed from Service endpoints but not restarted (use during warmup or when a dependency is down). Liveness: is the container healthy? Fail → kubelet restarts it (recovers deadlocks/hangs). Startup: protects slow-booting apps — liveness/readiness are held off until startup succeeds, so a slow start isn't mistaken for a hang. Classic admin failure: an aggressive liveness probe restart-looping a healthy-but-slow app — fix with a startup probe and sane thresholds. Probes should be cheap and test the app's real health, not just "process alive."

RBAC grants permissions via four objects. Role (namespaced) / ClusterRole (cluster-wide) define what — verbs (get/list/create/delete) on resources. RoleBinding / ClusterRoleBinding attach a Role to a subject — a User, Group, or ServiceAccount. Pods authenticate as their ServiceAccount. Principle: least privilege — scope to namespaces, avoid cluster-admin and wildcard verbs, don't mount the default SA token if a Pod doesn't call the API. kubectl auth can-i checks effective permissions. RBAC is additive (no deny rules) — there's no way to subtract a granted permission.

A PersistentVolume (PV) is a piece of storage in the cluster; a PersistentVolumeClaim (PVC) is a Pod's request for storage (size + access mode). A StorageClass enables dynamic provisioning: when a PVC references it, the CSI driver creates the underlying disk and a matching PV automatically (no pre-creating PVs). Access modes: RWO (one node), ROX (read-only many), RWX (read-write many — needs a shared filesystem like NFS/EFS). reclaimPolicy (Delete vs Retain) decides if the disk is destroyed when the PVC is deleted — Retain for anything you can't afford to lose. volumeBindingMode: WaitForFirstConsumer avoids provisioning a volume in the wrong AZ.

Programs iptables/IPVS (or eBPF) rules on each node to load-balance Service VIP traffic to ready endpoint Pods.

Limits how many Pods of an app can be voluntarily disrupted at once (drains respect it), protecting availability during maintenance.

Requests reserve capacity (scheduling); limits cap usage (memory over-limit = OOMKilled, CPU throttles). Their relationship sets QoS (Guaranteed/Burstable/BestEffort), which drives eviction order.

Scales Deployment replicas based on metrics (CPU/mem/custom) toward a target — the Pod-level autoscaler.

A container that runs to completion before app containers start — for setup like waiting on a dependency or fetching config.

Taints repel Pods from nodes; a Pod needs a matching toleration to schedule there — used to reserve/dedicate nodes.

Rules attracting/repelling Pods to nodes (affinity) or co-locating/spreading Pods relative to others (pod affinity/anti-affinity) for performance/HA.

The Deployment creates a new ReplicaSet and shifts Pods over gradually (maxSurge/maxUnavailable), keeping the app available; rollback reverts to the prior ReplicaSet.

A Service is the stable VIP+selector; EndpointSlices list the actual ready Pod IPs it routes to (updated as Pods come/go).

A Custom Resource Definition extends the API with new object types; an operator is a controller that reconciles those CRs to manage app lifecycle.

etcd holds all cluster state, so its health is the cluster's health. Run it as an odd-sized quorum (3 or 5 members) so it tolerates (n-1)/2 failures while keeping a majority for Raft consensus. Back it up regularly with etcdctl snapshot save (and test restores — a backup you can't restore is worthless). Protect it: TLS for peer + client traffic, firewall it to the control plane only, dedicated fast disk (etcd is latency-sensitive — slow disk = slow API server = cluster-wide pain). Watch the DB size against the quota (default ~2–8 GB), and defrag/compact periodically. Disaster recovery for the cluster basically is etcd restore — losing quorum without a backup means rebuilding state.

Respect the version skew rules and upgrade one minor version at a time (no jumping). Order: control plane first, then workers. (1) Back up etcd. (2) Read the release notes for deprecated/removed APIs and fix manifests (the classic upgrade-breaker). (3) Upgrade control-plane components (apiserver/controllers/scheduler) — with multiple control-plane nodes, do them one at a time to keep quorum and API availability. (4) Upgrade kubelet/kube-proxy on workers node by node: cordon → drain (respecting PDBs) → upgrade → uncordon, so workloads reschedule onto remaining nodes and capacity stays sufficient. (5) Bump CNI/CSI/addons as required by the new version. On managed (EKS/GKE/AKS) it's similar but the provider does the control plane; you still roll node groups. Always rehearse on staging and keep a rollback (etcd snapshot) ready.

kubectl describe node and check Conditions: MemoryPressure / DiskPressure / PIDPressure trigger kubelet eviction. DiskPressure is the most common — the node's disk (often /var/lib/kubelet or container image/log space) filled up; kubelet evicts Pods to reclaim it. Check df -h, image bloat, runaway logs, or emptyDir/ephemeral storage abuse. NotReady usually means the kubelet stopped reporting — check the kubelet service/logs (journalctl -u kubelet), container runtime health (crictl), node networking/CNI, and clock/cert issues. Also look for a noisy-neighbor Pod with no memory limit eating the node (set limits / use ResourceQuota / LimitRange). Fix the resource pressure, then add guardrails so one Pod can't take a node down.

Isolate on multiple axes. Namespaces per tenant as the unit of isolation. ResourceQuota (cap total CPU/mem/object counts per namespace) + LimitRange (default and max per-Pod requests/limits) so one tenant can't starve others. RBAC scoped to each namespace, least privilege. NetworkPolicies default-deny cross-namespace traffic. Pod Security Admission (restricted profile) to block privileged containers/host mounts. Optionally node isolation via taints + dedicated node pools (or even separate clusters / virtual clusters for hard multi-tenancy, since the kernel is shared — a true security boundary needs more than namespaces). Add policy-as-code (OPA Gatekeeper / Kyverno) to enforce org rules, and per-tenant quotas/cost visibility. The honest caveat: soft multi-tenancy reduces blast radius but isn't a hard security boundary.

Work layer by layer. (1) Endpoints: kubectl get endpoints svc / EndpointSlices — are the right Pods listed, and only ready ones? A flapping readiness probe adds/removes endpoints causing intermittent failures. (2) Selector match: Service selector vs Pod labels and the targetPort vs containerPort. (3) DNS: from a debug Pod, resolve the service name (CoreDNS issues, ndots, search-domain). (4) kube-proxy: stale iptables/IPVS rules on a node, or one node's proxy unhealthy → intermittent depending on which node you hit. (5) NetworkPolicy partially blocking. (6) App-level: one bad replica returning errors (check per-Pod), or connection-pool/keep-alive routing to a dead endpoint. Reproduce from inside the cluster (kubectl run -it --rm debug) to cut out Ingress/LB, then bisect inward.

Odd-sized quorum (3/5), regular snapshot backups (tested restores), TLS + firewalled to control plane, fast dedicated disk, monitor DB size vs quota, compact + defrag.

Respect version skew, one minor at a time: backup etcd, fix removed APIs, upgrade control plane (one node at a time for quorum), then workers node-by-node (cordon→drain respecting PDBs→upgrade→uncordon).

Namespaces + ResourceQuota + LimitRange, scoped RBAC, default-deny NetworkPolicies, Pod Security Admission (restricted), node isolation (taints/pools), and policy-as-code — note soft multi-tenancy isn't a hard security boundary.

Filtering (feasible nodes by resources/taints/affinity/topology) then scoring (best fit), then binds; extensible via the scheduling framework/plugins.

Check ready endpoints (flapping readiness), selector/targetPort, DNS, per-node kube-proxy rules, partial NetworkPolicy, or one bad replica; reproduce in-cluster to exclude Ingress/LB.

HPA scales Pods; when Pods can't schedule (insufficient nodes), cluster autoscaler adds nodes (and removes underused ones) — they work in tandem, watch for scale-up latency.

Least-privilege RBAC (no wildcard cluster-admin), default-deny NetworkPolicies, PSA restricted, encrypted etcd Secrets + external secrets, signed images + admission control, and runtime detection.

Node controller marks NotReady, taints it; after a timeout Pods are evicted/rescheduled elsewhere (if controller-managed); StatefulSet/PV reattachment and PDBs affect timing/safety.

Stable ordinal names + per-Pod PVCs that persist across reschedules; ordered, graceful rollout/scaling — for clustered stateful apps needing stable identity (DBs, Kafka).

From a debug Pod resolve service names; check CoreDNS pods/logs, the kube-dns Service/endpoints, ndots/search-domain in resolv.conf, and NetworkPolicy blocking port 53.

kubectl describe pod and read the scheduler Events — it states the reason. Common: insufficient resources (no node fits the requests — scale up / cluster-autoscaler / lower requests), taints with no toleration, nodeSelector/affinity nothing satisfies, unbound PVC (no provisioner / wrong AZ), or anti-affinity/topology-spread that can't be met. Cross-check kubectl get nodes and describe node for allocatable vs allocated.

kubectl describe node → Conditions: MemoryPressure/DiskPressure/PIDPressure trigger eviction; NotReady usually means the kubelet stopped reporting. Check the kubelet (journalctl -u kubelet), container runtime (crictl), CNI/networking, disk (/var/lib/kubelet full is the classic), and cert/clock issues. DiskPressure from image/log bloat is the most common. Fix the pressure/agent, and add limits so one Pod can't take the node down.

Layer by layer. kubectl get endpoints — are the right, ready Pods listed (a flapping readiness probe adds/removes endpoints = intermittent)? Check selector/targetPort match, DNS from a debug Pod (CoreDNS, ndots), kube-proxy (stale iptables/IPVS on one node → fails only via that node), a partial NetworkPolicy, and a single bad replica returning errors. Reproduce from inside the cluster (kubectl run -it --rm debug) to exclude Ingress/LB, then bisect inward.

Respect version skew, upgrade one minor at a time. Order: back up etcd, read release notes for removed APIs and fix manifests, upgrade the control plane (one node at a time to keep quorum/API up), then workers node by node: cordon → drain (respecting PodDisruptionBudgets) → upgrade kubelet/kube-proxy → uncordon. Bump CNI/CSI/addons as required. Rehearse on staging; keep the etcd snapshot as rollback. On managed clusters the provider does the control plane, you still roll node groups.

Per-namespace ResourceQuota (cap total CPU/mem/objects) + LimitRange (default and max per-Pod requests/limits) so no tenant can hog or run limitless pods. Use PriorityClasses so critical workloads preempt, and node isolation (taints/dedicated pools) for noisy tenants. Add NetworkPolicies and Pod Security Admission, and policy-as-code to enforce that every Pod sets requests/limits. The root cause is usually a Pod with no limits eating a node.

etcd is latency-sensitive and quota-bounded. Check the DB size vs quota (default ~2–8GB) — if exceeded, etcd goes read-only (alarm). Compact old revisions and defrag to reclaim space, then disarm the alarm. Ensure etcd is on a fast, dedicated disk (slow disk = slow API server = cluster-wide latency). Take a snapshot before maintenance. Long-term: reduce churn (events, large objects), monitor etcd metrics, and keep an odd-sized quorum.

describe pod → scheduler Events: insufficient resources, taints/affinity, unbound PVC, or topology constraints; check node allocatable.

describe node Conditions (Disk/Memory pressure), kubelet logs (journalctl), runtime (crictl), CNI, disk full on /var/lib/kubelet, certs/clock; fix pressure + add limits.

Ready endpoints (flapping readiness), selector/targetPort, DNS, per-node kube-proxy, partial NetworkPolicy, one bad replica; reproduce in-cluster, bisect inward.

One minor at a time: etcd backup, fix removed APIs, control plane node-by-node, then workers cordon→drain(PDB)→upgrade→uncordon; rehearse on staging.

ResourceQuota + LimitRange per namespace, PriorityClasses, node isolation, and enforce requests/limits via policy — usually a limitless Pod eats a node.

Check DB size vs quota, compact + defrag, ensure fast dedicated disk, snapshot before maintenance, reduce churn (events/large objects).

describe pod (OOMKilled/probe/mount/image) + logs --previous; common: bad config/missing secret, OOM (exit 137), aggressive liveness; reproduce locally.

Node resource pressure (usually DiskPressure from logs/images, or memory from a limitless Pod); set limits/quotas, add log rotation + disk-pressure alerts.

kubectl rollout status + describe: failing readiness on new Pods, image pull errors, insufficient resources, or maxUnavailable/PDB blocking; fix the new ReplicaSet's Pods.

StatefulSet with replicated storage/replication, back up first, drain one Pod at a time respecting PDB, verify replication/quorum between steps, and test failover before/after.