Kubernetes, the Engineering Way · Part 1

Stop Memorizing Kubernetes. Start Thinking Like an Engineer.

Don't memorize the resources. Understand the problems they solve.

Kubernetes has a lot of concepts: Pods, Deployments, Services, Ingress, ConfigMaps, Secrets, HPA, StatefulSets, Operators, controllers. You can spend weeks learning what each one does, memorize the YAML, and still get stuck the first time a real cluster misbehaves.

That's because knowing what a resource does is not the same as knowing why it exists. So let's start from the other end: not with the API objects, but with the engineering questions that produced them.

1.What happens when your application dies?

Your application is running in a Pod. Then the process crashes. Kubernetes should restart it, but "restart it" hides two different problems:

That something is a Deployment. A Deployment manages ReplicaSets, and a ReplicaSet keeps a set number of Pods running. Ask for three replicas and Kubernetes keeps working to have three, whatever happens to individual Pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3          # what you want, not how to get it
  ...

You never told Kubernetes how to restart anything. You stated what you want, and it keeps making that true.

The idea: declare the desired state; don't script the steps.

2.How do applications find each other?

Now another application needs to call yours. You could use Pod IP addresses, but Pods are replaced all the time and each replacement can get a new IP. Scale from three replicas to ten and the list changes again. Hard-coded IPs break almost immediately.

What you need is a stable name for the application, separate from whichever Pods happen to be running it. That's a Service. A Service selects Pods by label and gives them one stable address, and cluster DNS lets other workloads reach it by name. Your frontend calls backend, and never needs to know which Pod answers.

The idea: separate the logical identity from the physical instances.

3.How does outside traffic get in?

Customers need to reach your application from the internet. You could give every application its own cloud load balancer, but with 30 applications that's 30 load balancers to configure, secure and pay for.

You want one shared entry point, with rules like:

Ingress, and its newer successor the Gateway API, let you write those rules as Kubernetes resources. But a rule on its own moves no traffic. An Ingress controller or Gateway controller (for example NGINX, Traefik, or the AWS Load Balancer Controller) watches those resources and configures real proxies or load balancers to match.

The API records what should happen. A controller makes it happen.

The idea: separate intent from implementation.

4.Why keep configuration out of the image?

Your application needs a database address, API URLs and feature flags, and they differ between development, staging and production. Building a separate image for each environment would tie your code to where it runs.

A ConfigMap holds ordinary settings outside the image and passes them in as environment variables or files, so the same image runs everywhere.

Passwords, tokens and certificates go in a Secret instead, which keeps them apart from ordinary settings and lets you control who can read them. But a Secret is not secure by default. Its values are only base64-encoded, and anyone who can read Secrets through the API can read them. You still need RBAC, encryption at rest, and care with how credentials are handled.

$ kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
hunter2                # base64 is an encoding, not encryption

The better question isn't "did I use a Secret?" but "if someone can call the Kubernetes API, what can they read?"

The idea: separate configuration from code, and draw security boundaries explicitly.

5.What happens when demand changes?

Three replicas are fine until traffic grows tenfold. You could raise the count by hand, but then you're the one deciding when to scale up, and remembering to scale back down.

The Horizontal Pod Autoscaler (HPA) does it for you. It watches a metric, such as CPU use or a custom metric like requests per second, and adjusts the replica count.

Now go one step further. HPA asks for ten more Pods, but your nodes are full. The new Pods sit in Pending. HPA did its job; the cluster just has nowhere to put them. That's a second, separate scaling problem.

A node autoscaler solves it. On AWS EKS that's often Karpenter: it notices Pods that can't be scheduled, launches nodes that fit their requirements, and removes nodes again when they're no longer needed. (Cluster Autoscaler is the older alternative.)

So there are two layers of scaling:

Knowing why both exist is worth more than knowing their names.

The idea: scaling happens at more than one layer.

6.What if one application takes everything?

Ten applications share a node. One of them suddenly uses 4 GB of memory instead of the expected 500 MB, and everything else on the node starts to suffer.

Requests and limits stop one workload from starving the rest:

resources:
  requests: { cpu: 250m, memory: 512Mi }   # reserved; used for scheduling
  limits:   { cpu: "1",  memory: 512Mi }   # over 512Mi: OOMKilled

Limits don't replace capacity planning. Set requests too high and you waste capacity on idle reservations. Set them too low and Pods land on nodes that can't actually run them. The engineering work is finding the right numbers, and revisiting them.

The idea: shared resources need explicit allocation and isolation.

7.What if your application isn't disposable?

So far every Pod has been interchangeable: replace a frontend Pod and nobody notices. Databases, message brokers and other distributed systems are different. Each instance has its own identity and its own data.

A StatefulSet gives each Pod a stable name and network identity (db-0, db-1, db-2) and creates, updates and removes them in order. PersistentVolumes provide storage that outlives any single Pod, so db-0 gets the same data back after it's replaced.

That answers some questions and raises harder ones:

Kubernetes gives you the building blocks. It doesn't solve distributed systems for you.

The idea: different workloads need different lifecycle guarantees.

8.What if Kubernetes doesn't know about the thing you manage?

Say you run a database platform, and you want users to write: "I need a database with three replicas, 100 GB of storage and daily backups." Kubernetes has no such resource. You could build a separate management system, but then users have two control planes to learn.

Instead you can extend Kubernetes itself. A Custom Resource Definition (CRD) adds a new resource type to the API, say Database:

apiVersion: example.com/v1
kind: Database
metadata:
  name: orders
spec:
  replicas: 3
  storage: 100Gi
  backups: daily

A new resource type is only half the job, though. Something has to make each Database real. That's a controller: it watches the resource, compares what's asked for with what exists, and acts to close the gap. A CRD plus a controller that encodes how to run a particular system is the Operator pattern.

With it, you teach Kubernetes to manage things it was never designed for, using the same declarative model as everything else.

The idea: extend through APIs and reconciliation.


9.The idea that connects everything: reconciliation

Look back over the list:

They look like unrelated features, but they're all the same loop. A controller observes the system, compares what exists with what should exist, acts to reduce the difference, and repeats. Step through it with a Deployment:

A controller doesn't need to foresee every way things can go wrong. It only needs to know the goal and how to take one step toward it. If a step fails, the next pass of the loop tries again.

Nothing depends on one command succeeding. That's the real difference between Kubernetes and a folder of deployment scripts.


10.Learn to ask better questions

When you meet a new Kubernetes concept, start with the problem, not the YAML. Ask:

  1. What problem does this solve?
  2. Why wasn't an existing resource enough?
  3. What does it assume?
  4. What happens when the things it depends on fail?
  5. Which controller or component actually makes it work?
  6. What trade-offs does the design make?

Then read the docs, look at the implementation, and break things on purpose to see what happens.

The same goes for AI assistants. Don't just ask one what a resource does. Ask why it exists, push back on the design, and ask when it would be the wrong choice.

The goal isn't to remember more YAML. It's to be able to reason about the system.

Final thought

Kubernetes is one large system built from many small abstractions. Each abstraction is an engineering decision, each controller is a reconciliation loop, and each API is a contract between components.

Once you understand the problems, the design makes sense. Once you understand the design, you can reason about systems you haven't met yet.

Don't just learn Kubernetes. Learn to think the way it was designed.