This is the do-it version of the ephemeral vs persistent storage post. You run Postgres twice. The only thing that changes between the two runs is where it writes its data directory. You delete a pod both times and get opposite results.

Every step below shows the command and the output you should see, so you can tell “working” from “stuck” without guessing. The manifests are in the companion repo under manifests/.

Before you start

You need a running cluster and kubectl. k3s or k3d both work, since both ship local-path as the default StorageClass. Confirm it is there:

Two columns there matter later. RECLAIMPOLICY: Delete means removing the claim wipes the data. VOLUMEBINDINGMODE: WaitForFirstConsumer means the volume is not created until a pod mounts it. Both show up as “gotchas” further down, except now you will know they are expected.

Part 1: ephemeral, watch the database vanish

The data is real and it is there. Now break it.

The table is gone, not empty. Here is what just happened: the new pod got a brand new emptyDir, which is an empty directory. Postgres saw an empty data directory, assumed a fresh install, and ran initdb from scratch. Your table was never on that new disk. This is the failure the concept post opens with, reproduced in about thirty seconds.

Part 2: persistent, watch it survive

Pending with no volume looks broken. It is not. This is WaitForFirstConsumer from the storage class output earlier. The volume gets created when a pod actually needs it, not a second before.

Bound, with a real volume and 1Gi behind it.

Same deletion, opposite result. The new pod mounted the same claim, Postgres found its existing data directory, skipped initdb, and started on top of the data that was already there.

Where the data actually lives

The PVC is a real directory on the node’s disk. Find the path:

On k3s that directory is on the node’s real filesystem. On k3d the node is a container, so the same path is inside the k3d node container instead.

Gotchas and troubleshooting

These are the ones that actually cost me time. Each has a symptom so you can match it to what you are seeing.

If nothing is mounting it, that is expected (WaitForFirstConsumer). If a pod is scheduled and it still will not bind, check that the local-path provisioner is actually running:

The describe events at the bottom will tell you what it is waiting on.

Symptom: you change the persistent Deployment, the rollout never finishes, and kubectl get pods shows the new pod stuck in ContainerCreating with a multi-attach error in its events. Cause: a ReadWriteOnce volume can attach to one pod at a time, and the default RollingUpdate tries to start the new pod before the old one lets go. Fix is already in the manifest:

Recreate tears the old pod down before bringing the new one up, so the volume is free when the replacement needs it.

Symptom: the pod goes CrashLoopBackOff, and kubectl logs shows initdb refusing because the data directory is not empty. Cause: some volumes hand you a directory that already contains a lost+found from a fresh filesystem. local-path gives you a clean directory so this lab is fine, but on other provisioners point PGDATA at a subdirectory:

• emptyDir survives a container crash inside the pod. It does not survive a pod delete, a node drain, or a reschedule. It is not the safety net people assume.
• local-path pins data to one node. The PV carries node affinity, so the pod is forced back to the node holding the disk. On multiple nodes it cannot move freely, and if that node’s disk dies the data dies with it. No replication.
• The size request is not a hard limit. requests.storage: 1Gi is recorded on the PVC, but local-path does not enforce it. You can write past it until the node’s real disk fills.

Cleanup

The last command should return nothing for these resources. Deleting the PVC releases the PV, and local-path removes the directory on the node.