EngineeringPVC Retention Policies and Data Recovery Limits on Single-Node k3s
· PinLog
- #kubernetes
- #storage
- #persistence
- #recovery
When working with Kubernetes storage, it is easy to stop at “we attached a PVC, so the data persists.” But persists after what? A container restarting, a PVC being deleted, and a server losing its disk are different events. That distinction was what mattered when I examined PinLog’s storage configuration.
PinLog ran its k3s control plane and workloads together on a single AWS node. k3s is a lightweight Kubernetes distribution.
Problem
“We attached a PVC, so the data persists” did not distinguish a Pod restart, PVC deletion, and node loss. On its single node, PinLog kept the PostgreSQL data and backup PVCs on the same local storage.
Decision
I read the storage setup against those three events separately. I looked at each PV's actual reclaim policy rather than the StorageClass name, and treated deletion policy and recovery procedure as separate questions.
Result
Retain only stops automatic cleanup after PVC deletion, and a backup PVC on the same node does not cover node loss. Off-host copies and a restore procedure remained the next step, to be designed and verified on their own.
What Actually Survives
A Pod is a unit of execution. A PVC is a request for storage, and a PV represents the storage resource bound to that request. Writing application data to the volume mounted through a PVC separates its lifetime from the container’s lifetime. It does not preserve every file written anywhere inside a container.
After a Pod restarts or is replaced, using the existing data still requires an intact PVC and a surviving, accessible disk behind it. With local storage, access to the node holding the data matters too. Restarting execution and regaining access to storage are separate problems.
Two StorageClasses Compared
A StorageClass defines how volumes are provisioned, and its reclaim policy decides what happens to a volume after its PVC is deleted. The default StorageClass, local-path, had a Delete reclaim policy. local-path-retain used Retain. Both used node-local storage; the difference was how a volume would be handled after its PVC was deleted.
local-path · Delete
When PVC deletion releases the volume, the policy calls for deleting the PV and its backing storage. The provisioner carries out the storage cleanup.
local-path-retain · Retain
After PVC deletion, the PV and backing data are retained rather than automatically cleaned up. An operator must handle subsequent reuse or cleanup.
More precisely, the reclaim policy applies to the PV. A dynamically provisioned PV inherits that policy from its StorageClass. For an important volume, the class name is therefore not enough: the actual PV policy matters. Changing a StorageClass does not retroactively change the policies of all existing PVs.
Retain should not be read as “automatic recovery after deleting a PVC.” It does not preserve the PVC object itself, and a retained PV does not simply reconnect itself to a new claim. It stops automatic reclamation after deletion; it is not a procedure for locating the data and reconnecting it to the application.
When Binding Happens
Both classes used WaitForFirstConsumer as their volume binding mode. Rather than fixing the volume’s location as soon as a PVC is created, binding and dynamic provisioning account for the scheduling requirements of the Pod that will consume it. This helps align the location of local storage with the node where the Pod can run.
PVC request
Specify the StorageClassConsumer Pod
Account for scheduling constraintsVolume binding
Use local storage on the selected node
This is a setting for placement and binding time. Waiting to bind does not create a replica or move data to another server. In a single-node configuration, scheduling-aware binding does not take storage outside that node.
Data and Backup, One Boundary
The PostgreSQL data and backup PVCs, along with the other PVCs that held state, all used local-path-retain. Not just PostgreSQL’s data but other state, too, had a lifetime separate from its Pod.
The single-node storage boundary
Separate PostgreSQL data and backup PVCs distinguish their intended purposes. But two claims do not imply two failure domains. If both depend on local storage on the same node, Retain alone cannot address loss of the node or its entire storage. Calling a volume “backup” does not change that boundary.
Assessing backups must therefore go beyond the existence of storage space. What data is produced, when is it produced, is there a copy outside the original host, and can that copy be used to reconstruct the service? Those are separate questions. Providing backup space on the same node is not the same step as establishing an off-host recovery path.
Deletion Policy vs. Recovery
Retain matters because it prevents PVC deletion from directly triggering data cleanup. It also leaves someone responsible for the retained volume: who manages it, when it is reused, and when it is removed. A retention-friendly policy does not eliminate operational work.
The conclusion was not “we have persistent volumes, so we are safe.” A Pod restart requires the existing PVC and surviving disk. PVC deletion brings the PV’s reclaim policy into play. Losing the entire host requires data outside it and a recovery procedure. These conditions are too different to collapse into a single promise of persistence.
The first question to ask of storage is “How much can disappear while still leaving a way to start again?” A PVC separates execution from the lifetime of data; Retain changes reclamation after deletion. Recovery beyond the host is the next boundary, and it needs its own design and verification.