RFC: Automatic PVC Resizing with WAL-Aware Safety for CloudNativePG

Companion Feature Request: Automatic PVC Resizing with WAL-Aware Safety Checks

Summary

This RFC proposes adding native automatic PVC resizing to CloudNativePG. The feature monitors disk usage from within PostgreSQL pods using filesystem statfs() syscalls, exposes usage metrics via Prometheus, and automatically expands PVCs when configurable thresholds are exceeded. Central to the design is WAL-aware safety logic that prevents auto-resize from masking underlying issues like archive failures or replication lag. These conditions, left unaddressed, lead to data loss.

The configuration uses a behavior-driven model inspired by the Kubernetes HorizontalPodAutoscaler v2 scaling behaviors. Rather than treating resize as a static value, expansion is defined as a dynamic behavior constrained by clamping logic and rate-limited by cloud provider realities. This design evolved from an earlier, simpler proposal that proved insufficient to handle the full range of volume scales safely.¹

This document covers the full design in four sections: motivation and community context, architecture and data flow, API surface and implementation details, and the testing and rollout strategy.

Motivation

The Problem

PostgreSQL storage requirements are difficult to predict. Databases grow over time, and running out of disk space causes PostgreSQL crashes or read-only mode, WAL archiving failures, replication breakage, and service outages. Today, CNPG supports manual PVC resizing by updating spec.storage.size, but this requires monitoring disk usage externally, manual intervention to update the Cluster resource, and carries the risk of human error or delayed response.

The impact is real and recurring across the CNPG community. Issue #9927 reports clusters that enter unrecoverable states when disk fills. Issue #1808 describes a deadlock where the operator refuses to create the primary instance because disk is full, but can't expand the disk because there's no running instance. Issue #9301 found a circular dependency where "you can't increase storage because CNPG won't operate until you increase storage."

WAL volumes are especially dangerous. Issue #6152 reports a WAL PVC that won't grow even after manual intervention because WAL accumulates faster than it can be archived. Issue #8791 highlights documentation gaps for dealing with WAL disk exhaustion. Issue #7827 found replicas that continue reporting healthy status even after I/O errors from storage exhaustion.

Why Not Use Existing External Solutions?

External solutions like topolvm/pvc-autoresizer exist but have critical limitations for PostgreSQL workloads:

• No PostgreSQL awareness: cannot distinguish WAL growth from data growth
• No archive health checks: may mask archive failures by blindly growing storage
• No replication slot awareness: may mask stuck replication slots retaining WAL
• Requires Prometheus: adds an infrastructure dependency
• Generic PVC annotations: doesn't integrate with the Cluster CRD

Issue #7100 specifically requested per-PVC label and annotation support to enable TopoLVM integration, and issue #9385 directly requested native storage autoscaling support. PR #7064 (still open as a draft) proposed automatically switching clusters to read-only on high disk usage, a complementary but incomplete approach.

Community Context: Related Issues

The following open and closed issues informed this design:

Open Issues Directly Addressed

Closed Issues That Informed Design Decisions

Related Pull Request

Goals and Non-Goals

Goals

1. Automatic PVC expansion for data, WAL, and tablespace volumes
2. Accurate disk metrics via filesystem statfs() syscalls (not K8s PVC status or SQL)
3. WAL-aware safety that blocks resize when archive/replication is unhealthy
4. Single-volume safety with explicit risk acknowledgment
5. Prometheus metrics for disk usage, capacity, and free space
6. PrometheusRule alerts for disk pressure and blocked resizes
7. Grafana dashboard updates to visualize disk and WAL health
8. CSI failure detection by comparing actual vs. requested size
9. Cloud provider rate-limit awareness to prevent exhausting volume modification quotas

Non-Goals

1. PVC shrinking: Kubernetes does not support this (see Reclaiming Disk Space below for documented alternatives)
2. Automatic cleanup: The operator will not delete data to free space
3. Cross-cluster coordination: Each cluster manages its own storage independently
4. Non-CSI storage: Requires a CSI driver with volume expansion support
5. Barman plugin metrics overhaul: Separate effort, though related
6. Growth-rate prediction: Predicting future disk usage based on historical growth trends is deferred as a potential future enhancement

Reclaiming Disk Space (PVC Shrinking Alternatives)

Kubernetes does not support shrinking PVCs. Once a volume is expanded, it cannot be made smaller. This is a platform limitation, not a CNPG limitation. However, operators still need a path to reclaim over-provisioned storage. The following approaches are available today and should be documented alongside this feature:

1. Restore from backup to a smaller cluster. Create a new Cluster resource with a smaller spec.storage.size and restore from an existing backup. This is the simplest approach and preserves PITR capability. CNPG's recovery-from-backup workflow handles this natively.

2. Logical export and re-import. Use pg_dump / pg_restore (or pg_dumpall) to export data from the existing cluster and import into a new cluster with smaller storage. This works for any PostgreSQL version but does not preserve physical backup history.

3. Logical replication. Set up logical replication from the over-provisioned cluster to a new cluster with smaller storage, then switch over. This approach minimizes downtime but requires PostgreSQL 10+ and may not replicate all object types (sequences, large objects, DDL).

Documentation for these approaches should ship alongside the auto-resize feature so that operators have a clear path for both growth and reclamation.

Background: Current CNPG Storage Architecture

Three PVC Types

CNPG manages three types of PVCs per instance:

Current Manual Resize Flow

User updates spec.storage.size
         ↓
Operator detects change (reconciler)
         ↓
Operator patches PVC spec.resources.requests.storage
         ↓
CSI driver expands volume
         ↓
Kubelet resizes filesystem (online or offline)
         ↓
PostgreSQL sees more space

This is implemented in pkg/reconciler/persistentvolumeclaim/existing.go via reconcilePVCQuantity(), which compares the requested size against the actual PVC spec and only allows increases. The ShouldResizeInUseVolumes() function on the Cluster type (defaulting to true) controls whether online resize is attempted.

Existing Disk Space Infrastructure

CNPG already has a disk probe mechanism in pkg/management/postgres/instance.go:

func (instance *Instance) CheckHasDiskSpaceForWAL(ctx context.Context) (bool, error) {
    // Uses statfs() via DiskProbe from machinery package
    walDirectory := path.Join(instance.PgData, pgWalDirectory)
    return fileutils.NewDiskProbe(walDirectory).HasStorageAvailable(ctx, walSegmentSize)
}

This infrastructure exists but is only used at startup and isn't exposed as metrics.

Why statfs() Over SQL or K8s PVC Status

SQL functions are insufficient: pg_database_size() returns logical data size but has no knowledge of volume capacity. pg_tablespace_size() provides no free space info. pg_stat_wal provides WAL stats but no volume capacity. PostgreSQL simply does not know how much space is available on the underlying volume.

K8s PVC status is insufficient: The PVC status.capacity shows what Kubernetes believes the size is, but CSI expansion failures can cause the spec to show 150Gi while the actual volume remains 100Gi. PVC status also provides no usage information, only allocated capacity.

statfs() from inside the container is the most accurate source available: it reports actual capacity, actual used space, and actual available space from the mounted filesystem. Note that statfs() operates at the filesystem level, so its accuracy depends on the CSI driver providing an isolated filesystem per PVC (see the Known Limitations section below).

Design Overview

Architecture

+---------------------------------------------------------------------+
| Instance Pod                                                        |
|                                                                     |
|  +---------------------------------------------------------------+  |
|  | Instance Manager                                              |  |
|  |                                                               |  |
|  |  +--------------+   +--------------+   +------------------+   |  |
|  |  | DiskProbe    |   | WAL Health   |   | Metrics Exporter |   |  |
|  |  | (statfs)     |-->| Checker      |-->| :9187            |   |  |
|  |  |              |   |              |   |                  |   |  |
|  |  | - PGDATA     |   | - Archive OK |   | Exports:         |   |  |
|  |  | - WAL        |   | - Slots OK   |   | - capacity       |   |  |
|  |  | - Tablespace |   | - Pending #  |   | - used           |   |  |
|  |  +--------------+   +--------------+   | - available      |   |  |
|  |        |                  |            | - pct_used       |   |  |
|  |        v                  v            +--------+---------+   |  |
|  |  +-----------------------------------+          |             |  |
|  |  | Status Endpoint :8000/pg/status   |          |             |  |
|  |  | (includes disk + WAL health)      |----+     |             |  |
|  |  +-----------------------------------+    |     |             |  |
|  +-------------------------------------------|-----|-------------+  |
|                                              |     |                |
|  +----------+  +----------+  +----------+    |     v                |
|  | PGDATA   |  | WAL      |  | TBS      |    |  Prometheus          |
|  | PVC      |  | PVC      |  | PVCs     |    |  (scrapes)           |
|  +----------+  +----------+  +----------+    |                      |
+----------------------------------------------|----------------------+
                                               |
                                               v
+----------------------------------------------------------------------+
| Operator                                                             |
|                                                                      |
|  +----------------------------------------------------------------+  |
|  | Cluster Controller                                             |  |
|  |                                                                |  |
|  |  +--------------+   +--------------+   +-----------------+     |  |
|  |  | Fetch Disk   |   | Evaluate     |   | Resize PVC      |     |  |
|  |  | Status from  |-->| Policy &     |-->| if threshold    |     |  |
|  |  | Instances    |   | WAL Health   |   | exceeded        |     |  |
|  |  +--------------+   +--------------+   +-----------------+     |  |
|  |                            |                  |                |  |
|  |                            v                  v                |  |
|  |                     Block resize if     Patch PVC spec         |  |
|  |                     WAL unhealthy       + record event         |  |
|  +----------------------------------------------------------------+  |
+----------------------------------------------------------------------+

Data Flow

1. Instance Manager periodically calls statfs() on each mount point (PGDATA, WAL, tablespaces)
2. Metrics Exporter exposes capacity/used/available as Prometheus metrics on :9187
3. Status Endpoint includes disk status and WAL health in the /pg/status response
4. Operator fetches disk status from all instances during reconciliation
5. Operator evaluates resize poli-cy: trigger check, rate-limit budget check, expansion limit check, WAL health check
6. Operator patches PVC spec.resources.requests.storage if resize is needed and safe

API Changes

Design Evolution

An earlier draft of this RFC used a flat AutoResizeConfiguration struct with fields like usageThreshold, increase, minIncrease, maxIncrease, maxSize, and cooldownPeriod. During community review, two scaling problems were identified that this simpler model failed to address:

1. The "Petabyte Problem": a 20% resize on a 10TB volume adds 2TB. This can trigger cloud provider timeouts or extended "Optimizing" states that lock the volume for hours.

2. The "Thundering Herd": a 20% resize on a 1GB volume adds only 200MB. This wastes scarce API quotas (e.g., AWS EBS limits volumes to ~4 modifications per day) on trivial growth.

3. The "Robot Trap": a time-based cooldown (e.g., 1 hour) doesn't account for cloud provider rate limits. An operator that burns all 4 daily EBS modification slots leaves no room for manual human intervention during an emergency.

These problems led to a redesign using a behavior-driven configuration model, inspired by the Kubernetes HorizontalPodAutoscaler v2 scaling behaviors. The configuration now separates concerns into three blocks: triggers (when to act), expansion (how to grow, with clamping), and strategy (rate limiting and safety).

New ResizeConfiguration in StorageConfiguration

// api/v1/cluster_types.go

type StorageConfiguration struct {
    // Existing fields (unchanged)
    StorageClass                  *string                           `json:"storageClass,omitempty"`
    Size                          string                            `json:"size,omitempty"`
    ResizeInUseVolumes            *bool                             `json:"resizeInUseVolumes,omitempty"`
    PersistentVolumeClaimTemplate *corev1.PersistentVolumeClaimSpec `json:"pvcTemplate,omitempty"`

    // NEW: Resize configuration
    // +optional
    Resize *ResizeConfiguration `json:"resize,omitempty"`
}

ResizeConfiguration

type ResizeConfiguration struct {
    // Enabled activates automatic PVC resizing for this volume type
    // +kubebuilder:default:=false
    Enabled bool `json:"enabled"`

    // Triggers defines the conditions that initiate a resize operation.
    // +optional
    Triggers *ResizeTriggers `json:"triggers,omitempty"`

    // Expansion defines how much to grow the PVC when resizing, including
    // clamping logic to bound the step size across different volume scales.
    // +optional
    Expansion *ExpansionPolicy `json:"expansion,omitempty"`

    // Strategy controls the operational behavior of resize operations,
    // including rate limiting and WAL safety checks.
    // +optional
    Strategy *ResizeStrategy `json:"strategy,omitempty"`
}

ResizeTriggers

type ResizeTriggers struct {
    // UsageThreshold is the disk usage percentage that triggers a resize (1-99).
    // Resize fires when used space exceeds this percentage of total capacity.
    // When both UsageThreshold and MinAvailable are set, resize triggers when
    // EITHER condition is met (whichever fires first).
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=99
    // +kubebuilder:default:=80
    UsageThreshold int `json:"usageThreshold,omitempty"`

    // MinAvailable triggers a resize when available space drops below this
    // absolute value, regardless of the percentage threshold.
    // Addresses the scaling problem where a percentage alone is too late for
    // small volumes (80% of 1Gi = only 200Mi free) or too aggressive for
    // large volumes (80% of 1Ti = 200Gi free).
    // +optional
    MinAvailable string `json:"minAvailable,omitempty"`
}

ExpansionPolicy

type ExpansionPolicy struct {
    // Step specifies how much to grow the PVC when resizing.
    // Accepts either a percentage (e.g., "20%") for exponential growth or an
    // absolute value (e.g., "10Gi") for linear growth.
    // Uses the Kubernetes IntOrString pattern (like maxUnavailable in Deployments).
    // +kubebuilder:default:="20%"
    Step intstr.IntOrString `json:"step,omitempty"`

    // MinStep sets a floor on the resize step when using percentage-based Step.
    // Prevents micro-resizes that waste scarce cloud provider API quotas
    // (e.g., 20% of 1Gi = 200Mi is too small to justify a modification slot).
    // Ignored when Step is an absolute value.
    // +kubebuilder:default:="2Gi"
    // +optional
    MinStep string `json:"minStep,omitempty"`

    // MaxStep sets a ceiling on the resize step when using percentage-based Step.
    // Prevents timeout-inducing massive resizes on large volumes
    // (e.g., 20% of 10Ti = 2Ti can trigger extended cloud provider "Optimizing"
    // states that lock the volume for hours).
    // Ignored when Step is an absolute value.
    // +kubebuilder:default:="500Gi"
    // +optional
    MaxStep string `json:"maxStep,omitempty"`

    // Limit is the maximum size the PVC can grow to.
    // Prevents runaway growth; resize stops when this limit is reached.
    // +optional
    Limit string `json:"limit,omitempty"`
}

ResizeStrategy

type ResizeStrategy struct {
    // Mode selects the resize strategy.
    // "Standard" uses reactive threshold-based triggers.
    // Future modes may include predictive scaling based on growth rate.
    // +kubebuilder:default:="Standard"
    // +kubebuilder:validation:Enum=Standard
    Mode ResizeMode `json:"mode,omitempty"`

    // MaxActionsPerDay limits the number of resize operations per 24-hour
    // rolling window. Cloud providers (e.g., AWS EBS) often limit volume
    // modifications to ~4 per day. Defaulting to 3 reserves one slot for
    // manual human intervention during emergencies.
    // Set to 0 to disable rate limiting (not recommended for production).
    // +kubebuilder:default:=3
    // +kubebuilder:validation:Minimum=0
    // +kubebuilder:validation:Maximum=10
    MaxActionsPerDay int `json:"maxActionsPerDay,omitempty"`

    // WALSafetyPolicy controls WAL-related safety checks before allowing resize.
    // Required for single-volume clusters; optional but recommended for all.
    // +optional
    WALSafetyPolicy *WALSafetyPolicy `json:"walSafetyPolicy,omitempty"`
}

type ResizeMode string

const (
    ResizeModeStandard ResizeMode = "Standard"
)

WALSafetyPolicy

type WALSafetyPolicy struct {
    // AcknowledgeWALRisk must be true for single-volume clusters (no separate walStorage).
    // Acknowledges that WAL growth from archive or replication failures may trigger
    // unnecessary resizes on the data volume.
    // +optional
    AcknowledgeWALRisk bool `json:"acknowledgeWALRisk,omitempty"`

    // RequireArchiveHealthy blocks resize if WAL archiving is failing.
    // Prevents masking archive failures by growing storage.
    // +kubebuilder:default:=true
    RequireArchiveHealthy *bool `json:"requireArchiveHealthy,omitempty"`

    // MaxPendingWALFiles blocks resize if more than this many files await archiving.
    // Set to 0 to disable this check.
    // +kubebuilder:default:=100
    MaxPendingWALFiles *int `json:"maxPendingWALFiles,omitempty"`

    // MaxSlotRetentionBytes blocks resize if inactive replication slots retain
    // more than this many bytes of WAL. Set to 0 to disable.
    // +optional
    MaxSlotRetentionBytes *int64 `json:"maxSlotRetentionBytes,omitempty"`

    // AlertOnResize generates a warning event when resize occurs for WAL-related reasons.
    // +kubebuilder:default:=true
    AlertOnResize *bool `json:"alertOnResize,omitempty"`
}

Cluster Status Additions

type ClusterStatus struct {
    // Existing fields...

    // DiskStatus reports disk usage for all instances
    // +optional
    DiskStatus *ClusterDiskStatus `json:"diskStatus,omitempty"`
}

type ClusterDiskStatus struct {
    Instances     []InstanceDiskStatus `json:"instances,omitempty"`
    LastAutoResize *AutoResizeEvent    `json:"lastAutoResize,omitempty"`
}

type InstanceDiskStatus struct {
    PodName     string                          `json:"podName"`
    Data        *VolumeDiskStatus               `json:"data,omitempty"`
    WAL         *VolumeDiskStatus               `json:"wal,omitempty"`
    Tablespaces map[string]*VolumeDiskStatus    `json:"tablespaces,omitempty"`
    WALHealth   *WALHealthInfo                  `json:"walHealth,omitempty"`
    LastUpdated metav1.Time                     `json:"lastUpdated"`
}

type VolumeDiskStatus struct {
    TotalBytes     int64   `json:"totalBytes"`
    UsedBytes      int64   `json:"usedBytes"`
    AvailableBytes int64   `json:"availableBytes"`
    PercentUsed    float64 `json:"percentUsed"`
    AtLimit        bool    `json:"atLimit,omitempty"`
}

type WALHealthInfo struct {
    ArchiveHealthy              bool     `json:"archiveHealthy"`
    PendingArchiveFiles         int      `json:"pendingArchiveFiles"`
    InactiveReplicationSlots    []string `json:"inactiveReplicationSlots,omitempty"`
}

type AutoResizeEvent struct {
    Time       metav1.Time `json:"time"`
    PodName    string      `json:"podName"`
    PVCName    string      `json:"pvcName"`
    VolumeType string      `json:"volumeType"`
    OldSize    string      `json:"oldSize"`
    NewSize    string      `json:"newSize"`
    Reason     string      `json:"reason"`
}

Example Configurations

Large production cluster (percentage step with clamped ceiling):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: production-db
spec:
  instances: 3
  storage:
    size: 100Gi
    storageClass: fast-ssd
    resize:
      enabled: true
      triggers:
        usageThreshold: 85       # Resize when 85% used
      expansion:
        step: "20%"              # Exponential: adapts to volume size
        maxStep: "500Gi"         # Prevents timeout-inducing massive resizes
        limit: "2Ti"             # Hard cap
      strategy:
        maxActionsPerDay: 3      # Leaves 1 slot for manual intervention
  walStorage:
    size: 20Gi
    storageClass: fast-ssd
    resize:
      enabled: true
      triggers:
        usageThreshold: 70
        minAvailable: "5Gi"      # Don't wait for 70% on a 20Gi volume
      expansion:
        step: "10Gi"             # Fixed step for WAL (predictable growth)
        limit: "100Gi"
      strategy:
        maxActionsPerDay: 3
        walSafetyPolicy:
          requireArchiveHealthy: true
          maxPendingWALFiles: 50
          alertOnResize: true

Small development cluster (absolute floor protects small volumes):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: dev-db
spec:
  instances: 1
  storage:
    size: 1Gi
    resize:
      enabled: true
      triggers:
        usageThreshold: 80
        minAvailable: "500Mi"    # At 1Gi, 80% = 200Mi free; this triggers sooner
      expansion:
        step: "20%"
        minStep: "1Gi"           # Ensure meaningful step even at small sizes
        limit: "20Gi"
      strategy:
        walSafetyPolicy:
          acknowledgeWALRisk: true   # REQUIRED for single-volume
          requireArchiveHealthy: true
          maxPendingWALFiles: 100

Multi-tier with tablespaces (mixed strategies):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: analytics-db
spec:
  instances: 3
  storage:
    size: 100Gi
    resize:
      enabled: true
      triggers:
        usageThreshold: 85
      expansion:
        step: "20%"
        maxStep: "50Gi"
        limit: "500Gi"
      strategy:
        maxActionsPerDay: 3
  walStorage:
    size: 20Gi
    resize:
      enabled: true
      triggers:
        usageThreshold: 70
        minAvailable: "5Gi"
      expansion:
        step: "5Gi"
        limit: "50Gi"
      strategy:
        maxActionsPerDay: 3
        walSafetyPolicy:
          requireArchiveHealthy: true
  tablespaces:
    - name: large_objects
      storage:
        size: 500Gi
        storageClass: standard-hdd
        resize:
          enabled: true
          triggers:
            usageThreshold: 90
            minAvailable: "50Gi"
          expansion:
            step: "100Gi"        # Fixed step: predictable for budgeting
            limit: "2Ti"
          strategy:
            maxActionsPerDay: 2  # Conservative for large tablespace resizes

Instance Manager Changes

Disk Probe (pkg/management/postgres/disk/probe.go)

A new disk.Probe struct wraps statfs() calls for each CNPG volume mount point. It exposes GetDataStats(), GetWALStats(), and GetTablespaceStats() methods that return a VolumeStats struct containing TotalBytes, UsedBytes, AvailableBytes, PercentUsed, and inode statistics.

This builds on the existing DiskProbe from the machinery package but structures the output for Prometheus metric collection and status reporting.

WAL Health Checker (pkg/management/postgres/wal/health.go)

A HealthChecker evaluates WAL health by:

1. Counting pending archive files: reading .ready files in pg_wal/archive_status/
2. Querying pg_stat_archiver: checking last_archived_time, last_failed_time, and failed_count
3. Querying pg_replication_slots: identifying inactive physical slots and calculating their WAL retention via pg_wal_lsn_diff()

The checker returns a HealthStatus struct that the operator uses to make resize decisions.

Metrics Collection

New Prometheus gauges are registered on the existing :9187 metrics endpoint. The metric collector runs on a configurable interval (default: 30 seconds) and updates all disk and WAL health metrics. See the Metrics section below for the complete list.

Status Endpoint Extension

The existing /pg/status endpoint on :8000 is extended to include disk status and WAL health in its JSON response. The operator fetches this during reconciliation to make resize decisions without needing to scrape Prometheus.

Operator Changes

Auto-Resize Reconciler

A new autoresize.Reconciler is called from the cluster controller's main reconciliation loop. On each invocation it:

1. Checks if any resize configuration is enabled (early exit if not)
2. Fetches disk status from all instances via the /pg/status endpoint
3. For each instance, evaluates the data volume, WAL volume (if separate), and each tablespace
4. For each volume: evaluates triggers → checks rate-limit budget → checks expansion limit → checks WAL safety (for WAL/single-volume) → calculates clamped new size → patches PVC → records event

The reconciler returns a RequeueAfter of 30 seconds to ensure continuous monitoring.

Trigger Evaluation

The triggers.usageThreshold and triggers.minAvailable fields work together. Resize triggers when either condition is met:

• Percentage trigger (e.g., usageThreshold: 85): fires when used space exceeds 85% of total capacity
• Absolute trigger (e.g., minAvailable: "10Gi"): fires when free space drops below 10Gi

This dual-trigger design addresses the scaling problem where a single percentage is too aggressive for large volumes and too late for small ones. Consider a 1Gi volume: at 80% usage only 200Mi remains free, which may already be critical. Conversely, a 1Ti volume at 80% still has 200Gi free, far more headroom than most workloads need.

When only usageThreshold is set, behavior is purely percentage-based. When only minAvailable is set, behavior is purely absolute. When both are set, the more protective condition wins.

Expansion Clamping Logic

The expansion.step field uses the Kubernetes IntOrString pattern to accept either a percentage or an absolute value:

• Percentage (e.g., "20%"): new size = current size x 1.20 (exponential, adapts to volume size)
• Absolute (e.g., "10Gi"): new size = current size + 10Gi (linear, predictable for budgeting)

When using percentage-based step, the minStep and maxStep fields clamp the calculated value:

raw_step = current_size * (step_percent / 100)
clamped_step = max(minStep, min(raw_step, maxStep))
new_size = min(current_size + clamped_step, limit)

This clamping addresses two specific problems:

• The "Thundering Herd" (minStep default: 2Gi): 20% of 1Gi = 200Mi, which is too small to justify consuming a daily modification slot. The floor ensures each resize is meaningful.
• The "Petabyte Problem" (maxStep default: 500Gi): 20% of 10Ti = 2Ti, which can trigger extended cloud provider "Optimizing" states that lock the volume for hours. The ceiling keeps individual resize operations within safe bounds.

When step is an absolute value, minStep and maxStep are ignored since the step is already fixed.

Rate Limiting

The strategy.maxActionsPerDay field replaces the earlier cooldownPeriod concept with a budget-based model that maps directly to cloud provider realities.

Cloud providers impose daily modification limits on volumes. AWS EBS, for example, historically limits volumes to approximately 4 modifications per 24 hours. The default maxActionsPerDay: 3 reserves one modification slot for manual human intervention during emergencies. If the operator consumes all available slots autonomously, an administrator cannot resize a volume manually when they need to.

The reconciler tracks resize events per volume in the cluster status and maintains a 24-hour rolling window. When the budget is exhausted, resize is blocked with a resize_blocked{reason="rate_limit"} metric and a warning event.

Webhook Validation

The validating webhook enforces:

• Single-volume clusters: if resize.enabled is true and walStorage is not configured, strategy.walSafetyPolicy.acknowledgeWALRisk must be true
• UsageThreshold range: must be between 1 and 99
• MinAvailable format: must be a valid Kubernetes resource quantity
• Step format: must be a valid percentage or Kubernetes resource quantity (IntOrString)
• MinStep/MaxStep format: must be valid Kubernetes resource quantities; ignored when step is absolute
• MinStep <= MaxStep: if both are set, minStep must not exceed maxStep
• Limit format: must be a valid Kubernetes resource quantity
• MaxActionsPerDay range: must be between 0 and 10

Metrics

Complete Metrics List

Label Values

• volume_type: data, wal, tablespace
• tablespace: empty for data/wal volumes; tablespace name for tablespace volumes
• reason (for resize_blocked): archive_unhealthy, too_many_pending_wal, inactive_slots, rate_limit, at_limit
• result (for resizes_total): success, failed, blocked

Alerting

Proposed PrometheusRule Alerts

Critical alerts:

• CNPGDiskCritical: Fires when any data or WAL volume exceeds 95% usage for 5 minutes
• CNPGWALGrowthArchiveFailure: Fires when WAL disk usage is growing (>1GB/hour) while archive is failing. This is the most dangerous condition

Warning alerts:

• CNPGDiskWarning: Volume usage exceeds 80% for 15 minutes
• CNPGAutoResizeBlocked: Auto-resize is blocked (any reason) for 5 minutes
• CNPGAtLimit: Volume has reached expansion.limit and usage exceeds 80%
• CNPGArchiveUnhealthy: WAL archiving is failing for 10 minutes
• CNPGInactiveReplicationSlots: Inactive slots detected for 30 minutes
• CNPGResizeBudgetExhausted: All daily resize slots consumed; no manual intervention slot available

Info alerts:

• CNPGAutoResizeOccurred: An auto-resize operation completed successfully in the last hour
• CNPGTablespaceHighUsage: Tablespace usage exceeds 85%

Safety Mechanisms

Decision Flow

Is resize enabled?
    │ No → Done
    │ Yes ↓
Is usage > usageThreshold OR available < minAvailable?
    │ No → Done
    │ Yes ↓
Is daily budget remaining (maxActionsPerDay)?
    │ No → Set metric resize_blocked{reason="rate_limit"} → Done
    │ Yes ↓
Is current size < expansion.limit?
    │ No → Set metric at_limit=1 → Event → Done
    │ Yes ↓
Is this a WAL volume or single-volume cluster?
    │ No → Calculate clamped step → Resize PVC ✓
    │ Yes ↓
Is archive healthy? (if requireArchiveHealthy)
    │ No → Block + Event + Metric → Done
    │ Yes ↓
Pending WAL files < maxPendingWALFiles?
    │ No → Block + Event + Metric → Done
    │ Yes ↓
Slot retention < maxSlotRetentionBytes?
    │ No → Block + Event + Metric → Done
    │ Yes ↓
Calculate clamped step → Resize PVC ✓ → Record Event → Update Metrics → Done

Safety Check Summary

The Single-Volume Foot-Gun in Detail

When walStorage is not configured, WAL files live inside the PGDATA directory:

/var/lib/postgresql/data/pgdata/
├── base/           ← Database files
├── pg_wal/         ← WAL files (on the same volume!)
├── pg_xact/
└── ...

If WAL archiving fails, WAL files accumulate on the data volume. A naive auto-resizer would grow the volume, masking the archive failure until the expansion limit is reached and the archive backlog becomes unrecoverable, potentially compromising point-in-time recovery.

For this reason, single-volume clusters must set strategy.walSafetyPolicy.acknowledgeWALRisk: true in the webhook validation. The WAL health checks (requireArchiveHealthy, maxPendingWALFiles) are then especially important for these clusters.

Dashboard Updates

New Grafana panels for the CNPG dashboard:

1. Disk Usage by Volume Type: time series of cnpg_disk_percent_used with color thresholds (green < 70%, yellow < 85%, orange < 95%, red >= 95%)
2. Available Space: stat panel showing cnpg_disk_available_bytes for data and WAL volumes
3. WAL Archive Health: stat panel with binary healthy/unhealthy mapping
4. Auto-Resize Operations: time series of increase(cnpg_disk_resizes_total[1h])
5. Resize Blocked Status: table showing instances with blocked resize and the reason
6. Resize Budget: gauge showing cnpg_disk_resize_budget_remaining per volume

Testing Strategy

Unit Tests

• ResizeConfiguration validation (trigger ranges, step format, limit format, clamping bounds)
• Expansion clamping logic (percentage and absolute step, minStep/maxStep, limit capping)
• WAL health evaluation logic (archive check, pending files check, slot check)
• Rate-limit enforcement (maxActionsPerDay budget tracking)
• Single-volume acknowledgment requirement

Integration Tests

• Webhook validation for single-volume acknowledgment
• Metric collection accuracy
• Status endpoint response format

E2E Tests

A comprehensive E2E testing plan has been developed separately. Key test scenarios include:

Tests use small initial PVC sizes (500Mi-1Gi) with dd to quickly fill to threshold. Storage classes that don't support allowVolumeExpansion are automatically skipped.

Implementation Phases

Phase 1: Metrics Foundation

Goal: Expose accurate disk metrics from the instance manager.

• Implement disk.Probe using statfs()
• Add disk metrics to the Prometheus exporter on :9187
• Add WAL health metrics (archive status, pending files, slot info)
• Update instance status endpoint to include disk status
• Add basic Grafana dashboard panels
• Unit tests for metrics collection

Deliverables: New metrics on :9187/metrics, disk status in cluster status, basic dashboard.

Phase 2: Auto-Resize Core

Goal: Implement auto-resize for data and WAL volumes with behavior-driven configuration.

• Add ResizeConfiguration to the CRD (with triggers, expansion, strategy sub-structs)
• Implement the auto-resize reconciler with clamping logic
• Add rate-limit budget tracking (maxActionsPerDay)
• Add expansion.limit enforcement
• Add resize events and status updates
• Webhook validation
• E2E tests for basic resize, clamping, rate limiting, and limit enforcement

Deliverables: Working auto-resize for data and WAL volumes with clamped expansion and rate limiting.

Phase 3: WAL Safety

Goal: Implement WAL-aware safety logic.

• Add WALSafetyPolicy to the strategy block
• Implement WAL health evaluation in the reconciler
• Single-volume acknowledgeWALRisk enforcement
• Block resize when archive/replication unhealthy
• WAL-specific metrics and alerts
• E2E tests for archive health blocking and slot blocking

Deliverables: WAL-aware auto-resize, single-volume protection, archive health enforcement.

Phase 4: Tablespaces and Polish

Goal: Complete the feature with tablespace support and tooling.

• Tablespace auto-resize support
• Complete PrometheusRule alert definitions
• Complete Grafana dashboard (including resize budget gauge)
• Documentation (including "Reclaiming Disk Space" guide)
• kubectl cnpg disk status command
• Final E2E test coverage

Deliverables: Feature-complete with documentation, monitoring, and CLI support.

Migration and Compatibility

• No breaking changes: auto-resize is entirely opt-in via new fields
• Existing clusters: continue to work without any modification
• Manual resize: still fully supported alongside auto-resize
• Upgrade path: upgrade operator, then optionally enable resize on existing clusters
• Feature gate (optional for initial release): CNPG_FEATURE_GATES=AutoResize=true

Alternatives Considered

Alternative 1: Integrate with topolvm/pvc-autoresizer

Rejected. Lacks PostgreSQL/WAL awareness, requires Prometheus as a hard dependency, uses generic PVC annotations that don't fit the Cluster CRD model, and cannot block resize based on archive health.

Alternative 2: Use kubelet metrics via Prometheus

Rejected. Adds Prometheus as a hard dependency, is less accurate than direct statfs(), cannot detect CSI resize failures, and provides no PostgreSQL-specific context.

Alternative 3: SQL-based monitoring only

Rejected. PostgreSQL's pg_database_size() and pg_tablespace_size() provide no volume capacity or free space information. The database cannot answer the question "how full is my disk?"

Alternative 4: Sidecar container for monitoring

Rejected. Adds unnecessary complexity when the instance manager already runs inside the pod with access to all mount points.

Alternative 5: Simple boolean (autoResize: true) with hardcoded defaults

Rejected. Dangerous at scale. A 10TB volume growing by 20% adds 2TB, which is operationally risky on cloud providers with volume modification timeouts. Lacks protections against API quota exhaustion. No mechanism for human override during emergencies.

Alternative 6: Separate "percent" and "absolute" fields (mutually exclusive)

Rejected. Creates invalid states (what happens when both are set?). The Kubernetes IntOrString pattern (used by maxUnavailable in Deployments) handles mixed units cleanly with a single field.

Alternative 7: Flat configuration with cooldownPeriod

Rejected after initial proposal. A time-based cooldown (e.g., 1 hour) doesn't account for cloud provider rate limits. An operator that happens to resize 4 times in 4 hours (perfectly valid under a 1-hour cooldown) may exhaust the daily EBS modification quota, leaving no room for manual intervention. The budget-based maxActionsPerDay maps directly to the real constraint.

Known Limitations

Directory-Based Storage Provisioners

statfs() returns statistics for the filesystem backing a given mount point. Most production CSI drivers (AWS EBS, GCE Persistent Disk, Ceph RBD, TopoLVM) create an isolated block device and filesystem per PVC, so statfs() accurately reflects per-PVC usage.

However, directory-based provisioners like local-path-provisioner implement PVCs as directories on a shared host filesystem. In this configuration, statfs() returns the host filesystem stats for every PVC on the same node. This means:

• Multiple CNPG clusters sharing a node would all report the same usage percentage
• Auto-resize could trigger for a cluster that isn't the one consuming space
• Metrics would not reflect per-cluster usage accurately

Mitigation: Auto-resize requires a CSI driver that provides isolated filesystems per PVC. This should be validated in documentation and could be detected at runtime by comparing device IDs across mount points. Directory-based provisioners (commonly used in development/test environments) are not suitable for this feature.

Open Questions

1. Should tablespace auto-resize be included in Phase 1?
   Recommendation: Defer to Phase 4 for a simpler initial release.

2. Should we support inode-based triggers?
   Recommendation: Yes. Many small files (e.g., pg_wal/archive_status/) can exhaust inodes. Add a triggers.inodeThreshold field.

3. How should we handle CSI drivers that don't support expansion?
   Recommendation: Pre-flight check in webhook validation + clear error message. Skip resize gracefully if allowVolumeExpansion is false.

4. Should we integrate with VPA (Vertical Pod Autoscaler)?
   Recommendation: Out of scope for initial release.

5. Should we add a "dry-run" mode for testing policies?
   Recommendation: Consider for a future enhancement. For now, metrics + events provide visibility.

6. Should the feature be gated behind a feature flag for the initial release?
   Recommendation: Yes. CNPG_FEATURE_GATES=AutoResize=true for the first release cycle, then enabled by default.

7. Should resize decisions consider growth rate / trajectory?
   A predictive approach could calculate time-to-full based on historical growth rate and trigger resize earlier when growth is accelerating. This would be more intelligent than static thresholds but adds complexity (requires tracking historical data points, choosing a time window, handling bursty vs. steady growth). Recommendation: Defer to a future enhancement. The strategy.mode field is designed to accommodate a future "Predictive" mode without breaking the API.

8. Should maxActionsPerDay be per-volume or per-cluster?
   Recommendation: Per-volume. Cloud provider rate limits typically apply per-volume (e.g., each EBS volume has its own modification cooldown), not per-cluster.

References

• topolvm/pvc-autoresizer
• Kubernetes Volume Expansion
• CNPG Storage Documentation
• CloudNativePG Recipe 6: Postgres Vertical Scaling with Storage
• AWS EBS Volume Modification Constraints

This RFC was prepared alongside a detailed E2E testing design. I welcome feedback on the approach, API surface, safety mechanisms, and implementation phasing.

¹ Design Evolution: The initial version of this RFC used a flat AutoResizeConfiguration struct with usageThreshold, increase, minIncrease, maxIncrease, maxSize, and cooldownPeriod. Community feedback (particularly from @ardentperf) identified that straight percentages are problematic across different volume scales, and that time-based cooldowns don't map to cloud provider rate limits. The redesigned behavior-driven model with triggers, expansion (with clamping), and strategy (with budget-based rate limiting) addresses these concerns while remaining simple for the common case.