OSC Ceph S3 Extension

Introduction

The OSC Ceph S3 Extension provides S3-compatible object storage for Gardener Shoot clusters backed by Ceph RadosGW. It enables users to create S3 users and buckets directly from their Shoot cluster using Kubernetes Custom Resources.

The extension manages the full lifecycle of storage resources: provisioning, credential management, quota enforcement, and deletion with configurable retention policies.

Enabling the Extension

To enable the Ceph S3 extension for a Shoot cluster, add the extension service ceph-s3 to the Shoot manifest:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-shoot
  namespace: garden-my-project
spec:
  extensions:
    - type: ceph-s3

Check if the extension is active:

kubectl get cm -n kube-system shoot-info -o jsonpath='{.data.extensions}'

Disabling the Extension

To disable a globally enabled extension, set disabled: true:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-shoot
  namespace: garden-my-project
spec:
  extensions:
    - type: ceph-s3
      disabled: true

Before you start

kubectl get s3u   # ceph-s3 S3User
kubectl get s3b   # ceph-s3 S3Bucket

Quick Start

Step 1: Create an S3 User

Create a minimal S3User resource:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: my-user
  namespace: default
spec:
  displayName: "My Application User"
  tier: Tier3

Apply the manifest:

$ kubectl apply -f s3user.yaml
s3user.s3.osc.t-systems.com/my-user created

Step 2: Verify the User is Ready

$ kubectl get s3users
NAME      PHASE   CREDENTIALS   SECRET                MESSAGE                                     AGE
my-user   Ready   Transferred   my-user-credentials   User provisioned, credentials transferred   17s

Step 3: Create an S3 Bucket

Create a bucket owned by the user:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: my-bucket
  namespace: default
spec:
  userRef:
    name: my-user

Apply the manifest:

$ kubectl apply -f s3bucket.yaml
s3bucket.s3.osc.t-systems.com/my-bucket created

Step 4: Verify the Bucket is Ready

$ kubectl get s3buckets
NAME        NAMESPACE   PHASE   BUCKETNAME                         VERSIONING   MESSAGE                          AGE
my-bucket   default     Ready   my-bucket-7505d64a-54ef1f-dc99dd   false        Bucket is ready and accessible   6s

Step 5: Access Credentials

The S3 credentials are stored in a Kubernetes Secret referenced by the S3User status:

$ kubectl get s3user my-user -o jsonpath='{.status.secretName}'
my-user-credentials

Retrieve the credentials:

$ kubectl get secret my-user-credentials -o go-template='
ACCESS_KEY_ID: {{ .data.accessKeyID | base64decode }}
SECRET_ACCESS_KEY: {{ .data.secretAccessKey | base64decode }}
ENDPOINT: {{ .data.endpoint | base64decode }}
'

The secret contains the following fields:

accessKeyID

The S3 access key identifier for authentication

secretAccessKey

The S3 secret access key for authentication

endpoint

The S3-compatible endpoint URL for API requests

ca.crt

The CA certificate used to sign the S3 endpoint's TLS certificate. Required when connecting with TLS verification enabled. Present only when the storage backend uses a non-public CA.

Accessing S3 Storage

Use the credentials from the secret with any S3-compatible client. The endpoint uses S3v4 signature and path-style addressing.

$ kubectl get s3bucket my-bucket -o jsonpath='{.status.bucketName}'
my-bucket-7505d64a-54ef1f-dc99dd

Using the AWS CLI

export AWS_ACCESS_KEY_ID=$(kubectl get secret my-user-credentials -o jsonpath='{.data.accessKeyID}' | base64 -d)
export AWS_SECRET_ACCESS_KEY=$(kubectl get secret my-user-credentials -o jsonpath='{.data.secretAccessKey}' | base64 -d)
export S3_ENDPOINT=$(kubectl get secret my-user-credentials -o jsonpath='{.data.endpoint}' | base64 -d)

# Save CA certificate if present
kubectl get secret my-user-credentials -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/s3-ca.crt

# Get the provisioned bucket name
export BUCKET_NAME=$(kubectl get s3bucket my-bucket -o jsonpath='{.status.bucketName}')

# List objects in the bucket
aws s3 ls "s3://$BUCKET_NAME" --endpoint-url "$S3_ENDPOINT" --ca-bundle /tmp/s3-ca.crt

Using the MinIO CLI (mc)

export S3_ENDPOINT=$(kubectl get secret my-user-credentials -o jsonpath='{.data.endpoint}' | base64 -d)
export ACCESS_KEY=$(kubectl get secret my-user-credentials -o jsonpath='{.data.accessKeyID}' | base64 -d)
export SECRET_KEY=$(kubectl get secret my-user-credentials -o jsonpath='{.data.secretAccessKey}' | base64 -d)

# Save CA certificate if present
kubectl get secret my-user-credentials -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.mc/certs/CAs/s3-ca.crt

# Configure alias
mc alias set s3 "$S3_ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY"

# Get the provisioned bucket name
export BUCKET_NAME=$(kubectl get s3bucket my-bucket -o jsonpath='{.status.bucketName}')

# List objects in the bucket
mc ls s3/$BUCKET_NAME

Deleting Resources

To delete a bucket:

kubectl delete s3bucket my-bucket

To delete a user:

kubectl delete s3user my-user

S3 Users

The S3User resource represents an S3 storage identity in your Shoot cluster. Each S3User maps to a dedicated user in the Ceph RadosGW storage backend with its own access credentials, quotas, and lifecycle management.

Creating an S3 User

A minimal S3User requires only a display name. The storage tier is optional — if omitted, the controller applies an environment-derived default (see Storage Tiers):

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: my-user
  namespace: default
spec:
  displayName: "My Application User"
  tier: Tier3

$ kubectl apply -f s3user.yaml
s3user.s3.osc.t-systems.com/my-user created

Full example with all available fields:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: my-user
  namespace: default
spec:
  displayName: "My Application User"
  tier: Tier3
  quotas:
    buckets:
      hard: 10
      soft: 8
  suspended: false
  deletionPolicy: Retain
  retentionExpiresAt: "2026-09-01T00:00:00Z"

Verify the user is ready:

$ kubectl get s3users
NAME      PHASE   CREDENTIALS   SECRET                MESSAGE                                     AGE
my-user   Ready   Transferred   my-user-credentials   User provisioned, credentials transferred   17s

Storage Tiers

The storage tier determines the underlying hardware. The tier is immutable after creation.

tier is optional. If omitted, the controller applies a default: if the region allows exactly one tier, that tier is used; otherwise the operator-configured default tier is used. Check status or spec.tier after creation to see which tier was applied.

Currently supported tiers:

Bucket Quotas

You can limit how many buckets a user is allowed to own using the quotas.buckets configuration:

• hard: Maximum bucket count. Enforced by the storage backend — creating a bucket beyond this limit fails.
• soft: Optional warning threshold. When set, a Kubernetes warning event is emitted, and a status condition is set once usage reaches it (see Quota Warning Events). It does not block operations.

When set, soft must be strictly less than hard; otherwise the resource is rejected on apply.

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: limited-user
  namespace: default
spec:
  displayName: "Limited User"
  tier: Tier3
  quotas:
    buckets:
      hard: 5
      soft: 4

If quotas is omitted, the user inherits the full tenant quota.

Checking the Granted Quota

When a bucket quota is set, the value actually approved by the storage backend appears in status.grantedQuota once the reservation succeeds:

$ kubectl get s3user limited-user -o jsonpath='{.status.grantedQuota}'
{"maxBuckets":5}

The granted value can differ from the requested spec.quotas — for example when you increase the limit but the additional buckets no longer fit into the remaining tenant quota. In that case spec.quotas shows the requested value while status.grantedQuota keeps the previously approved one, so comparing the two tells you whether an increase was actually granted. When no quota is set on the user, status.grantedQuota is absent.

Suspending a User

Setting suspended: true immediately blocks all S3 API requests for this user at the storage backend level. Credentials remain valid and are not changed — access is simply denied.

kubectl patch s3user my-user --type=merge -p '{"spec":{"suspended":true}}'

To restore access:

kubectl patch s3user my-user --type=merge -p '{"spec":{"suspended":false}}'

Use cases for suspension:

• Incident response (immediately revoke access without deleting data)
• Maintenance windows
• Billing holds
• Compliance freezes

The user phase changes to Suspended while suspended and returns to Ready once unsuspended.

Credentials

When an S3User reaches the Ready phase, credentials are automatically delivered to a Kubernetes Secret in the same namespace.

Retrieve the secret name and namespace from the status:

$ kubectl get s3user my-user -o jsonpath='{.status.secretName}'
my-user-credentials

$ kubectl get s3user my-user -o jsonpath='{.status.secretNamespace}'
default

Both status.secretName and status.secretNamespace are set together when credentials are available. Use them to locate the secret precisely, especially when the S3User is in a different namespace than expected.

The secret contains the following keys:

Retrieve credentials:

$ kubectl get secret my-user-credentials -o go-template='
ACCESS_KEY_ID: {{ .data.accessKeyID | base64decode }}
SECRET_ACCESS_KEY: {{ .data.secretAccessKey | base64decode }}
ENDPOINT: {{ .data.endpoint | base64decode }}
'

Endpoint and Certificate Updates

The endpoint and ca.crt values are not fixed for the lifetime of the S3User. When the platform operator changes the storage backend's endpoint or renews its TLS certificate, the updated values are written into your existing credential secret automatically. The secret keeps the same name and the keypair is untouched — this is not a rotation, and no action is required from you.

The configHash key changes whenever endpoint or ca.crt changes, so a workload that needs to react to an update can watch that single value instead of comparing certificates:

kubectl get secret my-user-credentials -o jsonpath='{.data.configHash}' | base64 -d

If the storage backend stops using TLS, or its CA is removed from the configuration, the ca.crt key is deleted from the secret, so a certificate that is no longer in use cannot be trusted by mistake.

You can observe an endpoint or certificate update on the CredentialsConfigChanged condition, which reports ConfigChangeInProgress while the new configuration is being delivered to your secret and ConfigChangeCompleted once it has landed:

kubectl get s3user my-user -o jsonpath='{.status.conditions[?(@.type=="CredentialsConfigChanged")].reason}'

Rotating Credentials

You can rotate an S3User's credentials in place by annotating it. The credential secret keeps the same name; only the accessKeyID and secretAccessKey values change. The endpoint and CA certificate are unaffected.

Request a rotation:

kubectl annotate s3user my-user s3.osc.t-systems.com/rotation-requested=true

Rotation is zero-downtime: the old key stays valid until the new credentials are confirmed delivered to the credential secret (or a configured grace period elapses), so a running workload never loses access during the rotation. Once the new credentials are in the secret, restart or reload any workload that cached the old values (if you use secretKeyRef directly):

kubectl rollout restart deployment/my-app

Track progress with the CredentialsRotation condition and the status.lastRotationTime field:

$ kubectl get s3user my-user -o jsonpath='{.status.conditions[?(@.type=="CredentialsRotation")].reason}'
RotationInProgress

# once the new credentials are delivered to the secret, while the previous key
# stays valid until the rotation policy's revocation gate opens:
RotationSettling

# after completion:
RotationCompleted

During RotationSettling the secret already holds the new credentials; the condition message names what the rotation is still waiting for before revoking the previous key (the settle window, or your confirmation under the manual policy).

kubectl get s3user my-user -o jsonpath='{.status.conditions[?(@.type=="CredentialsRotation")].message}'

When the rotation completes, the s3.osc.t-systems.com/rotation-requested annotation is removed automatically, s3.osc.t-systems.com/last-rotation-at records the completion time, and status.lastRotationTime is updated. To rotate again, re-add the annotation.

Rotation policy

The s3.osc.t-systems.com/rotation-policy annotation selects how the previous key is revoked once the new credentials are delivered. Set it alongside the rotation request:

kubectl annotate s3user my-user s3.osc.t-systems.com/rotation-policy=manual
kubectl annotate s3user my-user s3.osc.t-systems.com/rotation-requested=true

auto (default)

The previous key is revoked after a settle window elapses following delivery of the new credentials. The CredentialsRotation condition message names the settle period.

manual

The previous key stays valid until you confirm that your workloads have adopted the new key. Confirm by passing the new access key ID:

kubectl annotate s3user my-user s3.osc.t-systems.com/rotation-confirm=<new-accessKeyID>

The confirmation is scoped to the access key ID, so a confirmation recorded for an earlier rotation can never silently confirm a later one.

If the confirmation names a key that is not the newly published one, the rotation does not complete: the CredentialsRotation condition reports RotationConfirmMismatch and its message names the value you applied. The expected key is deliberately not included — the confirmation proves the rotated credentials actually arrived in your secret. Read the accessKeyID from your credential secret, verify your workloads use it, and re-apply the annotation with that value.

forced

The previous key is revoked as soon as delivery of the new credentials is confirmed.

The rotation-policy and rotation-confirm annotations only take effect together with a rotation request: set alone, they stay on the S3User and are not acted on until you also set s3.osc.t-systems.com/rotation-requested=true. When the rotation completes, both annotations are removed automatically along with the request — each rotation chooses its policy explicitly, so a manual or forced choice from an earlier rotation is never silently reused by the next one.

If a rotation makes no visible progress, check the CredentialsRotation condition: it reports RotationStalled when new credentials are waiting on a delivery or adoption confirmation, and RotationRejected when the request named an unknown rotation policy.

Credential Lifecycle Events

In addition to the conditions above, the S3User records a Kubernetes Event the moment its CredentialsRotation/CredentialsConfigChanged condition transitions, giving you a timestamped history in kubectl describe. Each event fires exactly when the corresponding condition reaches that state:

$ kubectl describe s3user my-user
...
Events:
  Type    Reason              Age    From                      Message
  Normal  CredentialsRotated  15m    ceph-s3-shoot-controller  Credentials rotated: the credential secret now holds the new access key and secret key
  Normal  CredentialsRevoked  10s    ceph-s3-shoot-controller  Previous credentials revoked: only the current key in the secret is valid

Under the auto and manual policies the two rotation events are separated by the settle window (the default is 15 minutes for auto): CredentialsRotated fires when the new key lands in the secret, and CredentialsRevoked fires later when the old key is revoked. The forced policy revokes immediately without a settle window, so there is no RotationSettling state — only CredentialsRevoked is recorded.

Deleting a User

kubectl delete s3user my-user

To configure immediate deletion without retention:

spec:
  deletionPolicy: Delete

To schedule automatic purge after a date:

spec:
  deletionPolicy: Retain
  retentionExpiresAt: "2027-01-01T00:00:00Z"

When set, the value is mirrored to status.retentionExpiresAt so you can always read the active expiry from the resource status:

$ kubectl get s3user my-user -o jsonpath='{.status.retentionExpiresAt}'
2027-01-01T00:00:00Z

If retentionExpiresAt is omitted with Retain policy, the resource enters PendingPurge state with a default 7-day grace period from the deletion date. The computed expiry date is shown in status.message. You can also apply the force-delete annotation to purge immediately without waiting.

S3 Buckets

The S3Bucket resource represents an S3 storage bucket in your Shoot cluster. Each bucket is owned by an S3User and can optionally grant access to additional users with configurable permissions.

Creating a Bucket

A minimal bucket requires only a reference to its owner:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: my-bucket
  namespace: default
spec:
  userRef:
    name: my-user

$ kubectl apply -f s3bucket.yaml
s3bucket.s3.osc.t-systems.com/my-bucket created

Full example with all available fields:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: my-bucket
  namespace: default
spec:
  userRef:
    name: my-user
    namespace: other-namespace
  additionalUsers:
    - userRef:
        name: reader-user
      permissions: ReadOnly
    - userRef:
        name: writer-user
        namespace: other-namespace
      permissions: ReadWrite
  deletionPolicy: Retain
  retentionExpiresAt: "2026-09-01T00:00:00Z"
  quotas:
    storage:
      hard: "100Gi"
      soft: "80Gi"
    objects:
      hard: 1000000
      soft: 800000

Requirements:

• The referenced owner (userRef) must exist and be in Ready phase
• If you use additionalUsers, all users (owner and additional) must share the same storage tier

Verify the bucket is ready:

$ kubectl get s3buckets
NAME        NAMESPACE   PHASE   BUCKETNAME                         VERSIONING   MESSAGE                          AGE
my-bucket   default     Ready   my-bucket-7505d64a-54ef1f-dc99dd   false        Bucket is ready and accessible   6s

Sharing Access with Additional Users

You can grant other S3Users access to a bucket with configurable permission levels:

graph LR
    subgraph S3 Bucket
        B[("my-bucket")]
    end

    subgraph S3 Users
        O["owner-user"]
        R["reader-user"]
        W["writer-user"]
        U["uploader-user"]
    end

    O -- "FullAccess (owner)" --> B
    R -- "ReadOnly" --> B
    W -- "ReadWrite" --> B
    U -- "WriteOnly" --> B

Permission Levels

Example: Shared Bucket

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: shared-bucket
  namespace: default
spec:
  userRef:
    name: owner-user
  additionalUsers:
    - userRef:
        name: reader-user
      permissions: ReadOnly
    - userRef:
        name: writer-user
      permissions: ReadWrite
    - userRef:
        name: uploader-user
      permissions: WriteOnly

Validation Rules

• The bucket owner cannot be listed as an additional user (owner already has full access)
• All additional users must be in the same tier as the bucket owner
• All referenced users must exist
• User names cannot be empty

Adding and Removing Users

Add an additional user:

kubectl patch s3bucket shared-bucket --type=json -p '[
  {"op": "add", "path": "/spec/additionalUsers/-", "value": {
    "userRef": {"name": "new-reader"},
    "permissions": "ReadOnly"
  }}
]'

Remove an additional user (by index):

kubectl patch s3bucket shared-bucket --type=json -p '[
  {"op": "remove", "path": "/spec/additionalUsers/0"}
]'

Cross-Namespace Ownership

Buckets can reference an S3User in a different namespace. This enables multi-team setups where a central user owns storage used across namespaces:

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: team-bucket
  namespace: team-b
spec:
  userRef:
    name: shared-user
    namespace: team-a
  additionalUsers:
    - userRef:
        name: team-b-user
        namespace: team-b
      permissions: ReadWrite

If namespace is omitted in userRef or additionalUsers[].userRef, it defaults to the bucket's own namespace.

Storage and Object Quotas

You can limit how much storage a bucket consumes and how many objects it can hold:

• hard: Maximum limit. Enforced by the storage backend — operations fail when the limit is reached.
• soft: Optional warning threshold. When set, a Kubernetes warning event is emitted, and a status condition is set once usage reaches it (see Quota Warning Events). It does not block operations.

When set, soft must be strictly less than hard; otherwise the resource is rejected on apply.

Storage Quota

Uses Kubernetes resource quantity format (e.g., "10Gi", "500Mi", "1Ti"):

spec:
  quotas:
    storage:
      hard: "50Gi"
      soft: "40Gi"

Object Count Quota

Limits the total number of objects in the bucket:

spec:
  quotas:
    objects:
      hard: 500000
      soft: 400000

Full Quota Example

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: limited-bucket
  namespace: default
spec:
  userRef:
    name: my-user
  quotas:
    storage:
      hard: "100Gi"
      soft: "80Gi"
    objects:
      hard: 1000000
      soft: 800000

If quotas is omitted, the bucket inherits the full tenant quota.

Checking the Granted Quota

When storage or object quotas are set, the values actually approved by the storage backend appear in status.grantedQuota once the reservation succeeds:

$ kubectl get s3bucket limited-bucket -o jsonpath='{.status.grantedQuota}'
{"maxSize":"100Gi","maxObjects":1000000}

The granted values can differ from the requested spec.quotas — for example when you increase a limit but the additional capacity no longer fits into the remaining tenant quota. In that case spec.quotas shows the requested values while status.grantedQuota keeps the previously approved ones, so comparing the two tells you whether an increase was actually granted. When no quota is set on the bucket, status.grantedQuota is absent.

Bucket Name

The actual bucket name in the storage backend differs from the Kubernetes resource name. This is by design for global uniqueness and the zero-trust security model.

Retrieve the provisioned bucket name:

$ kubectl get s3bucket my-bucket -o jsonpath='{.status.bucketName}'
my-bucket-7505d64a-54ef1f-dc99dd

Deleting a Bucket

kubectl delete s3bucket my-bucket

To configure immediate deletion (bucket and all its data are permanently removed):

spec:
  deletionPolicy: Delete

To schedule automatic purge after a date:

spec:
  deletionPolicy: Retain
  retentionExpiresAt: "2027-01-01T00:00:00Z"

When set, the value is mirrored to status.retentionExpiresAt so you can always read the active expiry from the resource status:

$ kubectl get s3bucket my-bucket -o jsonpath='{.status.retentionExpiresAt}'
2027-01-01T00:00:00Z

If retentionExpiresAt is omitted with Retain policy, the resource enters PendingPurge state with a default 30-days grace period from the deletion date. The computed expiry date is shown in status.message. You can also apply the force-delete annotation to purge immediately without waiting.

Bucket Versioning

You can manage versioning for your S3 bucket using MinIO Client (mc) via a temporary pod.

Checking Versioning Status via Kubernetes

The current versioning state of your bucket is automatically reflected in the S3Bucket resource status. You can check it without connecting directly to the S3 endpoint.

kubectl get s3buckets.s3.osc.t-systems.com -o wide

The output looks like this:

NAME       NAMESPACE     PHASE   BUCKETNAME                                 VERSIONING   MESSAGE                          AGE
mybucket   kube-system   Ready   mybucket-e853ff73-64c8c7-7a10dd            false        Bucket is ready and accessible   15m

Managing Versioning

Enable versioning:

mc version enable mybucket/$BUCKET
mybucket/mybucket-e853ff73-64c8c7-7a10dd versioning is enabled

Once the controller reconciles the S3Bucket, the status.versioningEnabled field reflects the actual versioning state:

NAME       NAMESPACE     PHASE   BUCKETNAME                                 VERSIONING   MESSAGE                          AGE
mybucket   kube-system   Ready   mybucket-e853ff73-64c8c7-7a10dd            true         Bucket is ready and accessible   20m

Suspend versioning:

mc version suspend mybucket/$BUCKET
mybucket/mybucket-e853ff73-64c8c7-7a10dd versioning is suspended

Check the current versioning status from API:

mc version info mybucket/$BUCKET
mybucket/mybucket-e853ff73-64c8c7-7a10dd versioning is suspended

Bucket Encryption

Currently, S3Bucket supports only SSE-C (Server-Side Encryption with Customer-Provided Keys), which enables object encryption using keys managed by the user.

SSE-C Customer-Provided Keys

This mode is implemented according to the Amazon SSE-C specification.

Considerations Before Using SSE-C

• Downloading SSE-C encrypted data requires the same encryption key that was used during upload.
• A reliable mapping between objects and encryption keys is required. If a key is lost, access to the corresponding object is lost.
• Because key management is client-side, safeguards such as key rotation are also client-side responsibilities.
• HTTPS is required for all requests.
• In versioning-enabled buckets, each object version can use a different encryption key, so key-to-object-version mapping must be tracked.

Prerequisites

• mc (MinIO Client) configured with your bucket alias or the aws CLI
• openssl available on your system

Usage

1. Generate an encryption key:

   KEY=$(openssl rand -hex 32)
   echo $KEY
   

2. Upload a file using a key:

   mc cp file.txt myminio/mybucket/file.txt \
     --enc-c "myminio/mybucket=$KEY"
   

3. Download a file:

   You must supply the same key used during upload:

   mc cp myminio/mybucket/file.txt downloaded.txt \
     --enc-c "myminio/mybucket=$KEY"
   

   Attempting to download without the key returns a 400 Bad Request error:

   mc cp myminio/mybucket/file.txt downloaded.txt
   mc: <ERROR> Unable to prepare URL for copying. 400 Bad Request
   

4. Verify object metadata:

   mc stat myminio/mybucket/file.txt \
     --enc-c "myminio/mybucket=$KEY"
   

   The output confirms the encryption mode:

   Name      : file.txt
   Date      : 2026-05-27 11:58:04 UTC
   Size      : 0 B
   ETag      : d41d8cd98f00b204e9800998ecf8427e
   Type      : file
   Encryption: SSE-C
   Metadata  :
     Content-Type: text/plain
   

Per-Object Keys

SSE-C supports a different key per object (and per version):

KEY_A=$(openssl rand -hex 32)
KEY_B=$(openssl rand -hex 32)

touch log.A
touch log.B

mc cp log.A   myminio/mybucket/report.pdf   --enc-c "myminio/mybucket=$KEY_A"
mc cp log.B   myminio/mybucket/secrets.json --enc-c "myminio/mybucket=$KEY_B"

Cross-Versioning Example

When bucket versioning is enabled, each new upload to the same object path creates a new object version. With SSE-C, each version can be encrypted with a different key.

1. Ensure bucket versioning is enabled:

   mc version enable myminio/mybucket
   

2. Create version-specific keys:

   KEY_V1=$(openssl rand -hex 32)
   KEY_V2=$(openssl rand -hex 32)
   

3. Upload version 1 of the same object path:

   echo '{"revision":"v1"}' > app-config.json
   mc cp app-config.json myminio/mybucket/app-config.json \
     --enc-c "myminio/mybucket=$KEY_V1"
   

4. Upload version 2 of the same object path using a different key:

   echo '{"revision":"v2"}' > app-config.json
   mc cp app-config.json myminio/mybucket/app-config.json \
     --enc-c "myminio/mybucket=$KEY_V2"
   

5. List object versions and collect the version IDs:

   mc ls --versions myminio/mybucket/app-config.json
   [2026-05-27 14:08:06 CEST]    18B STANDARD WGcpjBHX2VYz0C3WvhZtFwD5hZ219kx v2 PUT app-config.json
   [2026-05-27 14:07:22 CEST]    18B STANDARD 48clq5r7UzO1Yd7JysbFlhAU3DItguH v1 PUT app-config.json
   

Download each version with its matching key:

# Replace with actual IDs from: mc ls --versions myminio/mybucket/app-config.json
export VERSION_ID_V1="<version-id-v1>"
export VERSION_ID_V2="<version-id-v2>"

mc cp --version-id "$VERSION_ID_V1" myminio/mybucket/app-config.json app-config-v1.json \
  --enc-c "myminio/mybucket=$KEY_V1"

mc cp --version-id "$VERSION_ID_V2" myminio/mybucket/app-config.json app-config-v2.json \
  --enc-c "myminio/mybucket=$KEY_V2"

Verify:

cat app-config-v1.json
cat app-config-v2.json

If an incorrect key is used, the request returns HTTP 400 Bad Request:

mc cp --version-id "$VERSION_ID_V2" myminio/mybucket/app-config.json app-config-v2.json \
  --enc-c "myminio/mybucket=$KEY_V1"
  mc: <DEBUG> HTTP/1.1 400 Bad Request

Expected behavior:

• A version can only be decrypted with the key used for that version.
• Using the wrong key (or no key) for a specific version fails with a request error.
• Key-to-version mapping must be preserved, not only key-to-object mapping.

Key Rotation for SSE-C

SSE-C does not support in-place key rotation. To change the encryption key for an object, download it with the current key and re-upload it with the new key. When versioning is enabled, each version is independently encrypted — old versions remain accessible only with their original key, so either retain the old keys or re-upload the versions you need.

Key rotation using the mc CLI:

1. Set variables

   ALIAS="kind"
   BUCKET="cu-0000000000-bucket"
   OBJECT="file.txt"
   TEMP_FILE="/tmp/rotation-temp-$(date +%s)"
   

2. Generate keys

   OLD_KEY=$(openssl rand -hex 32)
   NEW_KEY=$(openssl rand -hex 32)
   echo "OLD_KEY: $OLD_KEY"
   echo "NEW_KEY: $NEW_KEY"
   

3. Create and upload test file with OLD_KEY

   echo "hello rotation test" > localfile.txt
   mc cp \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$OLD_KEY" \
     localfile.txt \
     "$ALIAS/$BUCKET/$OBJECT"
   

4. Verify OLD_KEY works

   mc stat \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$OLD_KEY" \
     "$ALIAS/$BUCKET/$OBJECT"
   

5. Download to a local temp file using OLD_KEY

   Ceph RGW does not support SSE-C on CopyObject (501 NotImplemented), so key rotation requires a local download first.

   mc cp \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$OLD_KEY" \
     "$ALIAS/$BUCKET/$OBJECT" \
     "$TEMP_FILE"
   

6. Re-upload from local temp file with NEW_KEY

   mc cp \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$NEW_KEY" \
     "$TEMP_FILE" \
     "$ALIAS/$BUCKET/$OBJECT"
   

7. Remove local temp file

   rm "$TEMP_FILE"
   

8. Verify NEW_KEY works

   mc stat \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$NEW_KEY" \
     "$ALIAS/$BUCKET/$OBJECT"
   

9. Verify OLD_KEY is rejected

   mc stat \
     --enc-c "$ALIAS/$BUCKET/$OBJECT=$OLD_KEY" \
     "$ALIAS/$BUCKET/$OBJECT" \
     && echo "OLD_KEY still works!"
   

Using the REST API and AWS SDKs

The S3Bucket endpoint is compatible with the AWS SSE-C request model, so SSE-C can be implemented through direct REST API calls or AWS SDKs, with key management handled directly in your application.

Required SSE-C Request Headers

For SSE-C encryption/decryption requests, include the following headers:

• x-amz-server-side-encryption-customer-algorithm (must be AES256)
• x-amz-server-side-encryption-customer-key (base64-encoded 256-bit key)
• x-amz-server-side-encryption-customer-key-MD5 (base64-encoded MD5 of the key)

For copy operations where the source object is SSE-C encrypted, include source decryption headers as well:

• x-amz-copy-source-server-side-encryption-customer-algorithm
• x-amz-copy-source-server-side-encryption-customer-key
• x-amz-copy-source-server-side-encryption-customer-key-MD5

SDK and Multipart Notes

• SDKs can attach SSE-C information to PUT, GET, HEAD, and COPY requests.
• For multipart uploads, encryption settings must be provided consistently across multipart requests.
• The same key used for upload must be provided for subsequent download/metadata operations.
• If the key is incorrect or missing, the request fails (400 Bad Request).
• TLS required: non-HTTPS requests will be rejected with an InvalidRequest error.

Go SDK Example

The following snippets demonstrate SSE-C key management using the AWS SDK for Go v2. These are usage examples only — production implementations should add proper error handling, key persistence, and secrets management.

Key generation and encoding

Every SSE-C request requires three values derived from the raw key: the base64-encoded key, its base64-encoded MD5 checksum, and the algorithm name AES256.

import (
    "crypto/md5"
    "crypto/rand"
    "encoding/base64"
)

func generateSSECKey() ([]byte, error) {
    key := make([]byte, 32) // 256-bit AES key
    _, err := rand.Read(key)
    return key, err
}

func encodeKey(key []byte) (keyB64 string, keyMD5 string) {
    keyB64 = base64.StdEncoding.EncodeToString(key)
    md5sum := md5.Sum(key)
    keyMD5 = base64.StdEncoding.EncodeToString(md5sum[:])
    return
}

Upload (PutObject)

keyB64, keyMD5 := encodeKey(key)

_, err := client.PutObject(ctx, &s3.PutObjectInput{
    Bucket:               aws.String(bucketName),
    Key:                  aws.String(objectKey),
    Body:                 strings.NewReader(content),
    ContentLength:        aws.Int64(int64(len(content))),
    SSECustomerAlgorithm: aws.String("AES256"),
    SSECustomerKey:       aws.String(keyB64),
    SSECustomerKeyMD5:    aws.String(keyMD5),
})

Download (GetObject)

The same three SSE-C fields are required on every read request:

keyB64, keyMD5 := encodeKey(key)

resp, err := client.GetObject(ctx, &s3.GetObjectInput{
    Bucket:               aws.String(bucketName),
    Key:                  aws.String(objectKey),
    SSECustomerAlgorithm: aws.String("AES256"),
    SSECustomerKey:       aws.String(keyB64),
    SSECustomerKeyMD5:    aws.String(keyMD5),
})

Head (metadata only)

resp, err := client.HeadObject(ctx, &s3.HeadObjectInput{
    Bucket:               aws.String(bucketName),
    Key:                  aws.String(objectKey),
    SSECustomerAlgorithm: aws.String("AES256"),
    SSECustomerKey:       aws.String(keyB64),
    SSECustomerKeyMD5:    aws.String(keyMD5),
})

Key rotation for SSE-C

The workaround is to download the object with the current key and re-upload it with the new key:

1. Download with the source key:

   data, err := getObjectBytes(ctx, client, srcKey, `<oldKey>`)
   

2. Re-upload with a new key:

   dstKeyB64, dstKeyMD5 := encodeKey(dstKey)
   _, err = client.PutObject(ctx, &s3.PutObjectInput{
       Bucket:               aws.String(bucketName),
       Key:                  aws.String(`<newKey>`),
       Body:                 bytes.NewReader(data),
       ContentLength:        aws.Int64(int64(len(data))),
       SSECustomerAlgorithm: aws.String("AES256"),
       SSECustomerKey:       aws.String(dstKeyB64),
       SSECustomerKeyMD5:    aws.String(dstKeyMD5),
   })
   

S3 client setup (Ceph-specific options)

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
    o.BaseEndpoint = aws.String(endpoint)
    o.UsePathStyle = true  // required for Ceph
    o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
    o.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
})

Other SDK References

For Java, .NET, and other SDKs, see the official AWS documentation:

• Specifying SSE-C (AWS User Guide)
• Using AWS SDKs for SSE-C operations
• Using AWS SDKs for SSE-C multipart uploads

Deletion and Retention

The Ceph S3 extension provides configurable deletion policies and retention mechanisms to protect against accidental data loss. By default, all resources use the Retain policy, which blocks deletion until a retention expiry date is reached (either user-specified or grace period default). You can customize retention by setting spec.retentionExpiresAt, cancel a pending deletion with the s3.osc.t-systems.com/recovery-requested=true annotation, or bypass retention entirely with the s3.osc.t-systems.com/force-delete=true annotation.

Deletion Policies

Retain (Default)

The Retain policy blocks deletion until a retention expiry date is reached. The retention period ends when one of the following conditions is met:

• The retentionExpiresAt timestamp is reached (user-specified or default period)
• The s3.osc.t-systems.com/force-delete=true annotation is applied

When a Retain-policy resource is deleted, it enters the Retention phase. During retention:

• The resource remains visible (kubectl get s3users / kubectl get s3buckets)
• The resource has a DeletionTimestamp but is not yet removed
• Data remains accessible until the retention period expires
• The resource cannot be recreated with the same name until purged
• Retention always has an expiry date (either user-specified or backend default)
• The deletion can be canceled with the recovery-requested annotation (see Recovery)

• S3User: The user is suspended — S3 access is blocked, credentials stop working
• S3Bucket: Data remains accessible via other valid credentials until the resource is purged

spec:
  deletionPolicy: Retain
  retentionExpiresAt: "2027-01-01T00:00:00Z"  # optional: auto-purge after this date

Delete

The Delete policy removes the resource immediately with no grace period.

• S3User: User identity and credentials are permanently removed
• S3Bucket: Bucket and all its data are permanently removed, regardless of content

spec:
  deletionPolicy: Delete

Retention State Machine

When a resource with Retain policy is deleted, it transitions through the following states:

stateDiagram-v2
    [*] --> None: Resource created
    None --> PendingPurge: kubectl delete (Retain policy)
    PendingPurge --> None: recovery-requested annotation
    PendingPurge --> Purging: retentionExpiresAt reached
    PendingPurge --> Purging: force-delete annotation
    Purging --> [*]: Kubernetes object deleted
    None --> [*]: kubectl delete (Delete policy)

Retention States

None

Normal operation. Resource is not in any retention lifecycle.

PendingPurge

Deletion was requested. The resource is blocked from permanent removal until the retention expiry date is reached (either user-specified via spec.retentionExpiresAt or storage backend default synced to status.retentionExpiresAt). The recovery-requested annotation can cancel the deletion, and the force-delete annotation can bypass retention for immediate purge.

Purging

The retention expiry date has been reached or the force-delete annotation was applied. The resource data is being removed from storage backend and the Kubernetes object will be deleted shortly.

Recovery: Canceling a Pending Deletion

While a resource is still inside its retention window (status.retentionState: PendingPurge), cancel the deletion by applying the recovery-requested annotation:

kubectl annotate s3user my-user s3.osc.t-systems.com/recovery-requested=true

kubectl annotate s3bucket my-bucket s3.osc.t-systems.com/recovery-requested=true

The resource converges back to the Ready phase within a few seconds, with the same name, spec, and underlying storage data:

• S3User: The suspension is lifted — S3 access works again with the same credentials; the credential secret is re-published
• S3Bucket: The bucket and all its contents remain exactly as they were before deletion

Only the Kubernetes UID of the resource changes, because a deletion in progress cannot be undone in place — the resource is recreated with an identical spec.

Requirements:

• The resource must be in status.retentionState: PendingPurge (inside the retention window)
• The annotation value must be exactly "true"

Force Delete

The force-delete annotation bypasses all retention policies and triggers immediate purge.

kubectl annotate s3user my-user s3.osc.t-systems.com/force-delete=true

kubectl annotate s3bucket my-bucket s3.osc.t-systems.com/force-delete=true

Requirements:

• The resource must already have a DeletionTimestamp set (i.e., kubectl delete was already issued)
• The annotation value must be exactly "true"

Workflow:

1. Delete the resource (enters retention): kubectl delete s3user my-user
2. Verify the resource is in retention: kubectl get s3user my-user -o jsonpath='{.status.retentionState}'
3. Apply force-delete to bypass retention: kubectl annotate s3user my-user s3.osc.t-systems.com/force-delete=true

Events

The system emits Kubernetes events during deletion lifecycle transitions:

Each event is emitted only once, on the transition into that state — not on every reconcile loop.

View events:

$ kubectl describe s3user my-user
Events:
  Type     Reason         Age   Message
  ----     ------         ----  -------
  Normal   PendingPurge   5m    Deletion blocked: active retention window, awaiting expiry
  Warning  RetentionExpired  1s Retention period expired: proceeding with deletion

Practical Examples

Standard Deletion with Retain Policy

# Create user with Retain policy and for e.g. 90-day retention period
$ cat <<EOF | kubectl apply -f -
apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: compliance-user
  namespace: default
spec:
  displayName: "Compliance User"
  deletionPolicy: Retain
  retentionExpiresAt: "2027-03-15T00:00:00Z"
  tier: Tier3
EOF

# Delete the user (enters retention blocked state)
$ kubectl delete s3user compliance-user
s3user.s3.osc.t-systems.com "compliance-user" deleted

# Check retention state (blocked until expiry)
$ kubectl get s3user compliance-user -o jsonpath='{.status.retentionState}'
PendingPurge

#Verify status
# kubectl get s3user compliance-user 
  ...
  message: 'User soft-deleted, retention expires 2027-03-15T00:00:00Z'
  phase: Retention
  retentionExpiresAt: "2027-03-15T00:00:00Z"
  retentionState: PendingPurge
  suspended: true

# Option 1: Wait for retentionExpiresAt to be reached (automatic purge)
# Option 2: Force-delete to bypass retention
$ kubectl annotate s3user compliance-user s3.osc.t-systems.com/force-delete=true --overwrite
# Option 3: Cancel the deletion and recover the user
$ kubectl annotate s3user compliance-user s3.osc.t-systems.com/recovery-requested=true

Recovering an Accidentally Deleted User

# The user was deleted and entered retention
$ kubectl get s3user compliance-user -o jsonpath='{.status.retentionState}'
PendingPurge

# Cancel the deletion
$ kubectl annotate s3user compliance-user s3.osc.t-systems.com/recovery-requested=true
s3user.s3.osc.t-systems.com/compliance-user annotated

# The user converges back to Ready within a few seconds
$ kubectl get s3user compliance-user
NAME              PHASE   AGE
compliance-user   Ready   12s

Immediate Deletion with Delete Policy

# Create user with Delete policy
$ cat <<EOF | kubectl apply -f -
apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: temp-user
  namespace: default
spec:
  displayName: "Temporary User"
  tier: Tier3
  deletionPolicy: Delete
EOF

# Delete immediately (no retention)
$ kubectl delete s3user temp-user
s3user.s3.osc.t-systems.com "temp-user" deleted

Timed Retention with User-Specified Expiry

# Create user with 90-day retention window
$ cat <<EOF | kubectl apply -f -
apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: audit-user
  namespace: default
spec:
  displayName: "Audit User"
  tier: Tier3
  deletionPolicy: Retain
  retentionExpiresAt: "2026-08-25T00:00:00Z"
EOF

# Delete the user (enters PendingPurge state, auto-purges on 2026-08-25)
$ kubectl delete s3user audit-user

Bucket Deletion

# With Retain policy (default), bucket enters retention
$ kubectl delete s3bucket app-data

# With Delete policy, bucket and all data are removed immediately
$ kubectl delete s3bucket app-data
s3bucket.s3.osc.t-systems.com "app-data" deleted

S3User Deletion with Dependent Buckets

Before deleting an S3User, ensure no S3Buckets reference it as owner (.spec.userRef). If buckets still reference the user:

• The deletion proceeds but dependent buckets will enter an Error phase because the referenced user no longer exists
• The bucket itself remains functional (data accessible via other valid credentials)
• Remove or reassign buckets before deleting their owner to avoid the error state

Shoot Cluster Deletion

To preserve data before Shoot deletion:

1. Migrate data to another Shoot's S3 bucket using standard S3 tools (aws s3 sync, mc mirror)
2. Ensure credentials for both source and destination buckets are available
3. Delete the Shoot only after confirming data migration is complete

Migrate Data from MinIO to Ceph S3

If your application is still using the legacy MinIO-backed S3 extension, you can migrate its object data to a new Ceph S3 bucket using standard S3 tooling such as the MinIO Client (mc) or the AWS CLI.

Prerequisites

• The ceph-s3 extension enabled on the Shoot cluster — see Enabling the Extension
• A new S3Bucket created and in Ready phase

Migrate with the MinIO Client (mc)

Configure aliases for both endpoints and mirror the bucket. mc mirror copies only objects that are missing or changed in the destination, so it is safe to re-run.

mc alias set source <s3_endpoint> <s3_access_key_id> <s3_secret_access_key>
mc alias set dest <endpoint> <accessKeyID> <secretAccessKey>
mc mirror source/<old-bucket> dest/<new-bucket>

Migrate with the AWS CLI

aws s3 sync performs the same incremental copy between two endpoints.

$ AWS_ACCESS_KEY_ID=<s3_access_key_id> AWS_SECRET_ACCESS_KEY=<s3_secret_access_key> \
    aws s3 sync s3://<old-bucket> ./migration-tmp --endpoint-url <s3_endpoint>

$ AWS_ACCESS_KEY_ID=<accessKeyID> AWS_SECRET_ACCESS_KEY=<secretAccessKey> \
    aws s3 sync ./migration-tmp s3://<new-bucket> --endpoint-url <endpoint>

If both endpoints are reachable from the same location, you can also sync directly without the intermediate directory.

Update Your Application

Once the data is copied, point your workload at the new S3Bucket.

Tenant Quotas

Tenant quota is the global capacity cap for a tenant. It defines the maximum resources your tenant can consume and cannot exceed. When you create your first resource, your tenant is granted a default quota.

What Tenant Quota Applies To

Tenant quota is applied to:

• Storage (always enforced)
• Buckets (optional, depending on configured limits)
• Objects (optional, depending on configured limits)
• Users (optional, depending on configured limits)

For details on which dimensions are actively enforced for your tenant, see Quota Enforcement.

Shared Quota Across Shoots

All Shoot clusters of a tenant on the same Seed consume from the same quota pool. Usage from any Shoot reduces what remains available for all others.

Quota Inheritance

If spec.quotas is omitted, the resource inherits the full currently available tenant quota. If spec.quotas is set, the resource is constrained to those limits, which should be lower than or equal to the available tenant quota.

Hard Limit Enforcement

Tenant quota is a hard limit. When usage reaches the limit, write operations are blocked.

Typical effects when the limit is reached:

• Uploads fail
• Creating new S3Users or S3Buckets fails
• Object creation fails
• Increasing per-resource quotas beyond the tenant cap fails

This applies to the tenant cap and to any stricter per-resource hard quota.

If you reach your tenant quota:

1. Delete unnecessary objects, buckets, or users to free capacity
2. If more capacity is required, contact support to request a tenant quota increase

Viewing Usage Per Resource

Users can see usage per resource in status.usage.

Check S3User usage:

kubectl get s3user my-user -o jsonpath='{.status.usage}'

Check S3Bucket usage:

kubectl get s3bucket my-bucket -o jsonpath='{.status.usage}'

Example output:

{
  "buckets": 0,
  "lastUpdated": "2026-05-29T08:46:13Z",
  "objects": 0,
  "storage": "0"
}

Viewing Tenant-Wide Quota

The S3TenantQuota resource provides a read-only, tenant-visible snapshot of your overall quota state. It has no spec — all information is in .status.

There is one S3TenantQuota per tier. If your tenant uses multiple tiers, you will see resource per each tier: tier3-quota, tierXY-quota, ...

The resource is deployed into kube-system by default. Depending on your cluster operator's configuration, it may also appear in the default namespace. Use -A if you are unsure which namespace is used:

kubectl get s3tenantquota -A

You can also use the short names s3tq or s3tqs:

kubectl get s3tq -A

Example output:

NAMESPACE     NAME          PHASE   TIER    OVER ALLOCATED   GRANTED STORAGE   CONSUMED STORAGE   REMAINING STORAGE   AGE
kube-system   tier3-quota   Ready   Tier3   false            2Ti               600Gi              1448Gi              10d

To inspect the full status for a specific tier:

kubectl describe s3tq tier3-quota -n kube-system

Or as YAML:

kubectl get s3tq tier3-quota -n kube-system -o yaml

Example status block:

status:
  lastSyncedAt: "2026-06-23T10:00:00Z"
  overAllocated: false
  phase: Ready
  quotaAssigned:
    buckets: 5
    objects: 0
    storage: 200Gi
    users: 3
  quotaConsumed:
    buckets: 2
    objects: 0
    storage: 120Gi
    users: 2
  quotaGranted:
    buckets: 500
    objects: 0
    storage: 500Gi
    users: 200
  quotaRemaining:
    buckets: 495
    objects: 0
    storage: 300Gi
    users: 197
  quotasEnforced:
    buckets: true
    objects: false
    storage: true
    users: true

Status Fields

Quota Enforcement

The quotasEnforced field shows which quota dimensions are actively enforced for your tenant. An enforced dimension blocks operations when its limit is reached — for example, uploads fail when the storage limit is hit. A dimension that is not enforced is tracked in the status but does not block operations when exceeded.

Enforcement is configured by the cluster operator and may differ between tenants. To check which dimensions apply to your tenant:

kubectl get s3tq -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.quotasEnforced}{"\n"}{end}'

Example output:

tier3-quota  {"buckets":true,"objects":false,"storage":true,"users":true}

In this example, storage, users, and buckets are enforced. The objects dimension is tracked but not enforced — exceeding an object count limit does not block operations.

Quota Relationship Diagram

The diagram below shows a practical example with a 1Ti tenant cap split across two Shoots. One bucket has an explicit storage quota; the other inherits the full remaining tenant capacity.

flowchart TD
    TQ["Tenant Quota Cap
    Limits: 1Ti storage, 10 users, 20 buckets
    Usage: 720Gi storage, 2 users, 2 buckets"]

    TQ --> ShootA
    TQ --> ShootB

    subgraph ShootA ["Shoot Cluster A"]
        direction TB
        UA["S3User: app-user
        quotas.buckets.hard: 5
        usage.buckets: 1"]

        BA["S3Bucket: app-bucket
        quotas.storage.hard: 200 Gi
        usage.storage: 120 Gi"]

        UA -. "owns" .-> BA
    end

    subgraph ShootB ["Shoot Cluster B"]
        direction TB
        UB["S3User: batch-user
        (no quotas set)
        usage.buckets: 1"]

        BB["S3Bucket: archive-bucket
        (no quotas set)
        usage.storage: 600 Gi
        inherits full tenant remaining"]

        UB -. "owns" .-> BB
    end

    classDef cap fill:#fff4d6,stroke:#d9a441,stroke-width:2px;
    classDef quotaset fill:#e8f2ff,stroke:#4a90e2,stroke-width:1px;
    classDef noquota fill:#fef3f3,stroke:#c0392b,stroke-width:1px,stroke-dasharray:4 3;

    class TQ cap;
    class UA,BA quotaset;
    class UB,BB noquota;

    style ShootA fill:#f8f9fc,stroke:#7a9cde,stroke-width:2px,stroke-dasharray: 4 4
    style ShootB fill:#f8f9fc,stroke:#7a9cde,stroke-width:2px,stroke-dasharray: 4 4

• Blue border: explicit quota set — resource is constrained to its hard limit
• Red dashed border: no quota set — resource may consume all remaining tenant capacity

In this example:

• Tenant Quota Setting: 1Ti storage, 10 users, 20 buckets
• Current Usage: 720Gi storage (120Gi from app-bucket + 600Gi from archive-bucket), 2 users, and 2 buckets
• Remaining Available: Approximately 304Gi storage, 8 users, and 18 buckets

Because app-bucket has an explicit quota set, it is safely bounded at 200Gi. On the other hand, archive-bucket has no explicit limit and could consume the entire remaining ~304Gi of storage capacity, leaving nothing for other apps or users in either Shoot.

Practical Examples

1. Explicit Per-Resource Quotas (Recommended)

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: app-bucket
  namespace: default
spec:
  userRef:
    name: app-user
  quotas:
    storage:
      hard: "200Gi"
    objects:
      hard: 1000000

2. Inherited Quota (No Explicit Limits)

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3Bucket
metadata:
  name: archive-bucket
  namespace: default
spec:
  userRef:
    name: batch-user
  # quotas omitted -> inherits full remaining tenant quota
  # not recommended for shared environments

3. Limiting Bucket Count Per User

apiVersion: s3.osc.t-systems.com/v1alpha1
kind: S3User
metadata:
  name: app-user
  namespace: default
spec:
  displayName: "Application User"
  tier: Tier3
  quotas:
    buckets:
      hard: 5

app-user can own at most 5 buckets. Attempting to create a sixth bucket fails at the storage backend.

Quota Warning Events

The extension warns you as usage approaches a quota limit, so you can react before write operations start failing. Two mechanisms work together:

• Kubernetes warning events fire exactly once each time a threshold is crossed. Events are informational only — they never block operations.
• Status conditions persist the current over-threshold state on the resource. Events expire, conditions do not — to check whether a threshold is currently crossed, read the conditions, not the event log.

View both with kubectl describe (or kubectl get events) on the resource:

kubectl describe s3bucket my-bucket
kubectl describe s3user my-user
kubectl describe s3tq tier3-quota -n kube-system

Per-Resource Events (S3User / S3Bucket)

For every quota dimension, up to two warnings can fire, each with its own event reason and status condition:

Every reason is dimension-specific, so a dimension crossing never overwrites or hides another:

The event reason is always identical to the status condition it accompanies.

• The percentage threshold is evaluated against status.grantedQuota — the cap actually approved by the storage backend — not against the requested spec.quotas.[dimension].hard. A dimension whose quota has been requested but not yet granted produces no threshold warning.
• If you set only hard, you get that dimension's *QuotaThresholdExceeded warning at the application threshold percentage.
• If you also set soft, you additionally get its *QuotaSoftLimitExceeded warning when usage reaches your soft value. Because soft must be less than hard, the two warnings mark two distinct points on the way to the hard limit.
• If the storage backend grants less than you requested, the granted value may end up below your soft setting; in that case the threshold warning fires first and the soft warning may never fire before the hard cap.

Dimensions covered:

• bucket count (S3User quotas.buckets),
• storage and object count (S3Bucket quotas.storage / quotas.objects).

Example events on an S3Bucket whose storage usage has passed both marks:

Type     Reason                          Message
----     ------                          -------
Warning  StorageQuotaThresholdExceeded   storage usage has reached the 80% application threshold of the hard quota (10Gi)
Warning  StorageQuotaSoftLimitExceeded   storage usage has reached the user-set soft quota (5Gi)

When several dimensions cross at the same time, each one gets its own entry:

Type     Reason                          Message
----     ------                          -------
Warning  StorageQuotaThresholdExceeded   storage usage has reached the 80% application threshold of the hard quota (10Gi)
Warning  ObjectsQuotaThresholdExceeded   objects usage has reached the 80% application threshold of the hard quota (1000)

The message never includes the live usage value — read that from .status.usage.

Status Conditions

Each warning has a matching condition on the resource, using the same name as the event reason listed in the table above. The condition is True while usage is at or above that limit.

The event fires exactly when the condition transitions from False to True. When usage drops back below the limit, the condition returns to False silently — no recovery event is emitted. A later re-crossing fires the same event again; because the message is identical, Kubernetes aggregates it into the existing event entry and increments its count instead of creating a new one.

kubectl get s3bucket my-bucket -o jsonpath='{.status.conditions}' | jq

Tenant Events (S3TenantQuota)

A warning event is emitted on the S3TenantQuota for each enforced dimension whose quotaConsumed reaches the application threshold percentage of quotaGranted (storage is always evaluated; other dimensions only when enforced). Tenant reasons carry a Tenant prefix so they read distinctly from the per-resource ones: TenantStorageQuotaThresholdExceeded, TenantUsersQuotaThresholdExceeded, TenantBucketsQuotaThresholdExceeded and TenantObjectsQuotaThresholdExceeded.

Each dimension gets its own event and its own *QuotaThresholdExceeded condition (including UsersQuotaThresholdExceeded), so a dimension crossing later adds its own warning without disturbing the others:

Type     Reason                               Message
----     ------                               -------
Warning  TenantStorageQuotaThresholdExceeded  tenant storage consumption has reached the 80% application threshold of the granted quota (100Gi)
Warning  TenantBucketsQuotaThresholdExceeded  tenant buckets consumption has reached the 80% application threshold of the granted quota (10)

Application Threshold Percentage

The threshold percentage (default 80%) is configured by your cluster operator. It applies to every *QuotaThresholdExceeded and Tenant*QuotaThresholdExceeded warning. Soft limits are unaffected — those fire at the exact value you set.

Validation

When soft is set, it must be strictly less than hard for the same dimension — otherwise both warnings would fire at the same point. Invalid combinations are rejected when you apply the resource.

Troubleshooting

Diagnostic Commands

Quick Status Check

# List all S3 resources with their phases
$ kubectl get s3users,s3buckets -A

# Detailed status of a specific resource
$ kubectl describe s3user <name>
$ kubectl describe s3bucket <name>

# Check conditions
$ kubectl get s3user <name> -o jsonpath='{.status.conditions[*].type}'

# Check events
$ kubectl get events --field-selector involvedObject.name=<name>

Common Issues

Problem: Error Phase with a "contact support" Message

Symptoms: S3User or S3Bucket shows phase Error with one of these messages:

• user API access is not configured for this environment - contact support
• tenant-id is not defined for this environment - contact support
• user API connection is not configured correctly for this environment - contact support

Causes:

• The storage platform for this environment is not fully provisioned or misconfigured

Solution:

1. No user action is possible — these are platform-side configuration errors. Contact support and include the resource name and the exact message.

2. Once the platform is fixed, the resource recovers automatically; no recreation is needed.

Problem: Credentials Not Appearing

Symptoms: S3User is in Ready phase but secretName is empty or the secret does not exist.

Causes:

• Credentials still being transferred (check credentialsPhase)

Solution:

1. Check the credentials phase:

   kubectl get s3user <name> -o jsonpath='{.status.credentialsPhase}'
   

   Phase	Action
   Pending	Wait for storage backend to generate credentials
   Issued	Credentials generated, transfer in progress
   Transferred	Check secretName — secret should exist

2. Verify the secret exists:

   kubectl get secret $(kubectl get s3user <name> -o jsonpath='{.status.secretName}')
   

Problem: Credential Secret Was Deleted

Symptoms: The credential secret referenced in status.secretName no longer exists, but the S3User is still in Ready phase.

Causes:

• The secret was forcefully deleted (e.g., kubectl delete secret <name>) while the S3User was still active.

Solution:

No manual action is required. The extension automatically detects the missing secret and re-triggers credential issuance. The status.credentialsPhase returns to Issued during re-issuance and transitions back to Transferred once the new secret is available. Wait for the reconcile cycle to complete (typically within seconds).

To monitor recovery:

kubectl get s3user <name> -o jsonpath='{.status.credentialsPhase}'

Problem: S3Bucket Stuck in Pending with User Not Found

Symptoms: S3Bucket shows Pending phase with message referencing user not found.

Causes:

• Referenced S3User does not exist
• S3User is in a different namespace than expected
• Typo in .spec.userRef.name

Solution:

1. Check the status message:

   kubectl get s3bucket <name> -o jsonpath='{.status.message}'
   

2. Verify the referenced user exists:

   kubectl get s3user <userRef-name> -n <userRef-namespace>
   

3. If the user is in a different namespace, set .spec.userRef.namespace explicitly.

Problem: S3Bucket Error with Tier Mismatch

Symptoms: S3Bucket shows Error phase with tier mismatch message for additional users.

Causes:

• An additional user has a different tier than the bucket owner
• Different tiers use different physical storage clusters, making cross-tier access impossible

Solution:

1. Check the error message:

   kubectl get s3bucket <name> -o jsonpath='{.status.message}'
   

2. Verify the owner's tier:

   kubectl get s3user <owner-name> -o jsonpath='{.spec.tier}'
   

3. Verify additional users' tiers:

   kubectl get s3user <additional-user-name> -o jsonpath='{.spec.tier}'
   

4. Ensure all users (owner and additional) share the same tier value.

Problem: Deletion Blocked (Retention)

Symptoms: kubectl delete was issued but the resource persists, showing Retention phase.

Causes:

• Default Retain deletion policy is active
• retentionExpiresAt has not been reached

Solution:

1. Check the retention state:

   kubectl get s3user <name> -o jsonpath='{.status.retentionState}'
   

2. Check retention expiry:

   kubectl get s3user <name> -o jsonpath='{.status.retentionExpiresAt}'
   

3. To force immediate deletion (irreversible):

   kubectl annotate s3user <name> s3.osc.t-systems.com/force-delete=true
   

See the Deletion and Retention section for full details.

Problem: Resource Deleted by Accident

Symptoms: kubectl delete was issued on an S3User or S3Bucket that is still needed. The resource shows Retention phase with status.retentionState: PendingPurge; an S3User is suspended and its credentials stop working.

Causes:

• The deletion was unintentional and the resource uses the Retain deletion policy, so it is still inside its retention window

Solution:

1. Verify the resource is still recoverable:

   kubectl get s3user <name> -o jsonpath='{.status.retentionState}'
   

   Recovery is only possible while the state is PendingPurge. If the state is Purging or the resource is gone, the storage data has been removed — recreate the resource and restore from backups.

2. Cancel the deletion:

   kubectl annotate s3user <name> s3.osc.t-systems.com/recovery-requested=true
   

   (Use s3bucket for buckets.)

3. Wait for the phase to return to Ready (typically a few seconds). The resource keeps its name, spec, credentials, and data; only its Kubernetes UID changes.

See Recovery: Canceling a Pending Deletion for full details.

Problem: S3User Shows Suspended Phase

Symptoms: S3User phase is Suspended, S3 API requests fail with access denied.

Causes:

• .spec.suspended is set to true

Solution::

1. Verify the suspension setting:

   kubectl get s3user <name> -o jsonpath='{.spec.suspended}'
   

2. To restore access:

   kubectl patch s3user <name> --type=merge -p '{"spec":{"suspended":false}}'
   

3. Wait for the phase to return to Ready.

Problem: S3Bucket Owner Cannot Be Additional User

Symptoms: Applying an S3Bucket manifest fails with validation error about owner being listed as additional user.

Causes:

• The same user referenced in .spec.userRef also appears in .spec.additionalUsers

Solution:

Remove the owner from the additionalUsers list. The owner already has full access to the bucket.

Problem: Resource Creation Fails — Quota Assignment Limit Reached

Symptoms: Creating a new S3User or S3Bucket with explicit spec.quotas fails. The error references quota exceeded or insufficient remaining quota.

Causes:

• The requested quota assignment would push quotaAssigned over quotaGranted for one or more dimensions.
• Example: quotaGranted.buckets is 20, quotaAssigned.buckets is already 18, and the new S3User requests quotas.buckets.hard: 5. The resulting assignment would be 23, which exceeds the granted limit of 20 — the request is rejected.

Solution:

1. Check the current tenant quota state:

   kubectl get s3tq -A
   

2. Identify which dimension is exhausted by comparing quotaAssigned against quotaGranted:

   kubectl get s3tq tier3-quota -n kube-system -o yaml
   

3. List existing S3Users and their assigned quotas to find candidates for reduction:

   kubectl get s3users -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.quotas}{"\n"}{end}'
   

4. Choose one of the following:

   • Reduce the quota on an existing user or bucket that has more assigned than it consumes
   • Set a lower limit on the new resource so the total assignment stays within the granted cap
   • Contact support to request a tenant quota increase if neither adjustment is feasible

Problem: Upload Fails — Storage Quota Consumed

Symptoms: S3 API rejects an upload with an error such as QuotaExceeded or EntityTooLarge. The S3Bucket and S3User are both in Ready phase.

Causes:

• quotaConsumed.storage is close to quotaGranted.storage and the upload would push actual consumption over the granted limit.
• Example: quotaGranted.storage is 1Gi, quotaConsumed.storage is 900Mi, and the file being uploaded is 200Mi. The resulting consumption would be 1100Mi, exceeding the 1Gi cap — the storage backend rejects the write.
• The S3Bucket itself has a spec.quotas.storage.hard limit set, and that per-bucket cap is reached regardless of remaining tenant capacity.

Solution:

1. Check how much storage has been consumed versus what is granted at the tenant level:

   kubectl get s3tq -A
   

2. Confirm storage is the constraint at the tenant level:

   kubectl get s3tq tier3-quota -n kube-system -o jsonpath='{.status.quotaConsumed}{"\n"}{.status.quotaGranted}'
   

3. If the tenant quota has remaining capacity, check whether the S3Bucket has a per-bucket storage limit that is being hit:

   kubectl get s3bucket <name> -o jsonpath='{.spec.quotas.storage}'
   

   If the bucket's hard limit is too low, increase it:

   kubectl patch s3bucket <name> --type=merge -p '{"spec":{"quotas":{"storage":{"hard":"<new-limit>"}}}}'
   

4. If the tenant quota is the constraint, free storage by deleting unused objects or buckets, then retry the upload.

5. If more capacity is required, contact support to request a tenant quota increase.

See the Viewing Tenant-Wide Quota section for details on reading quota status.