Skip to content

OSC S3 Bucket Extension

Introduction

This document describes the basic usage of the OSC S3 Bucket Extension.

Enabling the Extension for a Shoot Cluster

For enabling this extension for a Shoot cluster, the extension service named osc-s3-bucket-service needs to be added to the extensions in the Shoot Custom Resource manifest:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  …
spec:
  …
  extensions:
    - type: osc-s3-bucket-service
    …

You can issue this command to check the extensions on your Shoot cluster:

kubectl get cm -n kube-system shoot-info -o jsonpath={.data.extensions}

Disabling globally enabled extensions

To disable extensions which are enabled by default, add the following snippet to the shoot manifest:

kind: Shoot
...
spec:
  extensions:
  - type: osc-s3-bucket-service
    disabled: true
...

Create an S3 Bucket

The following example S3Bucket manifest can be used to request a new S3 bucket in a Shoot cluster :

apiVersion: storage.osc.extensions.gardener.cloud/v1alpha1
kind: S3Bucket
metadata:
  name: example
  namespace: default

Note

In this example the default namespace is used and therefore the namespace field is omitted in the further steps.

After the S3Bucket CR was created, its status can be checked in the status of the CR. If the bucket was created successfully and can be consumed, the status of the S3Bucket CR will be set to Ready (.status.Ready):

❯ kubectl get s3buckets.storage.osc.extensions.gardener.cloud
NAME      READY   AGE
example   true    4m35s

The access details for the bucket can be fetched from the secret referenced in the status of the S3Bucket CR (.status.credentialsSecret.name):

❯ kubectl get s3buckets.storage.osc.extensions.gardener.cloud example -o yaml
apiVersion: storage.osc.extensions.gardener.cloud/v1alpha1
kind: S3Bucket
metadata:
  …
status:
  …
  credentialsSecret:
    name: s3-bucket-example
  …

The secret contains the following information:

s3_access_key_id

The access key for the s3 bucket ("user identity")

s3_bucket

The bucket name (to avoid the upper limit of 64 characters for bucket names, the name is a sha224 hash)

s3_ca_bundle

The CA bundle used by the MinIO API (used to sign the server certificates of the MinIO API and needed for validating the provided certificates)

s3_endpoint

URL of the MinIO API

s3_secret_access_key

The secret access key for the s3 bucket ("user password")

The secret can be directly consumed by workloads running in the same Kubernetes namespace. by mounting the secret into the workload container or using the content as environment variables (see the Kubernetes documentation for more details).

If the workload is running in another namespace than the secret, the workload needs to fetch the secret via the Kubernetes API or the credentials must be provided manually.

Access S3 Bucket

To access the S3 bucket access credentials are needed. To get these we have to check our created S3 bucket:

kubectl get s3buckets.storage.osc.extensions.gardener.cloud example -o yaml

  credentialsSecret:
    name: s3-bucket-example
  ready: true

The access credentials can be fetched from the referenced Kubernetes secret as follows:

kubectl get secret s3-bucket-example -o go-template='{{ printf "\n%s: %s\n%s: %s\n%s: %s\n%s: %s\n\n" "ACCESS_KEY_ID" (.data.s3_access_key_id|base64decode) "SECRET_ACCESS_KEY" (.data.s3_secret_access_key|base64decode) "S3_ENDPOINT" (.data.s3_endpoint|base64decode) "S3_BUCKET_NAME" (.data.s3_bucket|base64decode) }}'

ACCESS_KEY_ID: <shoot_name>---<s3_bucket_namespace_in_shoot>-<s3bucket_name_in_shoot>
SECRET_ACCESS_KEY: <secret_access_key>
S3_ENDPOINT: <bucket_endpoint>
S3_BUCKET_NAME: <bucket_name>

Example output:

ACCESS_KEY_ID: shoot--dev--pgsql-dev-07---default-example
SECRET_ACCESS_KEY: JymNbAqouI2XhaJtEuyrOSjLfV380wwk
S3_ENDPOINT: minio.apps.ps-dev11.t-sov.live
S3_BUCKET_NAME: 4e411475f412ab5bb75da3fb6488a328a4fec2d0b968a4218e7e84e5

Note

The bucket name is a sha224 hash representation of the ACCESS_KEY_ID to avoid the upper character limit for bucket names.

To list the buckets the MinIO CLI can be used (see the installation instructions). The installation procedure should create a configuration folder for the MinIO CLI.

For the S3 bucket, a configuration file can be created as follows:

kubectl get secret s3-bucket-example -o go-template='{{ printf "{\"aliases\":{\"s3\":{\"%s\":\"%s\",\"%s\":\"%s\",\"%s\":\"https://%s\",\"api\":\"S3v4\",\"path\":\"dns\"}}}" "accessKey" (.data.s3_access_key_id|base64decode) "secretKey" (.data.s3_secret_access_key|base64decode) "url" (.data.s3_endpoint|base64decode) }}' > ~/.mc/config.json

Note

The OSC S3 service uses a (server) certificate signed by an internal OSC certificate authority. Therefore the client which connects to the S3 bucket needs to know this certificate authority. The needed CA bundle is included in the Kubernetes secret which contains the access credentials (the s3_ca_bundle key).

For the MinIO CLI the OSC CA bundle can be added as follows:

kubectl get secret s3-bucket-example -o go-template='{{ (.data.s3_ca_bundle|base64decode) }}' > ~/.mc/certs/CAs/s3-bucket-example.pem

We can list now the bucket with the command

mc ls <our_alias_from_config>/<our_s3_bucket_name>

Example:

mc ls s3/4e411475f412ab5bb75da3fb6488a328a4fec2d0b968a4218e7e84e5

User Management

Create S3 User

S3 Users can be created by applying the following S3User Custom Resource manifest:

apiVersion: storage.osc.extensions.gardener.cloud/v1alpha1
kind: S3User
metadata:
  name: example
  namespace: default

After the S3User Custom Resource was created, its status can be checked in the status of the CR.

Out of the box, no user credentials will be available for this user. To create access credentials, the S3User must first be authorized for a specific S3Bucket with one of following permissions types:

  • readwrite
  • readonly
  • writeonly

Authorize an S3User for an S3 Bucket

To authorize an S3User for a specific S3Bucket, the S3Bucket CR must be updated to include the user in the users list:

apiVersion: storage.osc.extensions.gardener.cloud/v1alpha1
kind: S3Bucket
metadata:
  name: example
  namespace: default
spec:
  users:
    - objectReference:
        name: example
        namespace: default
      policy: readwrite # readwrite | readonly | writeonly

Note

The existence of the S3User CR is checked when creating or updating the S3Bucket CR. If the S3User CR does not exist, the request will fail.

A credential Secret for the authorized S3User will be created in the same namespace as the S3User CR, not in the namespace of S3Bucket. Therefore, it is possible to create an S3User in a different namespace than the S3Bucket.

Information about the current S3Bucket authorization status and the credential Secret name will be propagated to the corresponding S3User BucketReference field:

   buckets:
    - objectReference:
        name: s3bucket-user-test
        namespace: default
      policy: writeonly
      secretReference:
        name: example-6n8dm

The credential secret name is generated from the S3User CR name by adding a random suffix. In our example, the credential secret name is example-6n8dm.

Note

The user credential Secret follows the same data fields naming style as the default user credential secret.

S3User deletion

Before deleting an S3User CR, make sure it is not authorized in any S3Buckets. If it is still authorized in some S3Bucket, the deletion request will be rejected.

S3Bucket Storage Utilization

The .Status.Capacity.Used field indicates the amount of storage consumed by respective S3Bucket CR. This information provides visibility into the storage utilization of the real S3 bucket storage associated with the S3Bucket CR.

Note that the size utilization in the .status.capacity.used field is represented in Gibibytes (Gi) scale. If the resulting size of all objects is less than 1 Gi, it will be rounded up to 1 Gi. For newly created S3Bucket CRs, the initial capacity of 1Gi will be added immediately. If the combined size of all objects exceeds 1 Gi but includes a fractional part, it will be rounded up to the nearest integer.

Object Expiration

Warning

Currently, object expiration rules cannot be specified via the S3Bucket CR (this is currently a feature on our roadmap).

Currently, object expiration rules must be set manually. For doing so, the MinIO CLI can be used, see the upstream documentation for Object Expiration.

For our example bucket one can define the corresponding lifecycle rule as follows:

mc ilm rule add \
  --expire-days <days> \
  --noncurrent-expire-days <days>
  <minio_alias>/<bucket_name>

For the example above:

mc ilm rule add \
  --expire-days 7 \
  --noncurrent-expire-days 7 \
  s3/4e411475f412ab5bb75da3fb6488a328a4fec2d0b968a4218e7e84e5

For listing the current lifecycle rules:

mc ilm rule ls <minio_alias>/<bucket_name>

For the example above:

mc ilm rule ls s3/4e411475f412ab5bb75da3fb6488a328a4fec2d0b968a4218e7e84e5

Bucket Quota

Warning

Currently, bucket quotas cannot be specified via the S3Bucket CR (this is currently a feature on our roadmap).

Currently, there is no way for users to define quotas for buckets.

Delete S3 Bucket

To delete s3 buckets simply issue:

kubectl delete s3buckets.storage.osc.extensions.gardener.cloud example

Warning

Currently there is no support for updating S3 Bucket objects.

Danger

Be aware that deletion of s3Bucket CR will cause deletion of s3 bucket storage in MinIO and all data will lost.

Shoot cluster - S3 Bucket storage coupling

Note that based on the architecture design of the osc-gardener-extension-shoot-s3-bucket, each S3 bucket storage is uniquely coupled with the Shoot cluster in which the respective S3Bucket custom resource (CR) was created.

Danger

Therefore, deleting the Shoot cluster (the Shoot CR) will cause the deletion of all S3 buckets defined in it. Therefore, all data in these buckets will be lost!

Note

In case the Shoot cluster, in which an S3 bucket is defined, is not functional, but is not deleted, the S3 bucket is not affected or deleted. The same is true for hibernated Shoot cluster.

To keep the data stored in an S3 bucket of a Shoot cluster, which is going to be deleted, the data must be be migrated manually to another S3 bucket. To do this, another S3 bucket can be created by creating an S3Bucket CR in another Shoot.

To migrate data between two buckets, it is possible to use the MinIO mc CLI, s3cmd, or any other tool supporting this functionality. For instance, using the mc this could be done by mc cp or mc mirror commands, depending on your purpose.

Note

The access credentials for both buckets are needed for the data migration. In case the access credentials for the first bucket are lost, the operations team needs to be contacted.