Skip to content

OIDC configuration for shoot clusters

OpenID Connect

The kube-apiserver of Shoot clusters can be provided with OpenID Connect configuration by providing specific label within the Shoot spec during the Shoot creation. The label in the Shoot manifest must match name of the seed cluster on which your Shoot should reside:

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: <name of the shoot>
  namespace: <namespace name>
  labels: settings.gardener.osc.t-systems.com/realm=<seed name>
spec:
  ...

If the label was not specified in Shoot's YAML manifest during creation time, it will be added automatically by webhook.

OIDC kubeconfig

OIDC kubeconfig is stored in shoot-name.kubeconfig-oidc secret in the Shoot's Project namespace in the Garden cluster. Only users with owner or admin role assign within specific Project are able to retreive secrets within Project's namespace.

The recommended way to use OIDC kubeconfig is the kubectl plugin called kubectl oidc-login.

Static Token kubeconfig

This kubeconfig contains a static token and provides cluster-admin privileges. It is stored in the shoot-name.kubeconfig secret in the Shoot's Project namespace in the Garden cluster.

Static token kubeconfig is not available for Shoot clusters using Kubernetes version >= 1.27.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
...
spec:
  kubernetes:
    enableStaticTokenKubeconfig: true
...

It is not the recommended method to access the shoot cluster with the static kubeconfig since there are security flaws associated with it:

  • The static token in the kubeconfig doesn’t have any expiration date.
  • The static token doesn’t have any user identity associated with it. The user associated with that token will always be system:cluster-admin, irrespective of the person accessing the cluster. Hence, it is impossible to audit the events in cluster.

When enableStaticTokenKubeconfig field is not explicitly set in the Shoot spec:

  • for Shoot clusters using Kubernetes version < 1.26 the field is defaulted to true.
  • for Shoot clusters using Kubernetes version >= 1.26 the field is defaulted to false.
  • for Shoot clusters using Kubernetes version >= 1.27 the field is locked to false.

Example of OIDC configuration in Shoot YAML manifest

spec:
  kubernetes:
    kubeAPIServer:
      enableBasicAuthentication: false
      oidcConfig:
        caBundle: |
          -----BEGIN CERTIFICATE-----
          # This CA certificate resides on the seed cluster
          -----END CERTIFICATE-----
        clientAuthentication:
          extraConfig:
            extra-scopes: >-
              email,offline_access,profile,groups,audience:server:client_id:kubernetes
        clientID: kubernetes
        groupsClaim: groups
        groupsPrefix: 'oidc:'
        issuerURL: https://auth.apps.fsd1-2.ffm.osc.live/dex 
        signingAlgs:
          - RS256
        usernameClaim: preferred_username
        usernamePrefix: 'oidc:'

How to create custom kubeconfig with limited access to cluster

  • create custom sa (ServiceAccount)
    kubectl create serviceaccount pods-list-sa -n minio
  • verify that sa was created successfully
    kubectl get sa -n minio
NAME           SECRETS   AGE
default        0         36d
minio-sa       0         36d
pods-list-sa   0         35s
  • create token for custom ServiceAccount via secret
    apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  # uid value from ServiceAccount
  name: minio-token-3c03ef14-e16d-4c91-ae62-55d4f714dc2f
  namespace: minio
  annotations:
    # newly created ServiceAccount
    kubernetes.io/service-account.name: pods-list-sa
  labels:
    reason: minio-list-pods-only
    requested-by: demo-user
type: kubernetes.io/service-account-token

    !!! Info Validity of the created token is 10 years.

  • verify that secret was created successfully
    kubectl get secret -n minio minio-token-3c03ef14-e16d-4c91-ae62-55d4f714dc2f 
NAME                                               TYPE                                  DATA   AGE
minio-token-3c03ef14-e16d-4c91-ae62-55d4f714dc2f   kubernetes.io/service-account-token   3      26s
  • create a custom role in namespace minio which will allow the ServiceAccount to only select pods and pod logs only from namespace minio
    kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: minio
  name: pod-list-read-only
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
  • create RoleBinding to bind created role pod-list-read-only with ServiceAccount pods-list-sa
    apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-list-sa-binding
  namespace: minio
subjects:
  - kind: ServiceAccount
    name: pods-list-sa
    namespace: minio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-list-read-only
  • generate custom kubeconfig

    Before generate custom kubeconfig we need to temporary define helper" variables.

    # name of the cluster
export clustername="<my-cluster>"

    # ca from secret in base64
export ca=$(kubectl get secret \
  -n minio \
  minio-token-3c03ef14-e16d-4c91-ae62-55d4f714dc2f \
  -o jsonpath='{.data.ca\.crt}')

    # cluster api
export server="https://api.<my-cluster>.ffm.osc.live"

    # ServiceAccount
export sa_name="pods-list-sa"

    # namespace
export ns="minio"

    # extracted token from newly created secret
export token=$(kubectl get secret \
  -n minio \
  minio-token-3c03ef14-e16d-4c91-ae62-55d4f714dc2f        
  -o jsonpath='{.data.token}' \
  | base64 --decode)

    cat <<EOF > $sa_name@${clustername}.yaml
apiVersion: v1
kind: Config
clusters:
  - name: ${clustername}
    cluster:
      certificate-authority-data: ${ca}
      server: ${server}
contexts:
  - name: ${sa_name}@${clustername}
    context:
      cluster: ${clustername}
      namespace: ${ns}
      user: ${sa_name}
users:
  - name: ${sa_name}
    user:
      token: ${token}
current-context: ${sa_name}@${clustername}
EOF

Custom kubeconfig will be printed to screen only.

  • test of newly created kubeconfig
    ## check nodes from cluster
## errors will be raised up because no permissions in role

-->kubectl get nodes --kubeconfig /tmp/sa-kubeconfig.yaml
Error from server (Forbidden): nodes is forbidden: 
User "system:serviceaccount:minio:pods-list-sa" cannot list resource "nodes" 
in API group "" at the cluster scope
-->

## check pods in kube-system namespace
## errors will be raised up because no permissions in role

-->kubectl get pods -n kube-system --kubeconfig /tmp/sa-kubeconfig.yaml
Error from server (Forbidden): pods is forbidden: 
User "system:serviceaccount:minio:pods-list-sa" cannot list resource "pods" 
in API group "" in the namespace "kube-system"
-->

## check pods in namespace minio --> working

-->kubectl get pods -n minio --kubeconfig /tmp/sa-kubeconfig.yaml
NAME             READY   STATUS    RESTARTS   AGE
minio-pool-0-0   2/2     Running   0          13d
minio-pool-0-1   2/2     Running   0          13d
minio-pool-0-2   2/2     Running   0          13d
minio-pool-0-3   2/2     Running   0          13d
-->

## check logs from minio pods

-->kubectl logs minio-pool-0-0 -n minio --kubeconfig /tmp/sa-kubeconfig.yaml \ 
  | head -5

Defaulted container "minio" out of: minio, sidecar, validate-arguments (init)
{"level":"INFO","time":"2024-07-18T08:42:39.432932792Z","message":
"Automatically configured API requests per node based on available memory 
on the system: 133"}

{"level":"INFO","time":"2024-07-18T08:42:39.433183586Z","message":
"All MinIO sub-systems initialized successfully in 5.210959ms"}
...

-->

Conclusion: As we can see from this short demo, we can define a limited access to a cluster, based on security policy defined within the customer organization.

Tip

We strongly recommended to not create token with infinite lifespan.

For more details about RBAC please refer to the official Kubernetes documentation