Skip to content

Shoot K8S version management

Downgrading

Currently not supported.

What to do if lower version is needed?

Deploy new shoot with one of lower version which are supported by us.

What to do, if upgrade is needed and we want to test it?

  1. Deploy another test shoot, with same version of K8S as you have on the shoot cluster you want to upgrade.
  2. Deploy applications you want/need to test on newer version to shoot cluster from step 1.
  3. Upgrade shoot cluster from step 1. to desired K8S version.
  4. Delete shoot cluster from step 1.
  5. Upgrade your existing cluster.

Upgrading

Before upgrading

Check if Sconified services, especially CAS is running in your cluster. If yes, follow the steps described in Upgrade K8s on Shoot Cluster with running CAS

Check supported K8S versions via kubectl directly on garden cluster or in the Gardener Dashboard (described later).

How to check directly on Garden cluster via kubectl

Execute following command:

kubectl get cloudprofiles.core.gardener.cloud onmetal -o=custom-columns='SUPPORTED K8S VERSIONS:.spec.kubernetes.versions[*].version' | sed s/,/\\n/g

example output of previous command:

SUPPORTED K8S VERSIONS
…some lines omitted…
1.28.10
1.28.11
1.28.14
1.28.15
1.29.9
1.29.12
1.30.8
1.31.4

Use grep to search for specific version, as on example below:

$ kubectl get cloudprofiles.core.gardener.cloud onmetal
-o=custom-columns='SUPPORTED K8S VERSIONS:.spec.kubernetes.versions[*].version'
| sed s/,/\\n/g | grep 1.25.9
# output after executing the command should be `1.25.9`

To upgrade K8S version, you can safely do it through gardener-dashboard or by updating shoot yaml in the project namespace on garden cluster.

Through Gardener-Dashboard

Open Gardener-dashboard

In the list of shoot cluster find the cluster you want to upgrade. Then as you can see on picture below follow these steps:

  1. Click on the current K8S version of shoot cluster in VERSION column.

  2. In pop-up window click on Upgrade to version.

  3. From drop-down menu Upgrade to version choose accordingly from available upgrade options based on your desired K8S version.

    shoot-k8s-version

    There is possibility, that you will need to upgrade more times, for example if shoot K8S version is 1.23.x and we want to upgrade to 1.26.x we would see in drop-down menu something like this, and we would need to upgrade first to 1.24.x, then 1.25.x and then 1.26.x:

    • 1.23.x → 1.24.x
    • 1.24.x → 1.25.x
    • 1.25.x → 1.26.x

  4. Then write or copy/paste shoot name for confirmation in the bottom-left field.

  5. Click UPDATE button to trigger the update which should be completed within some minutes.

    shoot-k8s-version

Through Shoot Manifest

This can be done in gardener-dashboard and via kubectl directly on the garden cluster.

In Gardener Dashboard

Let's take a look on how to do it through dashboard.

  1. Click on shoot name in the list as shown on picture below. shoot-pick

  2. Click on YAML

  3. Search for version and re-write it to your desired version as shown on picture below: shoot-k8s-version-upgrade-in-dashboard-yaml

On Garden Cluster using Kubectl

  1. List shoot clusters by using kubectl get shoots.core.gardener.cloud -A as on Example.

    Example:

    $ kubectl get shoots.core.gardener.cloud -A
NAMESPACE    NAME         CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION               STATUS      AGE
garden-dev   shmr20-t00   onmetal        onmetal    mdb      1.26.2        Awake         Reconcile Succeeded (100%)   unhealthy   2d23h
garden-dev   shmr20-t01   onmetal        onmetal    mdb      1.26.2        Awake         Reconcile Processing (88%)   unknown     6h31m
garden-dev   shmr20-t02   onmetal        onmetal    mdb      1.26.2        Awake         Reconcile Processing (82%)   unhealthy   5h55m

  2. Edit the same field version: in the shoot manifest. To do it we can use following command:

    kubectl edit shoots.core.gardener.cloud <your-shoot-cluster-name> -n <your-shoot-cluster-namespace-from-previous-command>

  3. Save by :wq! and wait for reconciliation to finish.

    We could do it locally by getting yaml from the cluster executing:

    kubectl get shoots.core.gardener.cloud <your-shoot-cluster-name> -n <your-shoot-cluster-namespace-from-previous-command> -o yaml > <your-shoot-cluster-name>.yaml

    then edit and save the manifest and apply it on the cluster using:

    kubectl apply -f <your-shoot-cluster-name>.yaml

Upgrade K8s on Shoot Cluster with running CAS

shoot-k8s-upgrade-with-running-cas

  1. Check Shoot Cluster number of worker nodes, please see how to check maximum number of nodes. The Shoot Cluster must have decreased-number-of-nodes < current-number-of-worker-nodes-max of worker nodes. If Shoot Cluster has maximum number of nodes, then decrease it to decreased- number-of-nodes - 4 < current-number-of-worker-nodes-max, please see worker.

  2. After that create additional worker pool with 3 nodes in each Availability Zone with following snippet

- name: tmp-wrk-pool
  kubernetes:
    version: 1.XX.xx
  minimum: 3
  maximum: 3
  maxSurge: 1
  machine:
    type: <machineclass-you-want-to-use>
    image:
      name: gardenlinux
      version: 1061.0.20
    architecture: amd64
  zones:
    - AZ1
    - AZ2
    - AZ3
  cri:
    name: containerd
  volume:
    type: fast
    size: 50Gi

and then copy and paste it on .spec.provider.workers:

spec:
  provider:
    workers:
     …
     # Paste tmp-wrk-pool under your actual worker pool

This snippet can be added through Gardener Dashboard or directly on Garden Cluster into Shoot Cluster YAML manifest, please see 08-Shoot-worker-pool-management. When you save the snippet configuration, wait for reconciliation to finish and then check if worker nodes from temporary worker pool were created and joined the cluster. Check status of CAS CR, wait for the migration to finish. When CAS CR show status HEALTHY migrate CAS to other worker pool nodes. CAS CR status must be healthy before proceeding.

To continue, you must follow these steps:

  1. Cordon the nodes from current-wrk-pool you can do it by executing

      kubectl get no -o name --no-headers | grep `current-wrk-pool` | xargs -I {} kubectl cordon {}

  2. Delete pod/cas-0, then wait for it to schedule on one of tmp-wrk-pool nodes.

  3. Delete pod/cas-backup-controller-*, then wait for it to schedule on one of tmp-wrk-pool nodes.

  4. Uncordon the nodes from current-wrk-pool you can do it by executing:

      kubectl get no -o name --no-headers | grep `current-wrk-pool` | xargs -I {} kubectl uncordon {}

  5. Monitor CAS CR status and wait until migration is finished and status is HEALTHY.

  6. Proceed with update of Shoot Cluster by following the steps 08-Shoot-k8s-version-management.
  7. Wait for upgrade to finish and CAS CR status is HEALTHY.
  8. Migrate CAS from tmp-wrk-pool to now upgraded current-worker-pool by following steps from 2..
  9. Delete tmp-wrk-pool.
  10. Upgrade with CAS is finished. Now, you can scale up your current-worker-pool back to number of worker nodes you had before starting upgrade process.