Migration of Shoot controlplane to another Seed

Overall info

Currently moving the control plane of a Shoot cluster can only be done manually and requires deep knowledge of how exactly to transfer the resources and state from one Seed to another.

Note

• Source Seed is the Seed which currently hosts the control plane of a Shoot Cluster
• Destination Seed is the Seed to which the control plane is being migrated

Migration at the end is making backup on source Seed and restore it on target Seed. In other words it is redeploying the same Shoot object with restored ETCD database.

Beside ETCD database needs to be also some secrets migrated:

- ca
- ca-front-proxy
- static-token
- ca-kubelet
- ca-metrics-server
- etcd-encryption-secret
- kube-aggregator
- kube-apiserver-basic-auth
- kube-apiserver
- service-account-key
- ssh-keypair

Other secrets can be regenerated from them.

Gardenlet deploys custom resources in the Source Seed cluster during Shoot reconciliation which are reconciled by extension controllers. The state of these controllers and any additional resources they create is independent of the gardenlet and must also be migrated to the Destination Seed. Following is a list of custom resources, and the state which is generated by them that has to be migrated.

• BackupBucket: nothing relevant for migration
• BackupEntry: nothing relevant for migration
• ControlPlane: nothing relevant for migration
• DNSProvider/DNSEntry: nothing relevant for migration
• Extensions: migration of state needs to be handled individually
• Infrastructure: terraform state
• Network: nothing relevant for migration
• OperatingSystemConfig: nothing relevant for migration
• Worker: Machine-Controller-Manager related objects: machineclasses, machinedeployments, machinesets, machines

This list depends on the currently installed extensions and can change in the future

Migration workflow

• Starting migration
  • Migration can only be started after a Shoot cluster has been successfully created so that the status.seed field in the Shoot resource has been set
  • The Shoot resource's field `spec.seedName="new-seed" is edited to hold the name of the Destination Seed and reconciliation is automatically triggered
  • The Garden Controller Manager checks if the equality between spec.seedName and status.seed, detects that they are different and triggers migration.
• The Garden Controller Manager waits for the Destination Seed to be ready
• Shoot's API server is stopped
• Backup the Shoot's ETCD.
• Extension resources in the Source Seed are annotated with gardener.cloud/operation=migrate
• Scale Down the Shoot's control plane in the Source Seed.
• The gardenlet in the Destination Seed fetches the state of extension resources from the ShootState resource in the Garden cluster.
• Normal reconciliation flow is resumed in the Destination Seed. Extension resources are annotated with gardener.cloud/operation=restore to instruct the extension controllers to reconstruct their state.
• The Shoot's namespace in Source Seed is deleted.

Migration of shoot by editing shoot manifest manualy

Shoot name: sts-01
Gardener project name: dev
Name of seed namespace for Shoot: Shoot--dev--sts-01
Name of source seed cluster: fsd2-0
Name of destination seed cluster: fsd2-1
Name of Garden cluster: fgd2

Migration could be initiated through the editing Shoot object from gardener cluster

kubectl edit Shoot/mcm1 -n garden-dev --kubeconfig <kubeconfig of fgd2 garden cluster>

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: sts-01
  ...
spec:
 seedName: fsd2-0 --> fsd2-1
  ...

Migration of shoot via Gardener dashboard

Second option how to modify Shoot object is through the Gardener dashboard (needed admin permission for Gardener cluster)

Migration Specification and Progress

Let's assume that some prerequisites are done and start with migration process:

Prerequisites:

• target seed already exists
• secret specified in yaml of the shoot field secretBindingName exists on target seed
• seed persists on the same Gardener cluster

Select proper destination seed and confirm action by writing shoot name in confirmation dialog.

After pressing SAVE button, redeployment / migration of shoot start.

After migration of shoot, we can see and check shoot specification for shoot. Shoot was successfully migrated.

Validation and further checks after migration of shoot

During the deployment of the shoot, various objects have been created by deployer also on api cluster, therefore after migration of the shoot, from source to target seed, we need to validate some objects which can be related to the old shoot and were not recreated, or stayed with wrong naming logic (objects are namespaced in kubernetes).

Findings:

NOTE: fsd2-1 - source seed, fsd2-0 - target seed

• namespace - during the initial deployment of seed, new namespaces are not created automatically on api cluster, therefore are not deleted automatically also.

• machines - still in old namespace
  • before migration:

• after migration:

• secrets - not deleted or recreated. Secrets (see below) created during initial deployment of shoot, were created in shoot namespace on api cluster, but during the migration of shoot were not deleted or recreated in new namespace.

• shoot yaml

After the migration of shoot between seeds, some information are not updated in shoot yaml.

  secretBindingName: fsd2-1-creds
  seedName: fsd2-0
  seedSelector:
    matchLabels:
      purpose: dev-fsd2-1

It is not recommended to have fields seedName and seedSelector in one shoot yaml file, because this can change internal logic of kubernetes actions during shoot reconciliation process. From observations, we can see that seeName field has higher precedence as seedSelector.

• check OIDC kubeconfigs after migration