Skip to content

Worker pool management

Note

If you want to use legacy Gardener Dashboard please refer to this documentation.

OSC Dashboard

  1. Click Settings button as shown on picture below. Shoot-osc-dashboard-overview-worker-pool-management Continue in pop-up window as shown on image below

  2. Click + sign to add worker group. Shoot-osc-dashboard-add-worker-group

  3. Configure your worker pool by setting these characteristics

  • Group Name: enter here name of new worker pool
  • Machine Type: select from drop-down menu of currently supported Machine Types, according to your needs
  • Machine Image: select from drop-down menu of currently supported Machine Images, according to your needs
  • Container Runtime: select from drop-down menu of currently supported Container Runtime, according to your needs
  • Volume Size: enter here volume size in GiB
  • Autoscaling configuration:
    • Autoscaling Min.: enter here minimum number of worker nodes
    • Autoscaling Max.: enter here maximum number of worker nodes
    • Max. Surge: enter here maximum number of nodes that can be created exceeding the maximum size during updates
    • Max. Unavailable: enter here maximum number of nodes that can be unavailable during update
  • Zone: select Availability Zones

Shoot-worker-pool-management

Garden Cluster

On the Garden cluster level the Shoot object can be edited (using kubectl edit or kubectl patch) to update worker pool configuration. Example worker group setup:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
spec:
  provider:
    type: onmetal
    # Please note that workers is a list of worker pool configurations.
    workers:
    - name: idp
      cri:
        name: containerd
      machine:
        architecture: amd64
        image:
          name: gardenlinux
          version: 1061.0.20
        type: <machine-class>
      maxSurge: 1
      maxUnavailable: 0
      maximum: 12
      minimum: 12
      systemComponents:
        allow: true
      volume:
        size: 100Gi
        type: fast
      zones:
      - mdb1-pool1
      - mdb2-pool1
      - mdb3-pool1
    # Here an another worker pool configuration can be added

Shoot Worker Nodes Settings

SSHAccess ensures that the sshd.service is running on the worker nodes. This is handled by a systemd service called sshd-ensurer.service, which runs every 15 seconds on each worker node. When set to true, the service ensures that sshd.service is unmasked, enabled, and running. When set to false, it ensures that sshd.service is disabled, masked, and stopped. This also terminates all active SSH connections on the host.

sshAccess.enabled is set to true by default.

Example Usage in a Shoot

spec:
  provider:
    workersSettings:
      sshAccess:
        enabled: false

For more details, refer to the upstream Gardener documentation: Shoot Worker Nodes Settings

Determining Available IP Addresses in a Shoot Cluster

It is essential to understand how to calculate available IP addresses within a shoot cluster.

Note

The calculation of available IP addresses within a shoot cluster is crucial for deploying new nodes or services without conflicts.

Calculating Free Available IP Addresses

The maximum possible number of IP addresses are determined by the Shoot cluster's Node CIDR IP range. This is defined in th Shoot manifest in .spec.networking.nodes, (e.g. 10.0.0.0/28). The important part is the number after the slash — the suffix. This determines the number of node network addresses for the range.

CIDR Suffix Number of nodes
/23 512
/24 256
/25 128
/26 64
/27 32
/28 16
/29 8
/30 4

Info

The maximum size of a Shoot Node range is /23, that is 512 Nodes.

All Shoot worker nodes reserve one address from this pool (can be checked with kubectl get nodes).

Also, every existing private Load Balancer reserves one address. Private Load Balancers have the annotation .metadata.annotations.service.beta.kubernetes.io/onmetal-load-balancer-internal=true. The following command prints out all Load Balancers, with this annotation.

kubectl get services \
  -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IS PRIVATE:.metadata.annotations.service\.beta\.kubernetes\.io/onmetal-load-balancer-internal'

Here's breakdown of the command:

  1. kubectl get services: is the base command to retrieve information about Services in your Kubernetes cluster.

  2. -A: This flag stands for "all namespaces". It tells kubectl to retrieve information about Services in all namespaces accross your cluster.

  3. -o custom-columns: Output format is custom columns.

  4. 'NAMESPACE:.metadata.namespace': Display namespace under column "NAMESPACE".

  5. ,NAME:.metadata.name:Display name under column "NAME".

  6. IS PRIVATE:.metadata.annotations.service\.beta\.kubernetes\.io/onmetal-load-balancer-internal Check if internal LB flag is set (true or false). Count lines with true in last column.

Example Calculation

Let's assume we have a shoot cluster with /28 on .spec.networking.nodes. That means our maximum possible number of IP addresses is 16 inside of shoot network. Then let's check how many private Load Balancers are in use, we have 2. Let's assume there are currently 10 worker nodes running in the shoot. Let's calculate:

16 - ( 10 + 2 )
16 - 12
4

We have 4 IP addresses free and with that we can either create 4 private Load Balancers or upscale the shoot cluster by 4 nodes.

Configuring Kernel Parameters for Worker Nodes in Gardener Shoots

Disclaimer — customer responsibility

By modifying kernel parameters for shoot nodes, you acknowledge that all changes are made at your own discretion and responsibility. Kernel parameter modifications could lead to system instability or performance degradation. Please proceed with caution and ensure thorough testing before implementing changes in production environments.

Gardener allows you to configure kernel parameters (sysctls) for worker nodes in a shoot cluster by specifying them in the shoot manifest. This guide explains how you can customize kernel parameters for your worker nodes.

Prerequisites

  • Access to the Gardener landscape and permission to manage shoot clusters.
  • Familiarity with editing and applying shoot manifests.

Adding Kernel Parameters to Worker Nodes

To customize kernel parameters for worker nodes, include the sysctls field in the spec.provider.workers section of your shoot manifest.

Example Shoot Manifest

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-shoot
  namespace: garden-myproject
spec:
  provider:
    type: onmetal
    region: ffm
    workers:
      - name: worker-pool-name
        machine:
          type: vm-dedicated-2-4
        volume:
          type: fast
          size: 50Gi
        zones:
          - ffm1-dev
        sysctls:
          net.ipv4.neigh.default.gc_thresh2: '1024' # Default is 512 - This is just example of some parameters which could be changed
          net.ipv4.neigh.default.gc_thresh3: '2048' # Default is 1024 - This is just example of some parameters which could be changed

This example will change kernel parameters for worker group for garbage collector. In more detailes explained bellow.

Explanation of Fields

  • spec.provider.workers: Defines the worker pool configuration for the shoot cluster.
  • sysctls: Specifies the kernel parameters to be applied to the worker nodes.
    • net.ipv4.neigh.default.gc_thresh2: Adjusts the garbage collection threshold for ARP table entries. Default is 512. (example)
    • net.ipv4.neigh.default.gc_thresh3: Adjusts the maximum garbage collection threshold for ARP table entries. Default is 1024. (example)

Applying the Shoot Manifest

  1. Save the shoot manifest to a file (e.g., my-shoot.yaml).
  2. Apply the manifest using kubectl:
   kubectl apply -f my-shoot.yaml
  1. Gardener will reconcile the shoot cluster and apply the specified kernel parameters to the worker nodes.

There is no need to change nodes to apply settings. Settings will be applied online without any disruption.

Verifying Kernel Parameters

Once the shoot cluster is provisioned or updated:

  1. Login to node through node-shell
  2. Check the kernel parameter values using the sysctl command:
   sysctl net.ipv4.neigh.default.gc_thresh2
   sysctl net.ipv4.neigh.default.gc_thresh3

Notes

  • Kernel parameters specified in the shoot manifest are applied during the provisioning or reconciliation process.
  • If you need to modify the kernel parameters, update the sysctls section in the shoot manifest and reapply it.
  • Unexpected manual changes to kernel parameters on worker nodes may be overwritten during Gardener’s reconciliation process.

Troubleshooting

If the kernel parameters are not applied as expected:

  1. Verify the syntax of the shoot manifest.
  2. Check the logs of the Gardener extension responsible for operating system configuration (e.g., os-gardenlinux).
  3. Ensure you have appropriate permissions to manage the shoot cluster.

Additional Resources