Skip to content

OSC Node Feature Discovery Extension

Introduction

This document describes the basic usage of the OSC Node Feature Discovery Extension.

Configure via OSC Dashboard

This extension can be enabled, disabled and configured directly in the OSC Dashboard, without editing the Shoot manifest. See Managing Cluster Services and the Node Feature Discovery Configuration page.

Enabling the Extension for a Shoot Cluster

For enabling this extension for a Shoot cluster, the extension service named osc-nfd-shoot-service needs to be added to the extensions in the Shoot Custom Resource manifest:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  …
spec:
  …
  extensions:
    - type: osc-nfd-shoot-service
    …

You can issue this command to check the extensions on your Shoot cluster:

kubectl get cm -n kube-system shoot-info -o jsonpath={.data.extensions}

Disabling globally enabled extensions

To disable extensions which are enabled by default, add the following snippet to the Shoot manifest:

kind: Shoot
...
spec:
  extensions:
  - type: osc-nfd-shoot-service
    disabled: true
...

Components of NFD extension

When the NFD extension is installed in the Shoot, the osc-nfd-shoot-controller-manager also automatically deploys the NodeFeatureRule Custom Resource in the Shoot cluster. Through the NodeFeatureRule configuration mechanisms, we have the capability to advertise node-level resources, such as available EPC memory as extended resources.

However, be aware that node-level resources added in this manner on node capacity are not transparent to Kubernetes, and as a result, there is no built-in mechanism for controlling their consumption. Hence, the responsibility for managing these resources falls upon the users.

Shoot extension modifications

The NFD extension is currently managing multiple helm charts. Helm values from the Shoot manifests can be passed down to these helm charts. Moreover, it's possible to disable each service individually.

The providerConfig in osc-nfd-shoot-service allows modifications to the following components:

  • cgroups-prometheus-exporter: This component exports the cgroups statistics to Prometheus.
  • node-feature-rule: This component defines the rules for node feature discovery.
  • node-feature-discovery: This component discovers the features of the node.
  • nri-sgx-epc: This plugin can be used to set limits on the SGX EPC memory using annotations.
  • nvgpu-operator: This component deploys NVIDIA GPU Operator in the Shoot cluster.

Warning

While it's possible to disable node-feature-discovery or node-feature-rule, it's not recommended due to the potential for unexpected behavior.

Example of a Shoot YAML manifest:

kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
  name: myshoot
  namespace: myproject
spec:
  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        cgroups-prometheus-exporter:
          enabled: true
        node-feature-rule:
          enabled: true
        node-feature-discovery:
          enabled: true
        nri-sgx-epc:
          enabled: true
        nvgpu-operator:
          enabled: true
      disabled: false

Note

The providerConfig allows you to specify Helm chart values for each component, as shown below. However, this feature is intended only for troubleshooting and is not recommended to define values.

The node-feature-rule doesn't currently have any values to overwrite, except for extensionName, which is a string type.

Example of a Shoot YAML manifest with custom values:

...
  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        cgroups-prometheus-exporter:
          enabled: true
          values: |
            image:
              repository: mtr.devops.telekom.de/osc/common/monitoring/cgroups-prometheus-exporter
              tag: v0.2.1
              pullPolicy: Always
            prometheus:
              enablePrometheusRule: false
              enableServiceMonitor: false
        node-feature-discovery:
          values: |
            image:
              repository:  mtr.devops.telekom.de/osc/gardener/node-feature-discovery
              # This should be set to 'IfNotPresent' for released version
              pullPolicy: IfNotPresent
              tag: v0.17.2-minimal
            imagePullSecrets: []
            # name is immutable! can be set only once
            nameOverride: ""
            fullnameOverride: ""
            namespaceOverride: ""
          enabled: true
        nri-sgx-epc:
          enabled: true
          values: |
            nri:
              runtime:
                patchConfig: true
            image:
              name: mtr.devops.telekom.de/osc/gardener/nri-sgx-epc
      disabled: false

Cgroups Prometheus Exporter

The cgroups-prometheus-exporter is responsible for publishing metrics based on the memory.current, misc.current, misc.events and misc.max files available on each Shoot cluster. This data provides information about the usage of standard and EPC memory. For details about the exporter, refer to the Cgroups Prometheus Exporter reference.

To scrape metrics easily, you can enable a predefined service monitor for Prometheus in the Shoot manifest. The service monitor requires CRDs from kube-prometheus to already be present in the Shoot cluster.

...
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        cgroups-prometheus-exporter:
          enabled: true
          values: |
            prometheus:
              enableServiceMonitor: true

NRI-SGX-EPC Plugin

The NRI-SGX-EPC plugin allows users to define the EPC limit. This is achieved by configuring the declared limit through the container runtime, containerd.

Containerd does not currently support miscellaneous cgroups. To address this, the containerd community introduced the concept of NRI. NRI plugins function similarly to mutating webhooks for Kubernetes, in that they "mutate" the container specification before containerd instructs the low-level container runtime (runc).

Implementation

The NFD extension deploys the NRI plugin only if it is enabled in the Shoot manifest, as shown in the example above. By setting the node selector to intel.feature.node.kubernetes.io/sgx: "true", the EPC NRI plugin pods will only be scheduled for SGX enabled nodes. This label is added by the NFD add-on automatically to SGX enabled nodes with running OSC Scone Service Operator. ContainerD config is defined in /etc/containerd/config.toml.

The NRI EPC plugin includes an init container. The container patches config of containerd automatically to enable NRI support in containerd.

For details about NRI, refer to the official NRI reference.

The Requirements for running NRI plugin:

  • containerd v1.7.x
  • Shoot cluster with SGX memory support
  • The OSC Scone Service Operator must be deployed to provide labels and the sgxplugin
  • NRI must be enabled:
    • config is stored in /etc/containerd/config.toml
    • NRI must be enabled in [plugins."io.containerd.nri.v1.nri"\] disable = false
    • config is automatically patched into enabled state

It's possible to trigger the init container to patch the config by setting the value pathConfig in Shoot manifest:

nri:
  runtime:
    patchConfig: true

This plugin can be disabled by modifying the Shoot manifest as shown in Shoot extension modifications. It is also possible to modify some values, for example:

values: |
  image:
    name: ghcr.io/containers/nri-plugins/nri-sgx-epc
    #tag: unstable
    pullPolicy: IfNotPresent

  resources:
    cpu: 25m
    memory: 100Mi

  nri:
    plugin:
      index: 90
    runtime:
      patchConfig: false

  initContainerImage:
    name: ghcr.io/containers/nri-plugins/nri-config-manager
    #tag: unstable
    pullPolicy: IfNotPresent

  tolerations: []
  affinity: []
  nodeSelector: []
  podPriorityClassNodeCritical: true

Deployment example

Annotations can be defined for the whole pod or its containers. The values are in bytes. For details about the plugin, refer to the NRI Plugins reference.

...
metadata:
  annotations:
    # for all containers in the pod
    epc-limit.nri.io/pod: "32768"
    # alternative notation for all containers in the pod
    epc-limit.nri.io: "8192"
    # for container c0 in the pod
    epc-limit.nri.io/container.c0: "16384"
...

Tolerations

Users can specify tolerations in the values. The supported components are cgroups-prometheus-exporter, node-feature-discovery, and nri-sgx-epc.

  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        cgroups-prometheus-exporter:
          enabled: true
          values: |
            tolerations:
            - operator: Exists
            - key: example1
              operator: Exists
              effect: NoExecute
            - key: example2
              operator: Exists
              effect: NoExecute

For the DaemonSets managed by the NVIDIA GPU Operator, custom tolerations are configured under daemonsets.tolerations in the nvgpu-operator values instead.

Toleration overwriting

By default, the node-feature-discovery-worker, osc-cgroups-prometheus-exporter, and nri-sgx-epc pods, as well as the DaemonSets managed by the NVIDIA GPU Operator, have operator toleration without requiring it to be specified:

  tolerations:
    - operator: Exists

If custom tolerations are defined in the Shoot manifest, this default toleration is overwritten. It is possible to omit the default toleration and not define any other custom tolerations by specifying a manifest like this:

        nri-sgx-epc:
          enabled: true
          values: |
            tolerations:

Known issues

Be aware that any pod or container attempting to consume more memory than allowed by the epc-limit annotation will be restarted automatically.

ContainerD must be restarted to reflect the misc.max values for limits if they are not shown properly.

SGX must have the same limits and requests specified for EPC memory. Otherwise, we will face the following error message:

The Deployment "sgx-epc-stress-test" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "1Gi": must be equal to sgx.intel.com/epc limit

If the annotation with epc-limit is changed, then ContainerD will provide data for both the older and newer resources for a few seconds. Newer resources might overlap in the cgroups directory and metrics provided by the osc-cgroups-prometheus-exporter.

NRI-SGX-EPC Support matrix

The NRI-SGX-EPC plugin was tested in the following configurations:

NRI-SGX-EPC Plugin version Garden Linux version Kubernetes version Containerd version
v0.7.1 1550.0 1.28.11 1.7.15
v0.7.1 1605.0 1.26.14 1.7.20
v0.7.1 1605.0 1.28.14 1.7.20
v0.7.1 1510.0 1.26.8 1.7.11
v0.7.1 1510.0 1.29.9 1.7.11
v0.7.1 1510.0 1.30.8 1.7.11
v0.7.1 1510.0 1.31.4 1.7.11
v0.9.4 1510.1 1.26.8 1.7.11
v0.9.4 1605.0 1.31.6 1.7.20
v0.9.4 1721.0 1.32.3 1.7.23

NVIDIA GPU Operator

The NVIDIA GPU Operator (deployed via the nvgpu-operator Helm chart) is responsible for managing and monitoring GPU resources within Shoot clusters, ensuring seamless integration with the OSC ecosystem. All configuration values for the NVIDIA GPU Operator are predefined and optimized — by default it only needs to be enabled or disabled, and the DRA integration can be toggled. This simplified configuration is possible due to the preinstalled NVIDIA drivers and supporting components in Garden Linux, providing an optimized and stable environment for GPU workloads. For details about GPU workload usage, refer to the Shoot NVIDIA GPU workload.

In addition to the enabled and dra options, it is possible to override the Helm chart values of the NVIDIA GPU Operator through the providerConfig, in the same way as for the other components:

spec:
  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        nvgpu-operator:
          enabled: true
          values: |
            mig:
              strategy: mixed

In this example, the MIG strategy is changed from the default single to mixed. The mixed strategy is required when a node uses a MIG configuration with multiple different profiles (for example all-balanced). The MIG instances are then advertised as individual nvidia.com/mig-<profile> resources instead of a generic nvidia.com/gpu resource count. For the required order of steps when changing the MIG configuration, refer to the Applying the MIG Configuration.

Danger

The NVIDIA GPU Operator is deployed by the nvgpu-operator chart with predefined values that are tuned and tested for the OSC ecosystem. Overriding these values must be done with great care: custom values are applied on top of the optimized defaults and can break the GPU driver stack, the DRA integration, or GPU workload scheduling in the Shoot cluster. Only override values if you fully understand their impact, and always verify GPU workloads afterwards as described in the Shoot NVIDIA GPU workload. Configurations resulting from custom value overrides are not covered by the tested configurations listed in the support matrix. If customer-specific overrides are applied incorrectly, any resulting malfunction, instability, or service disruption is the customer's responsibility, and we do not assume service responsibility for issues caused by faulty overrides.

Dynamic Resource Allocation (DRA)

The extension also deploys the NVIDIA DRA GPU driver (k8s-dra-driver-gpu v25.12.0) alongside the operator. On clusters running Kubernetes 1.34.2 or later, dynamic resource allocation (DRA) is the default GPU allocation mechanism and replaces the device plugin (nvidia.com/gpu resource requests). On older Kubernetes versions the device plugin is used automatically.

DRA is the default — device plugin workflows must be reworked

If your Shoot cluster previously used a GPU Operator version with the device plugin, note that after the upgrade DRA is deployed as the default GPU allocation mechanism. Existing workloads that request GPUs via nvidia.com/gpu resource requests must be reworked for the DRA integration: create ResourceClaim (or ResourceClaimTemplate) objects and reference them in your pod specifications instead of using resource requests. For details about the DRA workflow and step-by-step examples (resource claims, device classes, MIG slices), refer to the GPU Allocation Using DRA.

Disabling DRA

DRA is enabled by default. It can be disabled through the dra option in the providerConfig:

spec:
  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        nvgpu-operator:
          enabled: true
          dra:
            enabled: false

After the Shoot cluster is reconciled with dra.enabled: false, the device plugin is deployed in place of the DRA driver and GPUs are requested via nvidia.com/gpu resource requests. For usage examples of both allocation patterns, refer to the Shoot NVIDIA GPU workload.

NVIDIA GPU Operator support matrix

The NVIDIA GPU Operator was tested in the following configurations:

NVIDIA GPU Operator version Driver version CUDA version NVGPU Garden Linux version Kubernetes version Containerd version
v25.10.1 590.48.01 13.1 2108.0 1.34.6 2.0.5