NVIDIA GPU Operator

Overview

The NVIDIA GPU Operator is a Kubernetes operator that automates the deployment, configuration, and management of the NVIDIA software required to run GPU-accelerated workloads.

GPU compute nodes require an individual order

Please note that according to the Service Description the use of GPU compute nodes requires a corresponding order and a successful capacity check.

GPU compute nodes are not permanently maintained in the platform's standard capacity pool. Provisioning therefore only takes place after an individual order and approval. Because procurement is required, the usual provisioning times for additional compute nodes do not apply. The actual provisioning time is agreed with the customer as part of capacity planning.

GPUs are allocated to workloads through one of two allocation patterns:

GPU Allocation Using the Device Plugin — workloads request GPUs through nvidia.com/gpu resource requests
GPU Allocation Using DRA — workloads request GPUs and MIG slices through resource claims

Note

Dynamic resource allocation (DRA) is the default allocation pattern on clusters running Kubernetes 1.34.2 or later. On older Kubernetes versions, or when DRA is disabled, the device plugin is used automatically.

Getting Started

Setting Up Your Shoot Cluster with GPU Workers

To use NVIDIA GPUs in your Shoot cluster, at least one worker group must use a GPU-enabled machine type.

In your project, click Create Cluster to open the new Shoot form in the OSC Dashboard.
Select a Machine Type with GPU capabilities.
Set the Machine Image to NVGPU Garden Linux.
Select a Zone where GPU nodes are available.
Fill in the remaining fields and click Create.
Wait for the Shoot cluster to be created and reconciled.

Warning

If only a limited number of GPUs is available, set Autoscaler Max. equal to Autoscaler Min. and set maxSurge: 0 and maxUnavailable: 1. Otherwise a rolling update can get stuck, because no additional GPU node can be provisioned for the surge.

Verifying GPU Availability

Check Node Labels

The Node Feature Discovery component automatically labels nodes with GPU capabilities. You can verify this by checking the node labels:

kubectl get nodes -o json | jq '.items[].metadata.labels | with_entries(select(.key | startswith("nvidia.com/")))'

Check NVIDIA GPU Operator Status

Verify that the NVIDIA GPU Operator components are running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=nvgpu-operator

You should see various NVIDIA operator pods running:

NAME                                       READY   STATUS    RESTARTS   AGE
gpu-feature-discovery-fdxsc                1/1     Running   0          24h
nvgpu-dcgm-exporter-ms5zv                  1/1     Running   0          25h
nvgpu-operator-64db85f5-rr7qb              1/1     Running   0          25h
nvidia-container-toolkit-daemonset-9vtxn   1/1     Running   0          27h
nvidia-mig-manager-bcbb4                   1/1     Running   0          27h
nvidia-operator-validator-mfmt8            1/1     Running   0          24h

Note

The nvidia-device-plugin-daemonset pod only appears when DRA is disabled — on DRA-enabled clusters the device plugin is replaced by the DRA kubelet plugin. Its DaemonSet deploys a nvgpu-dra-driver-kubelet-plugin-<id> pod on each node supported by DRA; the plugin is verified separately in Verifying DRA Availability.

Check NVIDIA GPU Operator Validator

The NVIDIA GPU Operator includes a built-in validator that sanity-checks driver, device plugin, and CUDA toolkit readiness. Verify it by inspecting the logs of the nvidia-operator-validator pods:

kubectl logs -n kube-system -l app=nvidia-operator-validator -c nvidia-operator-validator

Expected output:

all validations are successful

GPU Allocation Using the Device Plugin

The device plugin advertises GPUs as nvidia.com/gpu resources on the nodes, and workloads request them through resource requests in the pod specification. The device plugin is used when the cluster runs a Kubernetes version older than 1.34.2, or when DRA is disabled.

Warning

On DRA-enabled clusters, the nvidia.com/gpu resource is not advertised and pods requesting it remain Pending. Use the examples in GPU Allocation Using DRA instead.

Verifying Device Plugin Availability

Verify that GPU resources are advertised on your nodes:

kubectl get nodes -o json |  jq -r '.items[] | .metadata.name, .status.allocatable'

On GPU nodes, the nvidia.com/gpu resource appears in the allocatable resources.

Requesting GPUs with Resource Requests

To use a GPU in your pod, specify the GPU resource and runtimeClassName in the pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-app
spec:
  runtimeClassName: nvidia
  containers:
  - name: my-container
    image: my-cuda-app:latest
    resources:
      requests:
        nvidia.com/gpu: 1

Note

Make sure you have enough resources to satisfy the requests.

Running Sample GPU Applications

To verify that GPU operations are working correctly, run a sample CUDA application.

CUDA VectorAdd Example

This example runs a simple CUDA vector addition operation to verify GPU functionality.

Create the cuda-vectoradd pod:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0"
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1
EOF

The pod starts, runs the vectorAdd command, and then exits.

View the logs from the container:

kubectl logs pod/cuda-vectoradd

Expected output:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

The Test PASSED message confirms that the GPU is working.

Remove the pod:
```
kubectl delete pods cuda-vectoradd
```

Simple nvidia-smi Test

For a quicker verification, run a pod that executes nvidia-smi:

Create the nvidia-smi pod:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  containers:
  - name: nvidia-smi
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1
EOF

View the logs from the container:

kubectl logs nvidia-smi

Expected output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A30                     Off |   00000000:06:00.0 Off |                    0 |
| N/A   28C    P0             30W /  165W |       0MiB /  24576MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

GPU Allocation Using DRA (Dynamic Resource Allocation)

Dynamic resource allocation (DRA) is the default way to allocate GPUs in Shoot clusters. When the NVIDIA GPU Operator is enabled, the NVIDIA DRA GPU driver is deployed automatically — no manual preparation is required.

With DRA, GPUs are not requested through nvidia.com/gpu resource requests. Instead, the DRA driver publishes all GPUs and MIG instances as ResourceSlice objects, and workloads request them with a ResourceClaimTemplate referencing one of the available device classes:

gpu.nvidia.com — allocates full physical GPUs
mig.nvidia.com — allocates MIG slices

Note

DRA requires Kubernetes 1.34.2 or later. On clusters with an older Kubernetes version, the device plugin is used instead, and GPUs are requested as described in GPU Allocation Using the Device Plugin.

Stay within the supported lifecycle

Customers remain responsible for scheduling upgrades on time and ensuring that their applications are compatible with the target version. We strongly recommend staying within the supported lifecycle, as it is necessary for security updates and reliable platform operation.

DRA can be disabled for a Shoot cluster through the NFD extension configuration, for example to keep using the device plugin. For details, refer to the OSC NFD Extension reference.

The examples in this section cover the common GPU allocation scenarios:

Allocate a Full GPU
Allocate a MIG Slice
Select a GPU by Capability (CEL Selector)
Atomic Multi-Instance Claim (Two MIG Slices)

Verifying DRA Availability

Verify that the DRA kubelet plugin is running on your GPU nodes:

kubectl get pods -n kube-system -l app.kubernetes.io/name=nvgpu-dra-driver

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
nvgpu-dra-driver-kubelet-plugin-x7x9k   1/1     Running   0          10m

Check that the device classes are available:

kubectl get deviceclasses

Expected output:

NAME             AGE
gpu.nvidia.com   10m
mig.nvidia.com   10m

List the GPU devices published for your nodes:

kubectl get resourceslices

Expected output:

NAME                                                        NODE                                   DRIVER           POOL                                   AGE
shoot--gpu--dra-wa301-z1-55dbb-4vqzk-gpu.nvidia.com-pllq5   shoot--gpu--dra-wa301-z1-55dbb-4vqzk   gpu.nvidia.com   shoot--gpu--dra-wa301-z1-55dbb-4vqzk   49s

Allocate a Full GPU

Use this when your workload needs exclusive access to an entire physical GPU — for example, a large model training job or a workload with high memory pressure.

Create a ResourceClaimTemplate:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: full-gpu-template
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            count: 1

kubectl apply -f full-gpu-template.yaml

Create a pod that references the template:

apiVersion: v1
kind: Pod
metadata:
  name: dra-full-gpu-test
  namespace: default
spec:
  restartPolicy: Never
  resourceClaims:
    - name: full-gpu
      resourceClaimTemplateName: full-gpu-template
  containers:
    - name: gpu-test
      image: nvidia/cuda:12.3.1-base-ubuntu22.04
      command:
        - /bin/bash
        - -c
        - |
          nvidia-smi -L
          nvidia-smi --query-gpu=name,index,uuid,memory.total --format=csv
      resources:
        claims:
          - name: full-gpu

kubectl apply -f dra-full-gpu-test.yaml

Verify that the pod is running:

kubectl get pods dra-full-gpu-test

Expected output:

NAME                READY   STATUS    RESTARTS   AGE
dra-full-gpu-test   1/1     Running   0          29s

Verify that the claim was allocated and bound to the pod:

kubectl get resourceclaims -n default

Expected output:

NAME                              STATE                AGE
dra-full-gpu-test-full-gpu-dtwrx  allocated,reserved   78s

kubectl describe resourceclaim dra-full-gpu-test-full-gpu-dtwrx -n default

Expected output:

Status:
  Allocation:
    Devices:
      Results:
        Device:   gpu-0               # Physical GPU bound to this claim — pod gets exclusive access
        Driver:   gpu.nvidia.com
        Pool:     shoot--gpu--dra-wa301-z1-55dbb-4vqzk
        Request:  gpu
    Node Selector:
      Node Selector Terms:
        Match Fields:
          Key:       metadata.name
          Operator:  In
          Values:
            shoot--gpu--dra-wa301-z1-55dbb-4vqzk
  Reserved For:
    Name:      dra-full-gpu-test
    Resource:  pods

The claim stays reserved until the pod terminates, at which point the device is released back to the pool.

Allocate a MIG Slice

Use this when multiple tenants or workloads need to share a single GPU with hardware-enforced memory and compute isolation — for example, inference services running alongside each other on an A30.

Note

MIG must be enabled and the desired profiles must be configured on the node before DRA can allocate slices — see Multi-Instance GPU for the configuration and Verifying the MIG Layout for listing the available MIG instances.

After a MIG profile change, the DRA kubelet plugin is restarted automatically and publishes the updated MIG layout in ResourceSlice objects.

Create a ResourceClaimTemplate:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-slice-template
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: mig
          exactly:
            deviceClassName: mig.nvidia.com
            count: 1

kubectl apply -f mig-slice-template.yaml

Create a pod that uses the MIG slice:

apiVersion: v1
kind: Pod
metadata:
  name: dra-mig-slice-test
  namespace: default
spec:
  restartPolicy: Never
  resourceClaims:
    - name: mig-slice
      resourceClaimTemplateName: mig-slice-template
  containers:
    - name: mig-test
      image: nvidia/cuda:12.3.1-base-ubuntu22.04
      command:
        - /bin/bash
        - -c
        - |
          nvidia-smi -L
          nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv
          sleep infinity
      resources:
        claims:
          - name: mig-slice

kubectl apply -f dra-mig-slice-test.yaml

Verify that the pod is running:

kubectl get pods dra-mig-slice-test

Expected output:

NAME                 READY   STATUS    RESTARTS   AGE
dra-mig-slice-test   1/1     Running   0          18s

Verify that the MIG instance was allocated:

kubectl describe resourceclaim dra-mig-slice-test-mig-slice-8fb26 -n default

Expected output:

Status:
  Allocation:
    Devices:
      Results:
        Device:   mig-gpu-0-1g.6gb-0  # MIG profile (1g.6gb) and instance index (0)
        Driver:   gpu.nvidia.com
        Pool:     shoot--gpu--dra-wa301-z1-55dbb-4vqzk
        Request:  mig
  Reserved For:
    Name:      dra-mig-slice-test
    Resource:  pods

Multiple pods can each hold their own MIG slice claim on the same physical GPU simultaneously, with hardware-enforced isolation between them.

Select a GPU by Capability (CEL Selector)

Use this when you need a GPU that meets a specific hardware requirement — for example, a minimum architecture generation — without hard-coding a GPU model name.

Create a ResourceClaimTemplate with a Common Expression Language (CEL) selector expression:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-min-24gi-template
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            count: 1
            selectors:
              - cel:
                  expression: >
                    device.attributes["gpu.nvidia.com"].architecture == "Ampere"

kubectl apply -f gpu-min-24gi-template.yaml

Create a pod that uses the capability-selected GPU:

apiVersion: v1
kind: Pod
metadata:
  name: dra-cel-gpu-test
  namespace: default
spec:
  restartPolicy: Never
  resourceClaims:
    - name: capable-gpu
      resourceClaimTemplateName: gpu-min-24gi-template
  containers:
    - name: cel-test
      image: nvidia/cuda:12.3.1-base-ubuntu22.04
      command:
        - /bin/bash
        - -c
        - |
          nvidia-smi --query-gpu=name,memory.total --format=csv
          sleep infinity
      resources:
        claims:
          - name: capable-gpu

kubectl apply -f dra-cel-gpu-test.yaml

Verify that the pod is running:

kubectl get pods dra-cel-gpu-test -n default

Expected output:

NAME               READY   STATUS    RESTARTS   AGE
dra-cel-gpu-test   1/1     Running   0          12s

Verify that the allocated device satisfies the expression:

kubectl describe resourceclaim dra-cel-gpu-test-capable-gpu-757qf -n default

Expected output:

Status:
  Allocation:
    Devices:
      Results:
        Device:   gpu-0               # Physical GPU that satisfied the architecture == "Ampere" expression
        Driver:   gpu.nvidia.com
        Pool:     shoot--gpu--dra-wa301-z1-55dbb-4vqzk
        Request:  gpu
  Reserved For:
    Name:      dra-cel-gpu-test
    Resource:  pods

If no device satisfies the CEL expression, the pod stays Pending with a clear scheduler event:

Events:
  Warning  FailedScheduling  ...  no ResourceSlice matches request "gpu" in ResourceClaimTemplate "gpu-min-24gi-template"

CEL attributes published by the NVIDIA DRA driver (verified on NVIDIA A30 with GPU Operator v25.10.1, DRA driver k8s-dra-driver-gpu v25.12.0, NVIDIA driver v590.48.1, CUDA 13.1):

Attribute	Type	Example expression
GPU architecture	`string`	`device.attributes["gpu.nvidia.com"].architecture == "Ampere"`
GPU product name	`string`	`device.attributes["gpu.nvidia.com"].productName == "NVIDIA A30"`
GPU brand	`string`	`device.attributes["gpu.nvidia.com"].brand == "Nvidia"`
Device type	`string`	`device.attributes["gpu.nvidia.com"].type == "gpu"` (`"mig"` for MIG slices)
Addressing mode	`string`	`device.attributes["gpu.nvidia.com"].addressingMode == "None"`
Driver version	`semver`	`device.attributes["gpu.nvidia.com"].driverVersion.compareTo(semver("525.0.0")) >= 0`
CUDA driver version	`semver`	`device.attributes["gpu.nvidia.com"].cudaDriverVersion.compareTo(semver("12.0")) >= 0`
CUDA compute capability	`semver`	`device.attributes["gpu.nvidia.com"].cudaComputeCapability.compareTo(semver("8.0")) >= 0`
Device UUID	`string`	`device.attributes["gpu.nvidia.com"].uuid == "GPU-61d77dc7-7671-3234-2450-e66940b99acb"`
PCI bus ID	`string`	`device.attributes["resource.kubernetes.io/pciBusID"] == "0000:06:00.0"`

Note

GPU memory is published as a device capacity, not an attribute. To select a GPU by minimum memory, use the capacity form:

device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0

Run kubectl get resourceslices -o json | jq '.items[0].spec.devices[0] | {attributes, capacity}' to confirm the attributes and capacities available on your cluster before writing CEL expressions.

Atomic Multi-Instance Claim (Two MIG Slices)

Use this when a workload needs exactly two MIG slices allocated simultaneously — for example, a pipeline that runs two independent inference stages in separate containers but must guarantee both hardware slices are available on the same node before the pod starts. With the device plugin model, one slice may be reserved while the second is unavailable. DRA solves this atomically.

Note

MIG must be enabled and profiles configured on the node before this claim can be satisfied. For the MIG configuration and profile setup, see Multi-Instance GPU; verify the available slices as described in Verifying the MIG Layout.

Create a ResourceClaimTemplate that requests two MIG instances in a single claim:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: two-mig-template
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: mig
          exactly:
            deviceClassName: mig.nvidia.com
            count: 2

kubectl apply -f two-mig-template.yaml

Create a pod with two containers, each consuming one MIG slice from the shared claim:

apiVersion: v1
kind: Pod
metadata:
  name: dra-two-mig-test
  namespace: default
spec:
  restartPolicy: Never
  resourceClaims:
    - name: two-mig-slices
      resourceClaimTemplateName: two-mig-template
  containers:
    - name: stage-a
      image: nvidia/cuda:12.3.1-base-ubuntu22.04
      command:
        - /bin/bash
        - -c
        - |
          nvidia-smi -L
          nvidia-smi --query-gpu=index,name,uuid,memory.total --format=csv
          sleep infinity
      resources:
        claims:
          - name: two-mig-slices
    - name: stage-b
      image: nvidia/cuda:12.3.1-base-ubuntu22.04
      command:
        - /bin/bash
        - -c
        - |
          nvidia-smi -L
          nvidia-smi --query-gpu=index,name,uuid,memory.total --format=csv
          sleep infinity
      resources:
        claims:
          - name: two-mig-slices

kubectl apply -f dra-two-mig-test.yaml

Verify that the pod is running:

kubectl get pods dra-two-mig-test

Expected output:

NAME               READY   STATUS    RESTARTS   AGE
dra-two-mig-test   2/2     Running   0          21s

Verify that both MIG instances were allocated atomically on the same node:

kubectl describe resourceclaim dra-two-mig-test-two-mig-slices-p7kx -n default

Expected output:

Status:
  Allocation:                           # Both devices allocated in a single atomic scheduler decision
    Devices:
      Results:
        Device:   gpu-0-mig-1g6gb-14-0  # First MIG slice — instance 0
        Driver:   gpu.nvidia.com
        Pool:     shoot--gpu--dra-wa302-z1-5bc87-7s2z6
        Request:  mig
        Device:   gpu-0-mig-1g6gb-14-1  # Second MIG slice — instance 1, same physical GPU
        Driver:   gpu.nvidia.com
        Pool:     shoot--gpu--dra-wa302-z1-5bc87-7s2z6   # Identical pool — co-location guaranteed
        Request:  mig
    Node Selector:
      Node Selector Terms:
        Match Fields:
          Key:       metadata.name
          Operator:  In
          Values:
            shoot--gpu--dra-wa302-z1-5bc87-7s2z6
  Reserved For:
    Name:      dra-two-mig-test
    Resource:  pods

Both slices are on the same pool, confirming co-location. If the node does not have two free MIG instances simultaneously, the pod stays Pending:

Events:
  Warning  FailedScheduling  ...  0/3 nodes are available: 3 Insufficient mig.nvidia.com (DRA).

Support Status

Feature	Status
GPU allocation via DRA (full GPU, MIG slices, CEL selectors)	Supported, enabled by default (Kubernetes >= 1.34.2)
Automatic fallback to the device plugin (Kubernetes < 1.34.2)	Supported
ComputeDomains (multi-node NVLink)	Not available

Multi-Instance GPU

MIG (Multi-Instance GPU) is an NVIDIA technology that slices a single physical GPU into multiple isolated GPU instances. Each instance has dedicated compute, memory, and cache, so different workloads can run securely without resource contention. In Kubernetes, the GPU Operator’s MIG Manager applies the desired layout on nodes, and GPU Feature Discovery advertises the resulting MIG resources for scheduling.

The MIG layout is configured in the same way for both allocation patterns — only the verification and the consumption in workloads differ.

Applying the MIG Configuration

Warning

Using a different MIG config is disruptive and causes pods on the node to be restarted or even removed.

You must select the correct MIG profile for the specific GPU model; for the list of profiles, refer to the NVIDIA Supported MIG Profiles reference.

With the default single MIG strategy, the GPU is partitioned into instances of one identical profile (for example all-1g.6gb). To split a GPU into instances with different profiles (for example all-balanced), the MIG strategy of the NVIDIA GPU Operator must be changed to mixed first, through a values override in the providerConfig of the NFD extension in the Shoot manifest:

spec:
  extensions:
    - type: osc-nfd-shoot-service
      providerConfig:
        apiVersion: nfd.osc.extensions.config.gardener.cloud/v1alpha1
        kind: Configuration
        nvgpu-operator:
          enabled: true
          values: |
            mig:
              strategy: mixed

Configure the MIG strategy before labeling the node

For MIG configurations with multiple different profiles, such as all-balanced, the mixed strategy must be configured in the NVIDIA GPU Operator and the Shoot cluster reconciled before the MIG configuration label is applied to the GPU node. With the mixed strategy, the MIG instances are advertised as individual nvidia.com/mig-<profile> resources instead of a generic nvidia.com/gpu resource count. For details about the mixed strategy, refer to the Example: Mixed MIG Strategy reference.

Set the MIG layout via a node label. This example uses the MIG config all-1g.6gb, which partitions the physical GPU into four isolated MIG instances:
```
kubectl label nodes <node> nvidia.com/mig.config=all-1g.6gb --overwrite
```
Wait for the NVIDIA GPU Operator to reconcile.

The nvidia-mig-manager DaemonSet reconfigures MIG on each node if applicable.
Verify the new layout as described in Verifying the MIG Layout.

To disable MIG and revert to using the full physical GPU, change the node label to all-disabled:

kubectl label nodes <node> nvidia.com/mig.config=all-disabled --overwrite

Verifying the MIG Layout

With the Device Plugin

The MIG instances appear in the allocatable resources of the node:

kubectl get node <node> -o json | jq '.status.allocatable | with_entries(select(.key|test("nvidia.com")))'

Expected output:

{
  "nvidia.com/gpu": "4"
}

With DRA

The DRA driver publishes each MIG instance as a device in the ResourceSlice objects:

kubectl get resourceslices -o json | jq '.items[].spec.devices[].name' | grep mig

Expected output:

"gpu-0-mig-1g6gb-14-0"
"gpu-0-mig-1g6gb-14-1"
"gpu-0-mig-1g6gb-14-2"
"gpu-0-mig-1g6gb-14-3"

Using MIG Slices in Workloads

Requesting GPUs with Resource Requests — with the device plugin and the default single MIG strategy, MIG instances are requested like full GPUs through nvidia.com/gpu resource requests
Allocate a MIG Slice — with DRA, MIG slices are requested through the mig.nvidia.com device class

If the MIG strategy is set to mixed — required for MIG configurations with multiple different profiles, such as all-balanced — the MIG instances are requested as individual nvidia.com/mig-<profile> resources instead. For changing the MIG strategy, refer to the values override example in the OSC NFD Extension reference.

Advanced Configuration

Running the NVIDIA GPU Operator on tainted nodes

Nodes with GPUs can be reserved exclusively for GPU workloads by tainting them, so that regular workloads are not scheduled on these nodes.

Apply a NoSchedule taint to the GPU node. In this example, the taint gpu="true" is used:

kubectl taint nodes <node> gpu=true:NoSchedule

From now on, only pods with a matching toleration are scheduled on this node.

The infrastructure resources of the NVIDIA GPU Operator are deployed with the catch-all toleration operator: Exists, which tolerates all taints with all effects — the DaemonSets managed by the NVIDIA GPU Operator are deployed on tainted GPU nodes automatically.

No manual toleration configuration required

In previous versions, the toleration for the DaemonSets managed by the NVIDIA GPU Operator had to be configured manually through the NFD extension in the Shoot manifest. This configuration is no longer needed — no action is required from you to run the NVIDIA GPU Operator on tainted GPU nodes.

Custom tolerations overwrite the default

Tolerations configured manually under daemonsets.tolerations in the nvgpu-operator values overwrite the default catch-all toleration. An incorrect custom configuration can prevent the infrastructure pods of the NVIDIA GPU Operator from being scheduled on the tainted node, leaving the GPU stack on that node non-functional.

Note

The taint affects all pods, not only the NVIDIA GPU Operator components. GPU workloads scheduled on the tainted node must also specify a matching toleration in their pod specification.

For details about the NFD extension configuration, refer to the OSC NFD Extension reference.

References

NVIDIA GPU Operator Documentation
NVIDIA GPU Operator with MIG
NVIDIA DRA Driver for GPUs (v25.12.0 is deployed with GPU Operator v25.10.1)
Kubernetes Dynamic Resource Allocation
OSC NFD Extension documentation