Configuring Cilium Networking for Your Shoot Cluster

This section explains how to enable the Cilium networking extension for a Gardener Shoot and describes every option you can set in the Cilium NetworkConfig. It also covers the routing modes (VXLAN, Geneve and native routing) and when to use each one.

Enabling Cilium

Cilium is currently the only CNI extension provided by OSC and is enabled by default for every Shoot created on OSC. To verify that Cilium is enabled, check the Shoot manifest for the label networking.extensions.gardener.cloud/cilium: 'true' and ensure that the networking section defines all necessary CIDRs.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-cluster
  namespace: garden-my-project
  labels:
  ...
  networking.extensions.gardener.cloud/cilium: 'true'
  ...
spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    # ipFamilies: [IPv4]          # IPv4 (default)
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig

providerConfig is optional. Without it, Cilium runs with the defaults described here

Shoot field	Meaning
`spec.networking.type`	Must be `cilium`.
`spec.networking.pods`	Pod CIDR. Cilium also uses it as the native routing CIDR when native routing is active.
`spec.networking.nodes`	Node CIDR. Cilium uses it for out-of-cluster SNAT.
`spec.networking.services`	Service CIDR.
`spec.networking.ipFamilies`	Must be `[IPv4]`

OSC recommendation

Concrete configuration recommendet by OSC is described here

Defaults applied automatically

Area	Default behaviour
kube-proxy	When a new Shoot is created, the extension sets `spec.kubernetes.kubeProxy.enabled: false`, and Cilium takes over the kube-proxy's job (`kube-proxy-replacement: true`). Existing Shoots keep their current setting.
Kubernetes API endpoint for Cilium	When kube-proxy is disabled, Cilium connects to the API server through the Shoot's external advertised address.
Routing	Depends on `overlay` and `tunnel`. See Routing modes and tunnels.
Masquerading	IPv4 and IPv6 masquerading are enabled. BPF-based masquerading is off, so iptables does the masquerading.
Load balancing	default value is `snat` load balancing mode.
DNS	CoreDNS is used for internal in-cluster resolution.
cilium-operator HA	The operator runs 2 replicas only if the `minimum` values of all worker pools add up to at least 2. Otherwise it runs 1 replica.
MTU	Detected automatically.
Hubble, debug, encryption, egress gateway, BGP, L2 announcements, policy audit mode	Disabled.
metrics-server policy	A `CiliumNetworkPolicy` called `metric-server-cilium` is created in `kube-system`. It allows `remote-node` ingress to `metrics-server`.

OSC recommendation

OSC strongly recommends using the default setting for kubeProxy (kubeProxy.enabled: false).

Routing modes and tunnels

Cilium can move pod-to-pod traffic between nodes in two ways:

Encapsulation (tunnel): Cilium wraps each packet from a pod in an extra UDP header and sends it to the other node's IP address. The underlying network only sees node IPs.
Native routing (no tunnel): Packets leave the node with the real pod IP as the source and destination. The underlying network must know how to route the pod CIDRs.

Two fields control this: overlay.enabled and tunnel.

What each tunnel value does

`tunnel: vxlan`

How it works: Cilium wraps each pod packet in a VXLAN header and sends it over UDP. Cilium uses UDP port 8472 by default.
Overhead: About 50 bytes per packet, so the pod MTU is lower than the node MTU. Cilium adjusts this automatically.
Network requirements: Nodes need IP connectivity to each other, and UDP 8472 must be open between them. The infrastructure doesn't need any routes for pod CIDRs.
Use it when: You need something that works on any infrastructure. The underlying network can't or shouldn't know pod IPs, or the nodes are in different L2 segments or subnets.
Trade-offs: Encapsulation adds a little CPU cost and lowers the MTU. Outside the cluster, pod IPs are hidden behind node IPs.

`tunnel: geneve`

How it works: Cilium wraps each pod packet in a Geneve header and sends it over UDP port 6081.
Overhead: Similar to VXLAN, about 50 bytes or more. Geneve also supports extensible option headers.
Network requirements: Nodes need IP connectivity to each other, and UDP 6081 must be open between them.
Use it when: You want an overlay but your environment or security policy prefers Geneve, or you plan to use Cilium features that rely on Geneve options.
Trade-offs: Same as VXLAN. Some older network devices and firewalls handle VXLAN better.

`tunnel: disabled` (native routing)

How it works: No encapsulation. Cilium sets routing-mode: native, and packets carry the real pod IPs on the wire.
Overhead: None. You get the full MTU and the best throughput and latency.
Network requirements: The underlying network must route the pod CIDR (spec.networking.pods) to the correct node.
Use it when: Performance matters and your infrastructure can route pod IPs.
Trade-offs: Pod traffic fails if the network can't route pod CIDRs. Egress Gateway is not available.

OSC recommendation

In OSC infrastructure, native routing is supported and highly recommended. Therefore, unless there is a technical reason to use a different mode, use native routing what means tunnel: disabled. For IPv4 native routing, the Shoot’s Pod CIDR is used as the native routing CIDR. A Variant with tunnel: disabled is the only variant that is fully tested on OSC infrastructure. If it is used a different mode, for example tunnel: geneve or vxlan than OSC cannot guarantee proper performance and reliability.

How `overlay` and `tunnel` combine

Cilium decides the effective routing from both fields:

`overlay.enabled`	`tunnel`	Effective routing mode	Native routing CIDR set automatically
(not set)	(not set)	VXLAN tunnel	No
(not set)	`vxlan`	VXLAN tunnel	No
(not set)	`geneve`	Geneve tunnel	No
(not set)	`disabled`	Native routing	No. You must also set `ipv4NativeRoutingCIDREnabled: true`. Otherwise the Cilium agent won't start.
`true`	(not set) or `disabled`	Native routing	yes (pod CIDR)
`true`	`vxlan`	VXLAN tunnel	No
`true`	`geneve`	Geneve tunnel	No
`false`	`disabled` or (any value)	Native routing	yes (pod CIDR)

Overlay tunnel config mismatch

In this distribution, overlay.enabled: true without an explicit tunnel value or with tunnel: disabled gives you native routing, not VXLAN. If you need encapsulation together with overlay.enabled: true, set tunnel: vxlan or tunnel: geneve explicitly.

With overlay.enabled: false, any tunnel value you set is overwritten with disabled.

Load balancing modes

These modes only apply when Cilium replaces kube-proxy.

Mode	Behaviour	When to use
`snat` (default)	The node that receives the request SNATs it to the backend. Replies return through the same node.	Works on any network.
`dsr`	Direct Server Return. The backend replies to the client directly and keeps the client source IP.	Lowest latency, and the backend sees the client IP. Requires native routing.
`hybrid`	DSR for TCP, SNAT for UDP.	Compromise when UDP traffic doesn't work well with DSR.

OSC recomended setup

OSC recommends network settings that make the most of our infrastructure capabilities. Using native routing with direct server return (DSR) load balancing provides optimized traffic management, improved scalability, and superior network performance without the need for tunneling protocols.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-cluster
  namespace: garden-my-project
spec:
  kubernetes:
    kubeProxy:
      enabled: false
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      debug: false
      overlay:
        enabled: false     <--- Ensuring native routing is active
      tunnel: disable      <--- Ensuring native routing is active
      bpfSocketLBHostnsOnly:
        enabled: true
      cni:
        exclusive: true
      loadBalancingMode: dsr  <---- Allowing DSR load balancing mode
  provider:
    workers:
      - name: worker
        minimum: 2   # >= 2 enables cilium-operator HA
        maximum: 5

`NetworkConfig` options in `providerConfig` section

Field	Type	Default	Description
`debug`	bool	`false`	Turns on debug logging for the Cilium agent.
`hubble.enabled`	bool	`false`	Deploys Hubble, Hubble Relay and Hubble UI for flow observability.
`tunnel`	`vxlan` \| `geneve` \| `disabled`	`vxlan`	Encapsulation mode. See Routing modes.
`overlay.enabled`	bool	not set	Switches between overlay and native routing - if tunnel field not configured than `overlay.enabled: true` has the same effect as `overlay.enabled: true` . See the combination table.
`overlay.createPodRoutes`	bool	not set	Installs direct node-to-node pod routes. Only applies when `overlay.enabled: false`, and requires a shared L2 network.
`store`	`kubernetes`	`kubernetes`	Identity store. Only `kubernetes` (CRD) is supported.
`kubeproxy.k8sServiceHost`	string (IP or DNS name)	Shoot external API address	API server host that Cilium uses when kube-proxy is disabled. It only takes effect if `k8sServicePort` is also set.
`kubeproxy.k8sServicePort`	int (1–65535)	—	API server port that Cilium uses when kube-proxy is disabled. It only takes effect if `k8sServiceHost` is also set.
`bpfSocketLBHostnsOnly.enabled`	bool	`false`	Uses socket-level load balancing only in the host namespace. Pods use service load balancing at their interface instead. Required for service meshes such as Istio or Linkerd.
`cni.exclusive`	bool	`true`	If `false`, Cilium won't overwrite other CNI config files, so plugins such as Istio CNI or Linkerd CNI can chain after Cilium.
`egressGateway.enabled`	bool	`false`	Turns on Cilium Egress Gateway. Requires kube-proxy to be disabled and a tunnel (`vxlan`/`geneve`).
`mtu`	int (≥ 0)	auto	Overrides the MTU that Cilium detects automatically. `0` means auto.
`devices`	[]string	auto	Network devices that face the cluster or external network. Cilium attaches its BPF NodePort, host firewall and masquerade programs to them.
`directRoutingDevice`	string	auto	Device used for direct routing between nodes, for example by NodePort/DSR.
`loadBalancingMode`	`snat` \| `dsr` \| `hybrid`	`snat`	Service load-balancing mode. See Load balancing modes.
`ipv4NativeRoutingCIDREnabled`	bool	depends on `overlay`	Sets `ipv4-native-routing-cidr` to the Shoot pod CIDR. Traffic to this CIDR isn't masqueraded.
`ipv6NativeRoutingCIDREnabled`	bool	depends on `overlay`	Sets `ipv6-native-routing-cidr` to the Shoot's IPv6 pod CIDR.
`enableIpv4Masquerade`	bool	`true`	Masquerades IPv4 pod traffic that leaves the node towards destinations outside the native routing CIDR.
`enableIpv6Masquerade`	bool	`true`	Same as above, for IPv6.
`enableBPFMasquerade`	bool	`false`	Uses eBPF instead of iptables for masquerading. Ignored if `snatToUpstreamDNS` or `snatOutOfCluster` is enabled.
`snatToUpstreamDNS.enabled`	bool	`false`	Masquerades pod traffic to the node's upstream DNS servers (from `/etc/resolv.conf`). Use this when the infrastructure blocks DNS requests that come from pod IPs.
`snatOutOfCluster.enabled`	bool	`false`	Masquerades all pod traffic to destinations outside the pod and node CIDRs, using iptables.
`l2Announcements.enabled`	bool	`false`	Turns on Cilium L2 announcements (ARP replies for LoadBalancer/External IPs).
`l2Announcements.leaseDuration`	duration	Cilium default	Maximum lease duration, for example `15s`.
`l2Announcements.leaseRenewDeadline`	duration	Cilium default	How long the current leader keeps retrying a lease renewal before it gives up.
`l2Announcements.leaseRetryPeriod`	duration	Cilium default	Wait time between lease retries.
`bgpControlPlane.enabled`	bool	`false`	Turns on the Cilium BGP Control Plane.
`policyAuditMode`	bool	`false`	Logs policy verdicts without enforcing them. Network policies aren't enforced while this is on.
`encryption.enabled`	bool	`false`	Turns on transparent encryption.
`encryption.mode`	`wireguard`	—	Encryption backend. Only `wireguard` is supported.
`encryption.nodeToNodeEnabled`	bool	`false`	Also encrypts host-to-host (node) traffic.
`encryption.strictMode`	bool	`false`	Drops unencrypted pod-to-pod traffic. Only valid with `mode: wireguard`.

Feature guides

Hubble (observability)

hubble:
  enabled: true

To open the UI locally:

kubectl port-forward -n kube-system deployment/hubble-ui 8081
# open http://localhost:8081

Running without kube-proxy

New Shoots run without kube-proxy by default, and Cilium handles all Service, NodePort and LoadBalancer traffic in eBPF.

Service mesh (Istio )

bpfSocketLBHostnsOnly:
  enabled: true
cni:
  exclusive: false

Istio sidecars often expect traffic handling in a way that can conflict with Cilium’s socket-level load balancing inside pod namespaces.

L2 announcements

l2Announcements:
  enabled: true
  leaseDuration: 15s
  leaseRenewDeadline: 5s
  leaseRetryPeriod: 2s

After that, create CiliumL2AnnouncementPolicy and CiliumLoadBalancerIPPool resources in the Shoot. This feature requires kube-proxy replacement.

Complete examples

1. Minimal (VXLAN tunnel, default)

spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13

2. VXLAN overlay with Hubble

spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: true
      tunnel: vxlan
      hubble:
        enabled: true

3. Geneve overlay

    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: true
      tunnel: geneve

4. Native routing (infrastructure routes pod CIDRs)

    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: false

6. Native routing with DSR load balancing

spec:
  kubernetes:
    kubeProxy:
      enabled: false
  networking:
    type: cilium
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: false
      loadBalancingMode: dsr
      tunnel: disable

Validation rules and common errors

Rule / message	Cause	Fix
`networking type "<x>" is not supported`	`spec.networking.type` isn't `cilium`.	Set `type: cilium`.
`unsupported value ... for tunnel`	`tunnel` isn't `vxlan`, `geneve` or `disabled`.	Use one of the supported values.
`unsupported value ... for store`	`store` isn't `kubernetes`.	Remove the field or set it to `kubernetes`.
`unsupported value ... for loadBalancingMode`	Unsupported value.	Use `snat`, `dsr` or `hybrid`.
`mtu must be a positive integer`	Negative MTU.	Use `0` (auto) or a positive value.
`device name ...`	Invalid name in `devices`/`directRoutingDevice`.	Use 1–15 characters, with no `/` or whitespace.
`serviceHost is neither a valid IP address nor a valid domain name`	Invalid `kubeproxy.k8sServiceHost`.	Use an IP address or a DNS-1123 name.
`servicePort must be between 1 and 65535`	Port out of range.	Fix the port.
`strict mode can only be used with wireguard`	`encryption.strictMode: true` with a different mode.	Set `mode: wireguard`.
`ipFamilies ... cannot have duplicate entries` / `more than two entries`	Invalid `ipFamilies`.	Use `[IPv4]`, `[IPv6]` or `[IPv4, IPv6]`.
`Running kube-proxy with IPVS mode is forbidden in conjunction with node local dns enabled`	IPVS mode + node-local DNS.	Disable kube-proxy, switch to iptables mode, or disable node-local DNS.
`cilium must run with an overlay network in order to enable egressGateway`	Egress Gateway with native routing.	Set `tunnel: vxlan`/`geneve` explicitly.
`kubeProxyReplacement must be set to 'strict' or 'true' in order to enable egressGateway`	Egress Gateway with kube-proxy enabled.	Set `spec.kubernetes.kubeProxy.enabled: false`.

Troubleshooting

# Cilium agent status on every node
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose

# Effective routing mode / tunnel protocol
kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'routing-mode|tunnel-protocol|native-routing-cidr|auto-direct-node-routes'

# Connectivity between nodes
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --all-health

Symptom	Likely cause
Pods on different nodes can't reach each other in native routing mode	The infrastructure doesn't route pod CIDRs. Use a tunnel, or `createPodRoutes: true` on a shared L2 network.
Pods on different nodes can't reach each other in tunnel mode	UDP 8472 (VXLAN) or 6081 (Geneve) is blocked between nodes.
Large packets are dropped or connections hang	MTU mismatch. Set `mtu` explicitly.
DNS lookups from pods time out	The upstream DNS server rejects pod source IPs. Enable `snatToUpstreamDNS`.
Cilium agent doesn't start with `tunnel: disabled`	No native routing CIDR. Use `overlay.enabled: false`, or set `ipv4NativeRoutingCIDREnabled: true`.
Network policies have no effect	`policyAuditMode: true` is set.

Configuring Cilium Networking for Your Shoot Cluster

Enabling Cilium

Defaults applied automatically

Routing modes and tunnels

What each tunnel value does

tunnel: vxlan

tunnel: geneve

tunnel: disabled (native routing)

How overlay and tunnel combine

Load balancing modes

OSC recomended setup

NetworkConfig options in providerConfig section

Feature guides

Hubble (observability)

Running without kube-proxy

Service mesh (Istio )

L2 announcements

Complete examples

1. Minimal (VXLAN tunnel, default)

2. VXLAN overlay with Hubble

3. Geneve overlay

4. Native routing (infrastructure routes pod CIDRs)

6. Native routing with DSR load balancing

Validation rules and common errors

Troubleshooting

`tunnel: vxlan`

`tunnel: geneve`

`tunnel: disabled` (native routing)

How `overlay` and `tunnel` combine

`NetworkConfig` options in `providerConfig` section