Skip to content

Container logs rotation and retention

It is possible to configure the amount of logs kept for a container by adding the following standard Kubelet configuration options to the Shoot manifest.

  • ContainerLogMaxSize -- the maximum size of the container log file before it is rotated
  • ContainerLogMaxFiles -- the maximum number of rotated container log files that can be present for a container

Warning

The Shoot worker nodes are configured with single root filesystem partition. This includes /var/log directory as well which can result in NodeDiskPressure errors if container's logging will be very verbose. This behaviour can cause pod eviction in some edge cases.

Kubelet configuration

It is possible to configure the containerLogMaxSize and containerLogMaxFiles fields in the Shoot specification. Both fields are optional. The default values are containerLogMaxSize: 10Mi and containerLogMaxFiles: 5

Note

In the majority of the cases, the defaults should do just fine. Custom configuration might be of use under rare conditions.

For example, in a case when application generates very verbose logs and consumes filesystem space.

Example of custom specification:

spec:
  provider:
    workers:
      - cri:
          name: containerd
        kubernetes:
          kubelet:
            containerLogMaxSize: 10Mi
            containerLogMaxFiles: 10

The values of the containerLogMaxSize and containerLogMaxFiles fields need to be considered with care since container log files claim disk space from the host.

On the opposite side, log rotations on too small sizes may result in frequent rotations which can be missed by other components observing these rotations.

Note

Only the contents of the latest log file are available through kubectl logs.

For example, if a Pod writes 40 MiB of logs and the kubelet rotates logs after 10 MiB, running kubectl logs returns at most 10Mi of data. Further details can be found in the kubernetes documentation.