Idle CPU usage of k3s is high and continuously increases over time

This post is a follow up of a previous post where I deployed k3s on Proxmox VMs. https://canthonyscott.com/setting-up-a-k3s-kubernetes-cluster-within-proxmox/

Given that my k3s cluster is run within virtual machines on a single host (in my closet) the cluster is mostly idle. I use it infrequently to test or experiment with things. The only real deployment I run on it is a Cronjob hourly.

However, over time I have regularly noticed its resource usage slowly creeps up and up until this idle cluster is using obnoxiously high levels of CPU.

I have no  idea what's its actually doing.

This isn't the worst I have found it behaving, but this shows it happening over the course of about a week.

Here is the slow climb of CPU use on the master node

And we can see the same CPU use climb on the agent node

I have been able to find reports online from others about k3s having high levels of CPU usage when idle, but unfortunately there is never really a resolution to these posts.

Here are some issues from GitHub and a thread on Reddit.

k3s causes a high load average · Issue #294 · k3s-io/k3s

Describe the bug I'm not sure if it's a bug, but I think it's not an expected behaviour. When running k3s on any computer, it causes a very high load average. To have a concrete example...

GitHubk3s-io

High CPU and Memory Load on single node k3s cluster · Issue #2375 · k3s-io/k3s

HI, i have an ubuntu 20.02 machine, which has k3s installe. After installing rancher, the memory and cpu consumption of the k3s service is skyrocketing. versions: "Ubuntu 20.04.1 LTS" - e...

GitHubk3s-io

Some reports claim that it appears to be an issue with 1.21. I did just upgrade to 1.22 today, we will see if this changes anything.

This can be resolved, temporarily, by restarting each node, but the cycle then starts again.

Either way, its very interesting and probably not-ideal behavior for a kubernetes solution that is designed to run lightweight on low-end hardware.