Patching and Updating Azure Kubernetes Service (AKS)

When running AKS it's a good idea to familiarize yourself with the various update processes. Updates to Kubernetes are released frequently. If you don't stay on top of things, you'll quickly find yourself on an unsupported version.

Updates to AKS ecosystem come in 2 different forms:

Node Security Channel
1. This involves security patches and bug fixes to the underlying AKS nodes / virtual machines.
2. In Bicep/ARM this option is called "nodeOSUpgradeChannel"
Automatic Upgrade
1. This involves upgrading the version of AKS itself.
2. In Bicep/ARM this option is called "upgradeChannel"

It's best to use both types of updates together. Scheduling can be fine-tuned, since both types of updates can have their own unique maintenance window.

This post is going to discuss the different options and the differences between them. This post will NOT go into detail on how to actually perform the updates.

Node Security Channel

This provides timely node-level OS security updates. Node-level OS security updates are released at a faster rate than Kubernetes version updates.

The available options are:

Unmanaged
- Linux nodes will apply OS security patches automatically through built-in OS patching mechanisms.
- This happens roughly once per day around 06:00 UTC.
- Some patches will require a reboot of the node. Rebooting the nodes is your responsibility, and you can manage this with a tool like kured. See the section below for more information on kured.
- Windows nodes will not automatically apply OS security patches, so this option is the same as None for Windows nodes.
SecurityPatch
- Microsoft will fully test and release OS security patches for your nodes.
- Typically, Microsoft does this at a cadence that is faster than once a week.
- Nodes are only re-imaged when it is absolutely necessary, like for certain kernel patches. When this happens, your configured maintenance window is honored.
- If a patch does not require re-imaging, then the nodes are live-patched, without draining any Pods from the nodes.
- This option requires that you store the node VHD images in your existing resource groups, which will cost a nominal amount.
- This option is not supported for Windows nodes.
NodeImage
- The is the default option for new clusters.
- Microsoft provides fully tested node images on a weekly basis for Linux, and a monthly basis for Windows.
- A node image contains up-to-date OS security patches, OS kernel updates, Kubernetes security updates, updated versions of binaries like kubelet, and component version updates that are listed in the release notes.
- These images do not change the Kubernetes version.
- When nodes are re-imaged your configured maintenance window is honored.
- Use the AKS Release Tracker to find out which images are currently available in each Azure region.
None
- You're responsible for updates, as the nodes will not automatically apply any updates with this option.

	Unmanaged	SecurityPatch	NodeImage
Schedule	Daily	Quicker than weekly	Weekly (Linux) Monthly (Windows)
Updates	OS Security	OS Security	OS Security, K8s security, K8s components
Maintenance Window	No	Yes	Yes
Windows	No	No	Yes

Kured

When using the Unmanaged option, it is your responsibility to reboot your nodes after patching. You will know when a Node requires a reboot because it will create a new file on its disk named /var/run/reboot-required.

There is a commonly used open-source tool called “kured” that is helpful in this situation. Kured stands for KUbernetes REboot Daemon. As the name suggests, it is deployed into your cluster as a DaemonSet, which means it will run one Pod on each Node.

Kured will automatically scan each Node for the existence of the file /var/run/reboot-required. When kured finds this file, it will initiate the process to reboot the Node. Kured does the reboot process in a safe manner. It ensures that only one Node reboots at a time. It also cordons and drains each Node before the reboot. Finally, after the reboot is complete it will uncordon the Node.

Automatic Upgrade

This provides updates to your AKS version. Auto-upgrade first upgrades the control plane, and then upgrades agent pools one by one. By enabling auto-upgrade, you can ensure your clusters are up to date and don't miss the latest features or patches from AKS and upstream Kubernetes.

AKS Minor Version Support

Microsoft supports AKS minor versions using an N-2 standard. That means they will support the current Minor version, as well as the previous 2 Minor versions. For example:

1.23.x = Latest supported minor version
1.22.x = Supported (N-1)
1.21.x = Supported (N-2)
1.20.x = No longer supported

AKS Patch Version Support

Microsoft supports AKS patches using a 2 month minimum lifecycle. Once a patch version is no longer supported by AKS, you have 30 days to upgrade to a supported patch version.

AKS Cluster Auto-Upgrade

Finally, let's talk about the available options for auto-upgrade:

Patch
- This is the Microsoft-recommended option.
- Automatically updates to the latest supported AKS Patch version within your current Minor version
Stable
- Automatically updates the AKS Minor version, always keeping you at N-1 (one Minor version behind)
Rapid
- Automatically updates the AKS Minor version, always keeping you on the most current Minor version (latest and greatest)
Node-Image
- This option is considered legacy and is planned to be deprecated.
- If you choose this option for Auto-upgrade, it will also force your Node Security Channel to use the NodeImage option and that can not be changed.
None
- This is the default option for new clusters.
- Always keeps the cluster at its current AKS version.
- You must manually upgrade yourself.

Note: Preview versions are not used with Auto-Upgrade

Nathan Nellans