AKS Automatic vs The AKS Checklist

Kubernetes is often considered complex, especially for those new to container orchestration and microservices architecture. This complexity arises from factors such as networking, security, cluster maintenance, and a steep learning curve, among others.

Azure Kubernetes Service (AKS) Automatic is a new SKU that simplifies the most common tasks associated with managing your AKS clusters while retaining the extensibility needed to adapt to evolving customer requirements. This new SKU ensures that your cluster is production-ready by default, with built-in best practices and a streamlined ‘code-to-Kubernetes’ experience.

But what does ‘production-ready by default’ mean? Let’s explore AKS Automatic through the lens 🔎 of the well-known AKS Checklist.

TL;DR

AKS automatic significantly accelerates AKS provisioning by streamlining the most common Kubernetes configuration tasks, making the process frictionless. It enables you to go from zero to a highly observable and well-configured Kubernetes cluster in minutes. AKS automatic incorporates numerous best practices, satisfying 26 out of the 33 high-priority controls we evaluated.

Does this means that is production ready by default? Well … it depends!

Get a K8S cluster up and running

To stick with the defaults, I provisioned an AKS cluster using a minimal command. A simple az aks create was all it took: “az aks create —resource-group rg-aks-automatic —name aksautomaticdefault —sku automatic ”

Now let’s go through the AKS checklist, with focus on the critical and medium controls. For this exercise I’ve skipped the “Windows”, “Application development” and “Image management” sections as they don’t apply to the overall goal .

Identity and authorization

💎 Integrate authentication with AAD (using the managed integration): Azure Kubernetes Service (AKS) can be configured to use Azure Active Directory (Azure AD) for user authentication. In this configuration, you can sign in to an AKS cluster by using your Azure AD authentication token.

✅ Let’s go through the json output produced by the AKS creation command. The integration with EntraID (AAD) is enabled: “aadProfile”: { “managed”: true, …

💎Integrate authorization with AAD RBAC: Control access to cluster resources using Kubernetes role-based access control and Azure Active Directory identities in Azure Kubernetes Service

✅ Azure RBAC is enabled “aadProfile”: { “enableAzureRbac”: true, … You can just use it by assigning the built-in AKS roles to give user access to the cluster, or for example restrict the rights to a namespace.

Note: The adminGroupObjectIDs is null, it means no specific AAD group has been designated for admin rights, and access is controlled solely through Azure RBAC roles. Admins need to be assigned roles like “Azure Kubernetes Service RBAC Cluster Admin” to access the AKS resources.

💎 Use managed identities instead of Service Principals: Each AKS cluster needs either a Managed Identity or Service Principal. We recommend using Managed Identity in AKS.

✅ AKS automatic leverages user assigned managed identities to access other Azure resources such as load balancers, managed disks, and more.

💎 Use AKS and ACR integration without password: AKS can authenticate to ACR without using any password, but by using either Service Principal or Managed Identity.

❌ AKS Automatic does not create a new ACR nor automatically attach it to the cluster. However, remember that ‘Automatic’ is just a SKU, so you can easily run the create command with the —attach-acr parameter to enable integration with ACR using Managed Service Identity (MSI).

💎 Disable AKS local accounts: AKS offers users the ability to disable local accounts via a flag, disable-local-accounts

✅ “disableLocalAccounts”: true

Cluster Security

💎 Maintain Kubernetes version up to date: To stay current on new features and bug fixes, regularly upgrade to the Kubernetes version in your AKS cluster. Support for Kubernetes is current and N-2 versions only.

✅ AKS automatic configures both cluster-level auto-upgrades ( “upgradeChannel”: “stable”) and the node OS auto-upgrade ( “nodeOsUpgradeChannel”: “NodeImage”) channel together. Consider to fine-tune the auto upgrades by applying two separate sets of maintenance windows

💎 Use Azure Key Vault: Use Azure Key Vault to store Secrets and Certificates.

✅ The “azure-keyvault-secrets-provider” addon is enabled along with the OIDC issuer provider and the workload identity.

💎 Remove vulnerable images from your cluster: With image cleaner, you can detect and automatically remove all unused and vulnerable images cached on AKS nodes keeping the nodes cleaner and safer.

✅ Image cleaner is enabled with the following configuration “imageCleaner”: { “enabled”: true, “intervalHours”: 168

💎 Enable Microsoft Defender for Containers: Microsoft Defender for Containers provides protections for your Kubernetes clusters wherever they’re running.

❌ No container registry is provided with the AKS automatic setup, same for defender for containers.

💎 If user Service Principals for the cluster, refresh credentials periodically (like quarterly): You may want to update, or rotate, the credentials as part of a defined security policy.

✅ Only managed identities are used, no sign of service principals.

💎 Use a private registry for your images, such as ACR: Azure Container Registry allows you to build, store, and manage container images and artifacts in a private registry.

❌ No container registry is provided with the AKS automatic setup.

💎 Use Mariner OS as host OS: Mariner Container-Optimized OS is a first-party operating system developed specifically for Azure Kubernetes Service (AKS). It is designed to optimize the execution of Kubernetes workloads, offering a secure, lightweight, and highly performant environment.

✅ AKS automatic only provision nodes with Mariner OS. “osSku”: “AzureLinux”

💎 Separate applications from the control plane with user/system nodepools: Manage system node pools in Azure Kubernetes Service (AKS) and add taint to your system nodepool to make it dedicated.

✅ Only the system nodepool is created because the user workloads are managed by the Node Auto Provisioning (NAP). The system node pool is tainted as per best practices (CriticalAddonsOnly=true:NoSchedule).

💡Note that user node pool is created. NAP monitors the pods in your cluster that are pending due to insufficient resources. Based on the requirements of these pending pods, NAP identifies the most appropriate node types that would satisfy the pods’ resource demands. It then provisions new nodes of these types into your cluster.

Just create a deployment to see NAP reacting to the pods in pending state and adding one or more nodes.

💎 Use Azure Policy for Kubernetes to ensure cluster compliance: Azure Policy integrates with the Azure Kubernetes Service (AKS) to apply at-scale enforcements and safeguards on your clusters in a centralized, consistent manner.

✅ Azure Policy are enabled. “azurepolicy”: { “enabled”: true

The “AKS Deployment Safeguards” initiative is automatically assigned to raise warnings for deployments not compliant to the best practices.

Networking

💎 Choose the best CNI network plugin for your requirements (Azure CNI recommended): For integration with existing virtual networks or on-premises networks, use Azure CNI networking in AKS. This network model also allows greater separation of resources and controls in an enterprise environment but be aware of the impact on the network topology/IP ranges. You can now use the CNI overlay model which gives more control and takes the benefits of both approaches.

✅Azure CNI overlay powered by Cilium is configured. If AKS automatic is created without any network parameters, it creates a /12 network with the following subnet and NSGs.

💎 If using Azure CNI, size your subnet accordingly considering the maximum number of pods per node: The size of your virtual network and its subnet must accommodate the number of pods you plan to run and the number of nodes for the cluster.

✅ CNI overlay is used therefore the PODS will source their IPs from a dedicated address space and not from the node subnet. This will allow the cluster to scale out to a large number of PODS without worrying about running out of IPs.

💎 Use private clusters if your requirements mandate it: In a private cluster, the control plane or API server has internal IP addresses and is not exposed to Internet

✅ The managed API server is projected into the AKS VNET space (aks-apiserver-subnet) making the communication from nodes to the API server fully private.

💎 Use an ingress controller to expose web-based apps instead of exposing them with LoadBalancer-type services: To distribute HTTP or HTTPS traffic to your applications, use ingress resources and controllers. Ingress controllers provide additional features over a regular Azure load balancer, and can be managed as native Kubernetes resources.

✅ The managed NGINX ingress add-on is configured. This add-on enables an easy configuration of the ingress-nginx controller, the integration with Azure DNS and the SSL termination with certificates from Azure KeyVault.

💎 Configure default network policies in each namespace: Start by creating a deny all policy in each namespace and then add specific policies.

❌ Cilium network policies are ready to be used for east-west traffic control. However if you want to lock down every single namespace, the policy definition must be created and rolled out.

💎 Don’t expose your container registry on Internet: When possible, use private link to only allow private network to reach your registry.

❌ AKS automatic does not create a Azure Container Registry.

💎 Use Azue NAT Gateway as outboundType for scaling egress traffic: Whilst AKS customers are able to route egress traffic through an Azure Load Balancer, there are limitations on the amount of outbound flows of traffic that is possible.

✅ The NAT gateway is created with a single IP address.

💎 Filter egress traffic with AzFW/NVA if your security requirements mandate it (egress lockdown): Use Azure Firewall to secure and control all egress traffic going outside of the cluster.

❌ This is considered best practice, especially in enterprise scenarios. However, in this case, it is not implemented to avoid additional complexity and cost.

💎 Verify non overlapping CIDR ranges for Service-CIDR and Pod-CIDR or Subnet-CIDR: The Service CIDR should not overlap with Pod CIDR or Subnet CIDR or even peered VNET CIDR. If not configured correctly, the cluster might have strange behavior.

✅ Peace of mind, AKS automatic takes care of this one.

Resource Management

💎 Sizing of the nodes: what type of worker nodes should I use, and how many of them is a critical question which requires the analysis of the workloads deployed on your cluster to get the best values of it

✅ automatically managed by the NPA. If you need an additional user pool just add it to the cluster creation command.

💎 Set memory limits and requests for all containers: Set CPU and memory limits and requests to all the containers. It prevents memory leaks and CPU over-usage and protects the whole platform

✅ Azure policy can be used to enforce best practices like CPU and MEM resource limits. Additionally, AKS automatic enables the Virtual Pod Autoscaler (VPA) which will analyze the resource usage over time and rightsize (between allowed ranges) the workload resource requests.

💎 Use Disruption Budgets in your pod and deployment definitions: To maintain the availability of applications, define Pod Disruption Budgets (PDBs) to make sure that a minimum number of pods are available in the cluster.

💡NPA and VPA (“verticalPodAutoscaler”: { “enabled”: true } ) will cause POD EVICTIONS for the sake of better sizing and efficiency. When the evictions happen it is key to make sure PDBs are defined to keep your application up and running.

It’s up to you to configure PDB, however the deployment safeguards for AKS (azure policies) can help in enforcing this and other best practices.

Cluster Management

💎 Have a regular process to upgrade the cluster node images periodically (weekly, for example), or use the AKS autoupgrade feature: AKS supports upgrading the images on a node so you’re up to date with the newest OS and runtime updates.

💎 Monitor your cluster metrics with Container Insights (or other tools like Prometheus): If default integration can collect telemetry data and basic metrics (CPU/Memory), they don’t collect custom metrics and more detailed information. It’s often necessary to install a 3rd party software (prometheus is recommend within Kubernetes) and they store these metrics to exploit them.

✅ Azure Managed Prometheus is configured to collect the metrics generated by the Kubernetes cluster.

💎 Provision a log aggregation tool: Ensure that you are always aware of what happens in your cluster. Monitor the health of the cluster (nodes, server) but also the pods.

✅The minimal az cli command we used to create the AKS automatic does not install and configure Grafana. However you get a managed Grafana up and running with a couple of clicks in the Azure portal or using your favorite infrastructure as code tool.

💎 Send master logs (aka API logs) to Azure Monitor or your preferred log management solution: To help troubleshoot your application and services, you may need to view the logs generated by the master components. Be aware that if you don’t enable these logs, there is no way for Microsoft to retrieve them for you.

❌ this is not configured (and I am not surprised) because it comes with a certain cost.

💎 Consider an appropriate node size, not too large or too small: Optimizing node size can lead to lots of benefits including cost optimization, optimized performance and proper resource allocation.

✅ This is where NAP shines.

💎 Store and analyze your cluster logs with Container Insights (or other tools like Telegraf/ElasticSearch): Monitoring the logs from your cluster can help prevent issues and identify areas of improvements

✅Log Analytics is deployed and configured to store the cluster logs.

💎 Configure alerts on the most critical metrics (see Container Insights for recommendations): Alerts can provide early warnings of issues, etc in your cluster.

⚠️By default no alerts are created but it’s extremely easy to configure the recommended metric alert rules for Prometheus.

💎 Develop own governance practices to make sure no changes are performed by operators in the node RG (aka ‘infra RG’): Develop own governance practices to make sure no changes are performed by operators in the node RG (aka ‘infra RG’)

❌ It would be nice the infrastructure RG will come with locks.

💎 Set Upgrade Channel: In addition to manually upgrading a cluster, you can set an auto-upgrade channel on your cluster

💎 Enable cluster autoscaling: To keep up with application demands in Azure Kubernetes Service (AKS), you may need to adjust the number of nodes that run your workloads. The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints.

💡The cluster autoscaler is not configured. VPA is used to add additional nodes.

💎 Don’t use the default namespace: It’s recommended to keep all applications in a namespace other than default

✅ Once again, the deployment safeguards for AKS are coming to the rescue to enforce this best practice via Azure policies.

Business Continuity

💎 Use Availability Zones if supported in your Azure region: An Azure Kubernetes Service (AKS) cluster distributes resources such as the nodes and storage across logical sections of the underlying Azure compute infrastructure. This deployment model makes sure that the nodes run across separate update and fault domains in a single Azure datacenter.

✅ agentPoolProfiles is configured with availability zone (“availabilityZones”: [“1”,“2”,“3” ])

❌ NPA is not using AZ by default but it can be configured by adding the desired availability zones in the Karperter NodePool resource.

💎 Use the standard-tier AKS offering: standard-tier (formerly Uptime SLA) is an optional feature to enable a financially backed, higher SLA for a cluster. It provides you a 99,95% SLA instead of the 99,5% SLO and is relevant for your production (and large) clusters

✅ The standard tier is used.

Storage

💎 Choose the right storage type: Understand the needs of your application to pick the right storage. Use high performance, SSD-backed storage for production workloads. Plan for network-based storage when there is a need for multiple concurrent connections.

✅ NAP will pick up the disk size for you. Node Auto Provisioning can be additionally configured to fine tune the storage needs for your workload.

💎 Use ephemeral OS disks: With ephemeral OS disks, you see lower read/write latency on the OS disk of AKS agent nodes since the disk is locally attached.

✅Ephemeral disks are used for the agent pools : “osDiskType”: “Ephemeral”,

Conclusion

AKS automatic significantly accelerates the provisioning of AKS by simplifying the most common Kubernetes configuration tasks, making them frictionless. With automation, you can go from zero to a fully observable, well-configured Kubernetes cluster in just minutes. By handling tasks like scaling, monitoring, and maintenance, organizations can shift their focus toward innovation and development, rather than manual infrastructure management.

Automatic covers a wide range of best practices without delving into every infrastructure detail, leaving some questions unanswered. This approach provides organizations with a simplified AKS experience while still allowing them to maintain control and tailor AKS to their specific needs..

So at the end of the day is AKS automatic production ready by default? Let me know what you think.