Azure @ Enterprise - DNS->PublicIP->Azure App Gateway->Pod Using App Gateway Ingress controller

This is another GitHub project introduction post. This time introduces a repo that shows how an end-to-end Azure solution can be set up from Azure DNS -> Public IP Address -> Azure App Gateway -> Azure Kubernetes Service with Application Gateway Ingress controller AGIC hereafter.

This is a standard use case of how to expose a web application from Kubernetes via standard Azure  networking services such as DNS, Public IP Address and AAG

Problem

When we want to route traffic to AKS, the primary question is how to set up the ingress. Traditionally we use NGINX ingress controller and it was not a question at all. Now with Azure Application Gateway Ingress Controller, we just need to justify the decision.

What is Application Gateway Ingress Controller?

It is one of the Kubernetes Ingress controllers that can only work with Azure Application Gateway. The major differences with NGINX are

• AGIC will monitor the cluster and update the Azure Application Gateway back with routing IPs.
• AGIC uses Pod IPs as the backend instead of the service IP address. If there are 3 pods there will be 3 IPs in the Application Gateway backend pool
• In order to update the Azure App Gateway, the AGIC should have Azure RBAC permissions via user-assigned managed identity. 

Photo credits go to the official docs.

My recommendation is not to use this AGIC as it's having issues.

There will be a separate post on issues with AGIC. 

Setting up

Below is the repo that has biceps and required PowerShell to set up end to end. 

https://github.com/ms-azure-demos/dns-publicip-aag-aks-agic

Please note not all things are automated. Readme.md has the details.

References

https://github.com/paolosalvatori/aks-multi-tenant-agic

https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml

https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/setup/install-new.md

Azure @ Enterprise - Networking at pod level inside AKS using Calico

This post is to introduce networking inside the Kubernetes cluster at the pod level. There may be questions about why as application developers we need to worry about networking inside the Kubernetes cluster? 

It depends on how the Kubernetes cluster is planned in the organization. If we are lucky our project may be getting its own Kubernetes cluster. But often we ended up as a tenant inside another Kubernetes cluster. Our application may need to access a public API URL, if we set the networking rules at the cluster level, all the applications inside the cluster will get access. Organizations generally prefer something other than that. Also, we don't want some other application in the cluster accessing our database, Redis cache, etc... even if they have the credentials.

Hence we will end up dealing with the network policies at the pod level

Theory

Kubernetes defines networking policies as resource natively. But it doesn't have a built-in mechanism to enforce the networking rules.

For that, we need to use one of the network plugins. For more theory please read the docs. Now let us come to some practicals

Azure NPM v/s Calico

Since we are going to use Azure Kubernetes Service, we can focus on 2 plugins. Those are kinda natively supported by AKS.  One is Azure NPM (Network Policy Manager) and the other is Calico.

Though we may tend to think the Azure NPM is fully implemented in AKS as both start with Azure, the reality is otherwise. As of writing this post, the calico has better support including Windows Server 2019 in AKS. Some of the features of Azure NPM are still in preview. See the comparison of Azure NPM v/s Calico for more details.

As the title of the post shows we are going to use Calico.

Installation of Calico on AKS

As per calico documentation, there is a way to install calico on an existing AKS cluster. But when I followed it was not working efficiently.

So better enable Calico when we create the AKS cluster.

How the network policy looks like

The networking feels alien to at least some developers. It requires knowledge about IP addresses and notations, complex network setup screens or commands, etc. But don't worry the networking can be set up via YAML files.

If YAML feels alien to a developer nowadays he should rethink being a developer.

Sample

Enough theory. Please clone the below repo and check out how the networking policies can be applied.

Step-by-step details are not going to be added here as that sample will get more scenarios in the future.

https://github.com/ms-azure-demos/aks-windows-container-network-policy-calico

All are welcome to give PRs or add issues to the repo.

References

• https://cloudblogs.microsoft.com/opensource/2019/10/17/tutorial-calico-network-policies-with-azure-kubernetes-service/
• https://medium.com/geekculture/aks-with-calico-network-policies-8cdfa996e6bb
• https://azure.microsoft.com/en-us/blog/integrating-azure-cni-and-calico-a-technical-deep-dive/

Docker images to echo HTTP headers and other details for testing purpose

When we want to test simple features in Kubernetes, we need simple web server images. We don't need a full-fledged docker image but something that responds with the IP address, HTTP headers, etc... That way we can quickly test the load balancing and other basic features.

Some create new docker images to do so. But we don't need to as this is a common problem and others have already created such images. The best way to search is with the keyword 'echo' in the docker hub.

Below are some echo images

• https://hub.docker.com/r/ealen/echo-server
• https://hub.docker.com/r/mendhak/http-https-echo/tags
• https://hub.docker.com/r/hashicorp/http-echo/

These are capable of responding differently according to the inputs given. It is really handy to simulate error messages, caching, etc...Check out for more details.

In case there is a better one, please comment.

Azure @ Enterprise - AGIC on AKS - Unexpected status code '403' while performing a GET on Application Gateway.

Context

When we want to route traffic from the internet to AKS through the Azure Application Gateway (AAG for short), there are 2 approaches to set up ingress in AKS. The traditional NGINX way where the flow is one way. But with  Application Gateway Ingress Contoller (AGIC for short), there is one more approach to solving the problem. AAG can route to the Pod IPs if it knows. 

• Who will let AAG knows about the pod IPs? 
• It is the AGIC running inside the AKS.
• How does the AGIC authenticate to modify the AAG?
• Using user-assigned managed identity with necessary permissions

Problem

When we simply create the AKS cluster with AGIC enabled, we expect that we can pass the pre-created managed identity to the AGIC that has permission to AAG. Note that the AGIC is not installed separately, instead enabling using AKS add-on. We are in the DevOps automation world, so we normally look for a declarative mechanism to do it. Obviously, the bicep will come on our way. When we look at the office bicep reference for AKS-managed clusters details on AGIC enablement are not available as of writing this post.  But there are links at the bottom to one of the ARM examples having AGIC capability. 

It has a section located at properties->addonProfiles->ingressApplicationGateway where we can give the details of AAG and the identity.

Since the ARM and bicep are like JavaScript and TypeScript, it is relatively easy to prepare the bicep given the equivalent ARM template.

After conversion when the bicep is run below warning showed up

Warning BCP073: The property "identity" is read-only. Expressions cannot be assigned to read-only properties. If this is an inaccuracy in the documentation, please report it to the Bicep Team. [https://aka.ms/bicep-type-issues]

If someone cares about the warning, he is not a developer. So ignored it and proceeded. Poor Azure documentation which is calling for help.

But after the creation of AKS it started showing errors when accessing via AAG. The logs of AGIC ingress pod shows the below error. 

kubectl logs ingress-appgw-deployment-<a random string> -n kube-system

E1206 02:34:30.289515       1 client.go:170] Code="ErrorApplicationGatewayForbidden" Message="Unexpected status code '403' while performing a GET on Application Gateway. You can use 'az role assignment create --role Reader --scope /subscriptions/<subid>/resourceGroups/K8ClusterWithAGIC --assignee dfb5d05b-cc65-46fa-a0b8-9fe93ebfed9e; az role assignment create --role Contributor --scope /subscriptions/1cd7467f-530b-4278-94ef-de3f5da74e0c/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/myappgw --assignee dfb5d05b-xxxx-xxxx-xxxx-9fe93ebfed9e' to assign permissions. AGIC Identity needs atleast has 'Contributor' access to Application Gateway 'myappgw' and 'Reader' access to Application Gateway's Resource Group '<rgName>'." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '41ff78bf-xxxx-xxxx-xxxx-180e4c7970a2' with object id '41ff78bf-3b92-xxxx-xxxx-180e4c7970a2' does not have authorization to perform action 'Microsoft.Network/applicationGateways/read' over scope '/subscriptions/<subid>/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/myappgw-' or the scope is invalid. If access was recently granted, please refresh your credentials.""

Step 1

Wonder why there is a section called solution? will come to know soon. The error is somewhat clear. But my pre-created managed identity has permissions. When we closely look we can see 2 issues 

1. The error message shows a different client id of MI instead of what we have given.
2. Since the client id used is different. It is newly created and obviously, it doesn't have permission to manage AAG.

The first issue because it is not using the identity that we passed to the bicep. We should have taken the warning seriously.

The solution is to give contributor access to AAG. Let us worry about the least permission later. The name of the newly created identity follows the below convention so it is easy to search.

"ingressapplicationgateway-<cluster name>"

Problem 2

Once we fix the issue, we may be thinking all good as AGIC has permission to AAG to create listeners, backend pools, and settings. But it is not enough as the below error started showing up.

E1206 02:59:09.217888       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '41ff78bf-xxxx-xxxx-xxxx-180e4c7970a2' with object id '41ff78bf-xxxx-xxxx-xxxxx-180e4c7970a2' has permission to perform action 'Microsoft.Network/applicationGateways/write' on scope '/subscriptions/<subid>/resourceGroups/<rgName>/providers/Microsoft.Network/applicationGateways/myappgw'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptio957", FieldPath:""}): type: 'Warning' reason: 'FailedApplyingAppGwConfig' network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '41ff78bf-xxxx-xxxx-xxxx-180e4c7970a2' with object id '41ff78bf-xxxx-xxxx-xxxx-180e4c7970a2' has permission to perform action 'Microsoft.Network/applicationGateways/write' on scope '/subscriptions/<subid>/resourceGroups/<rgName/providers/Microsoft.Network/applicationGateways/myappgw'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/<subid>/resourcegroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myappgw-ManagedIdentity' or the linked scope(s) are invalid."

Step 2

Here we can see 2 problems

1. The error message is misleading with the same client id and object id. In fact, they are different. It seems people at Microsoft itself are confused about this AAD security mechanism.
2. AGIC identity is looking for permissions on the identity of AAG.

God knows why the AGIC requires permission on AAG's managed identity to manage the AAG details. This issue can also be fixed by giving a contributor permission to the managed identity of AAG.

Happy debugging...

Read the docs

This is the standard answer we get when we do trial and error. As of writing this post, the permission on the identity of AAG is not mentioned in the brownfield installation.

References

1. https://www.torivar.com/2022/09/16/aks-with-agic-existing-agw/ - Almost the same experience but with Terraform IaC

Azure @ Enterprise - Upgrade blocked!!! create new AKS cluster

Finally, it happened when upgrading an AKS cluster. Its nothing but a dead-end in upgrade and we have to create a new AKS cluster and migrate the workload with all settings

Why it was not upgraded?

Any upgrade requires some planning. After the upgrade, we need to make sure all the apps are running. Why bother? Let us keep the best practice aside, and upgrade when needed. Ideally, the environment is expected to be scripted so it is a single click to recreate

Reason for the current upgrade

I wanted to add a new node pool to the AKS cluster. When I am trying to add a node pool, it says the Kubernetes version is old. The cluster needs to be upgraded.

What if I try to upgrade AKS Cluster

It is not showing the available upgrade.

As per Microsoft guidelines, do not upgrade forcefully.

What is left

Unfortunately, it is very difficult to make sure the entire environment is scripted out. But now need to reverse engineer the cluster to script everything back and create a new cluster.

Next time consider enabling the auto upgrade feature.

References

• AKS Upgrade Practices - https://learn.microsoft.com/en-us/azure/architecture/operator-guides/aks/aks-upgrade-practices

Azure @ Enterprise - Scheduling task inside Windows container in Kubernetes

Problem

As part of my day job, we are doing a lift & shift of a giant monolith .NET 4.8 Windows-based application to Azure Kubernetes Service. Currently the application supports scheduling tasks that can be created and configured by the admins. Behind the scene, it creates scheduled tasks in Windows Task Scheduler pointing to the respective executable and passing right parameters.

How to do this in AKS or Kubernetes in general?

Solution

Since we need to limit the changes, we are planning to use the Windows task scheduler inside the container. Yes, it is going to be a Windows container and it may not be the most innovative solution in the world of containers where containers may be destroyed and recreated at any time.

Keeping the design things aside, this post is to explain how the Windows scheduled tasks can be created inside the container.

Nothing fancy, the same command that works inside a virtual machine to create a scheduled task will work inside Windows containers as well. Below goes the command.

Debugging why the deletion of PVC in Kubernetes is running indefinitely

This is related to troubleshooting Kubernetes deployment and requires good working knowledge about Kubernetes.

Issue

It all started with the issue of HELM chart deployment failing that are triggered from DevOps pipelines. The pipelines were using --atomic flag to make sure the failed deployments are getting rolled back. 

The deployments were working till last Friday and no changes to the HELM charts after that date.

Troubleshooting

The observable missing that we found when running kubectl get all is as follows.

0/3 nodes are available: 3 persistentvolumeclaim "my-azurefile-pvc" is being deleted.           Normal   NotTriggerScaleUp  2m42s (x91 over 17m)  cluster-autoscaler  pod didn't trigger scale-up: 1 persistentvolumeclaim "my-azurefile-pvc" is being deleted

This shows there is a PersistentVolumeClaim hanging around that needs to be deleted. Here is the PVC

The container is not starting due to PVC is not ready and its waiting.

The PVC was in the terminating state. Since it's in the terminating state but not terminating, the next step is to delete the my-azure-pvc and redeploy the HELM chart. When we tried to delete it using the below command, the command was not getting completed.

kubectl delete pvc/my-azurefile-pvc -n <namespace> --force

Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.persistentvolumeclaim "my-azurefile-pvc" force deleted

Googling that revealed a possibility that it may have finalizers. We checked it and saw there is a finalizer.

Now we have to remove the finalizer by using the below command. The first command tried with namespace then it showed a wired error message as follows.

kubectl patch pvc my-azurefile-pvc -p '{"metadata":{"finalizers":null}}' -n <namespace>

Error from server (BadRequest): invalid character 'm' looking for beginning of object key string

There was no explanation I got from the internet where is this invalid 'm' character. 

Then we tried without namespace. Then it says the PVC resource is not available in default or I don't have the permission. Really I don't have permission in the default namespace instead I have permission to the namespace allocated to the application's dev instance. The behavior is wired as it's not able to find the resource in the right namespace.

kubectl patch pvc my-azurefile-pvc -p '{"metadata":{"finalizers":null}}'

Then get into research mode again. Then found that if there are any other containers in the cluster using the same PVC, the PVC resource cannot be deleted. We checked all the containers and could see there is another container from different HELM deployment is using it.

kubectl describe pod/<pod id> -n <namespace>

Uninstalled the deployment that created the above pod and the issue was solved.

Root cause

Not sure what caused the PVC to be not deleted in the first place. Maybe someone played in the cluster other than updating HELM charts from the DevOps pipeline.

Installing apt-get and bash into the .Net 6 Alpine container

The minimal container image of .Net 6 comes really lightweight. The name of the image at least we are using is dotnet/runtime:6.0.1-alpine3.14. It has very little footprint below 40-50 MB during runtime. But that doesn't even have a ping utility, bash shell, sudo command, and not even the apt-get to install other packages. This minimal behavior makes it difficult to debug.

Below goes the steps to install apt-get into that container to troubleshoot applications especially when deployed in Kubernetes.

[Video] Configuring container environment variables for ASP.Net WebAPI hosted inside Kubernetes on Docker Desktop

This is the second video in the Kubernetes series. Actually, when I started vlogging, I had decided that I will make short videos of max 5 minutes. But unfortunately, I was not able to stick to that. Here onwards I am planning to stick with my time limit. 

What is new?

We have to configure applications based on the environment it runs. It is not different for containerized applications running in Kubernetes. This video is showing how to configure the environment variables for the containers.

It uses an ASP.Net Core WebAPI application that returns the environment variable name when invoked with the name. It uses .Net 5 and basic YAML to deploy to Kubernetes that is running on Docker Desktop using WSL2 as a backend.

No more details. Please watch the video below.

References

https://kubernetes.io/docs/concepts/configuration/configmap/

[Video] Running simple ASP.Net Web API in Kubernetes on Docker Desktop

This post mainly contains the notes that I used to create a video. The video talks about hosting ASP.Net WebAPI into Kubernetes. The API can be accessible from the host machine using http://localhost:<port>/. The aim is to show simple hosting in Kubernetes that runs using Docker Desktop. 

It is advised to watch the video to get more clarity. 

Dev environment

Please refer to the previous post about setting up Docker Desktop and K8s environment. Along with that, it is required to have Visual Studio 2019 with docker container ASP.Net tools installed.

Source code

This post is not about creating a new visual studio project and enable the K8s support. Instead, download the sample from the below location

https://github.com/joymon/dotnet-demos/tree/master/web/webapi/simple-k8s-hosting/SimpleK8sHosting

Running using Docker Desktop

Just click on the run button in Visual Studio 2019 to run using Docker Desktop. The Dockerfile and other setups are already done in the sample.

Publish to Docker hub - the image registry

Below are the commands to tag and push to Docker Hub

Getting started with Kubernetes dev environment using Docker Desktop

This is 3rd or 4th time, I am learning Kubernetes (hereafter mostly refer as K8s short form) hands-on sessions. Every time I learn the kubectl command and its options, I forget as there were no chances to apply in the day job. Another mistake I did all those times was missing to post learning to this blog.

Hope this time I will get a chance to use it in the day job and not miss posting the Kubernetes learning on to this blog.

This post is very basic. The aim is to get started with the Kubernetes development environment using Docker Desktop. Below are the steps at a high level to get started. Detailed steps with videos are available on the internet.

Install Docker with Kubernetes support

The first step is to download the Docker Desktop and install it. It's straightforward as installing any other software in Windows. It is free and requires virtualization to be turned on. 

In order to run the Linux containers, it is better to enable the WSL2 backend. There are detailed instructions available to get it done including how to enable WSL2 on Windows.

Kubernetes support is only available when we use the Linux container mode. Note that Docker Desktop supports both Windows and Linux modes.

Points to note

• Better don't take the experimental build. Stay one version below the current stable version.
• This installation will modify our hosts file in Windows. Better not change anything.
• It is better to have an 8 core 16GB machine to work smoothly with container workloads.

Install Kubernetes dashboard

For beginners, it is easy to understand what is going on in the K8s cluster by looking at a UI dashboard application. K8s don't install a UI dashboard by default. That dashboard can be installed by using the below PowerShell commands.

---

Azure @ Enterprise - Why I recommend Kubernetes as app hosting platform?

I started writing this post 2-3 years back. Mainly when Apache Spark 2.3 started supporting Kubernetes (K8s) in 2018. It was obvious that Kubernetes is taking over app hosting space the same way virtual machines took over physical machines. All are expected to understand where the industry is moving and adopt. Hence I paused this post as there is nothing I need to endorse. But it's time to resume this post and publish it.

If you are already using container and K8s orchestration, feel free to skip this post and save time.

Basics

Containers

Please refer to my 2015 post about Software Containerization via Docker - First look and some thoughts for very low-level basics. The container can be understood as a micro virtual machine with the application(s) and runtime isolated from each other. It has a file system inside it, it can expose ports for networking. Mount external storage.

This mainly co-evolved with the microservice architecture where the huge systems are separately developed and deployed as small services that own their own data and release cadence.

Kubernetes

Officially 

"Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation." - Kubernets.io

In short, it is the container orchestration engine.

What is and why do we need container orchestration?

 In my own words, orchestration is  

• Where to run
• Horizontal scaling - Mechanism to use multiple nodes(VMs are called nodes) to run containers of the same system. Nodes are expected to fail at any time.
• Deciding where to run the container - The mechanism to specify what node executes what container. Eg: a particular container in the solution may need to run on the Linux node another may need a Windows node. Some containers may need GPU to be present on the node
• Can also include virtual nodes that even can use Azure Container Instances.
• Deployment and rollback
• Ability to rollback easily in case something goes wrong in deployment
• It is expected to have failures in deployments but the orchestration system should be able to roll back easily to the previous known state.
• Networking
• Determine how the containers are allowed to communicate with each other and what ports are exposed to outside.
• Also has load balancing capability
• Also has routing capability.
• Service discovery
• The services/components in the system should be able to discover the endpoints to other services as the services may be deployed to multiple nodes.
• Resource allocation
• Determines how much system resources need to be given to each container.
• Storage mounting
• Ability to mount external storage
• Scaling
• The mechanism to specify how many container instances are to be running at a time. Eg: Based on Azure queue messages it has to scale out / horizontal scale.
• Secret and configuration management
• Secrets can even be stored in Azure KeyVault and places like that. But that may not work on-prem. An orchestration engine needs to have secret management.
• Configuration management is another aspect of any software system. The containers need to be initialized with proper configuration wherever it runs.
• CRON
• Capability to execute a container based on CRON expression.
• Health monitoring & self-healing
• Monitor the state of the containers and make sure the state is true to the definition of the system. Eg: If a container crashes, the orchestrator makes sure it's started again.

Hope the above explains what and why orchestration?

Links

There are people out there who can explain orchestration better than me.

https://kube.academy/courses/kubernetes-101/lessons/why-container-orchestration

https://www.redhat.com/en/topics/containers/what-is-container-orchestration

https://www.ibm.com/cloud/architecture/content/course/kubernetes-101/dealing-with-many-containers/

https://docs.microsoft.com/en-us/learn/modules/intro-to-kubernetes/2-what-is-kubernetes

Why Kubernetes?

It is the currently available best container orchestration engine. It supports most of the capabilities mentioned in the above list of what and why orchestration. For further reading refer to K8s documentation.

What Kubernetes is not?

It is equally important to understand what Kubernetes is not for. They

• Don’t have native CI/CD concepts.
• Don't have a data storage engine. RDBMS Rollbacks are not supported out of the box. Better be backward compatible. 
• Don’t have a built-in blue-green deployment strategy
• Don’t have a built-in serverless function or event-driven execution
• Don’t have a built-in message broker, service bus, etc… we have to use external or run a container for message broker.

More in their documentation.

How and where to host

K8s can be run on top of virtual or physical machines or in the cloud.

On-premise

It can be installed on-premise on top of virtual machines or physical machines. The machines are called nodes as well. Control nodes need to be Linux, worker nodes can be either Windows or Linux. If your organization is a hard-core windows shop, no luck here.

Cloud

Major cloud providers provide an easier way to host K8s applications.

• Azure - AKS
• AWS - EKS
• GCP - GKE

Why Kubernetes is good for enterprises?

• Vendor-neutral and portable
• If cloud providers increase fees, easy to switch from one vendor to other
• Worst case start and maintain on-prem K8s cluster
• Never do own cluster if you are a start-up and hosting a single application.
• No vendor lock-in due to native PaaS model
• If we start using Azure Function, Lambda, etc... that has vendor-specific API, there is a high chance that we get stuck with it. Remember the Azure Cloud web and worker roles that are in extended support now.
• Environment parity
• Less 'it works in my machine' comments from developers
• High density
• Can pack more things in resources. Less wastage.
• Multi-tenancy
• Multiple apps of the enterprise can share a single cluster via namespaces-based isolation.
• Fewer operations cost
• Due to high density and multi-tenant nature less operation cost to make the cluster up and running.
• A clear view of architecture
• Just reviewing the YAML files will give a clear idea of how the system is designed. No more surprises because the settings and configurations are spread across many places.
• Repeatability
• Since the container is immutable, it can be used years later if we want to see how the app did run earlier.
• No need for private cloud software
• An on-premise cloud can be created on the K8s cluster without any private cloud software such as OpenStack. The K8s cluster is the cloud where applications are separated by namespaces. Oh yes, the storage needs to be managed.

Learning curve

It will not be an easy journey to the Kubernetes. Engineering teams (Dev and DevOps if you have separate teams) need to be upskilled as there are a lot of new concepts in K8s. Some below

• Namespaces
• Nodes - Control and worker nodes
• Controllers - ReplicaController, NodeController
• Deployment
• ReplicaSet
• Pod
• StatefulSet
• DaemonSet
• Service
• Job, CronJob
• Volume, Persistent Volume, etc...
• ConfigMaps
• Secrets
• Policies, Resource Quota
• Selectors
• YAML (Not that specific to K8s)
• Kubectl

I didn't get a comprehensive mind-map showing all so planning to prepare one. It is advisable to upskill the team by hiring external people who already have experience with K8s. Converting an existing team that doesn't know K8s to experts is tough unless you are ok to accept failures.

SQL Server from container - Before jumping into the bandwagon

It is easy to get started on development and simple testing using SQL containers. It was discussed in the previous post. But before putting into production and start developing real applications we had to make sure the below things at least.

SQL Server running inside Linux container doesn't mean its free

People from the Linux world or in general people tend to think that everything in Linux is free. But that is not true in the case of SQL Server similar to other enterprise software running in Linux. In order to use the SQL Server container in production, we need a valid license even in the Linux containers.

Configuring and operating

Running and operating containers is an added skill to the operation teams. We need to make sure our client or in-house operational teams are capable of doing it.

Feature parity

There are many factors influencing feature differences. Microsoft always says its the same codebase but features don't reflect that. Let's see the factors that decide features

• Versions - Obviously there will be differences between SQL 2016, SQL 2017, and SQL 2019 features. The codebase is never the same though the latest may be built on top of the previous.
• Editions - First level feature difference inside the same version is based on the edition. This is regardless of wherever we run the SQL instance. This makes sense as we need to give more money to unlock features. This is there from the early days of SQL Server.
• Running platform - There are feature differences based on where the SQL instances run such as standalone inside Windows OS, Azure, and Linux containers.
• Azure PaaS platform - Inside Azure, there are Azure SQL and Azure SQL managed Instance models. They have many feature differences.

The containerized SQL server is more like the Azure SQL than compared with desktop SQL installation. Features such as FileStream, CLR support are not available in SQL container. The full list of features that are not supported in the SQL container can be found here.

Production readiness

As per Microsoft, SQL Server on Windows Container is not production-ready. But SQL Linux containers are production-ready. They have documentation on High availability when running SQL Linux containers and how to orchestrate using Kubernetes.  But not sure currently companies are investing to just move their SQL workload to a containerized production environment. 

Everyone agrees to start using it for development and testing. Eventually, when more workloads start using the SQL Server Big Data Cluster features, production will also start using SQL Server in containers. There are also documentation says SQL container as is don't support clustering hence the DR can be done only by log shipping. The Kubernetes based HA for SQL containers may not be instantaneous for an RDBMS database like Always On Availability Groups or FCI. It may take some time to detect the failure and start the next container instance.

As of now, my takeaway is not to use SQL containers for critical production workloads.