Azure @ Enterprise - Windows containers in AKS - Memory allocation black magic

There were many predictions made in my blogs. Death of Silverlight and WPF. Oh WPF is still available Windows Phone prediction missed to post separately. Now I am predicting the future of Windows OS. The server side is already taken by Linux even in the Microsoft Azure infrastructure from 2015 onwards. The client side will soon be dominated by Chrome OS once the 5G is available to everyone and the browser gets more power with WebAssembly.

If we have a chance do not go for the Windows container. Go to the windows container if there is absolutely no way to use the Linux container. It may be due to budget constraints to migrate from .Net Framework to .Net Core, unavailability of certain legacy COM components, etc.

Kubernetes memory management

Kubernetes orchestrates containers by scheduling them into nodes which are mainly virtual machines. If the node has 32 GB memory will that all be available to containers? 

No. It will be distributed to the below participants

• Node operating system is software running in node.
• In order to manage the containers Kubernetes needs to run software in every node called kubelet, proxy, etc.
• If we have installed third-party CSI drivers and monitoring agents, they also need to run on every node. 
• Kubernetes keeps some aside to anticipate spikes. That is not considered when allocating

They all are processes from the side of the OS and all require node resources.

Not all memory, CPU and other node resources are available to containers

It's fine. Let's understand the calculation of how much these resources are needed for the infrastructure Since memory is easy to digest let's calculate how much memory is available to the containers.

It is documented in Kubernetes official docs so no need to repeat it here. The only thing missing is the difference between the Windows and Linux container and the driver memory.

Windows nodes in AKS resource reservations

AKS(Azure Kubernetes Service) is just a managed service of Kubernetes hosted by Azure. Something like a laundry shop managing washers and dryers for its customers.

The node memory reservation in AKS is well documented. 

Windows nodes are treated separately in AKS and they need additional 2GB

The internal implementation for Linux and Windows resource management is also different due to OS differences. Let us see the rules and a sample calculation.

• Eviction threshold
• 750Mi
• Kubelet
• 25% of the first 4 GB of memory
• 20% of the next 4 GB of memory (up to 8 GB)
• 10% of the next 8 GB of memory (up to 16 GB)
• 6% of the next 112 GB of memory (up to 128 GB)
• 2% of any memory above 128 GB
• Windows node
• Additional 2 GB

The resource reservation calculation for 7GB nodes 

7 GB Linux node memory reservations.

0.75 + (0.25*4) + (0.20*3) = 0.75GB + 1GB + 0.6GB = 2.35GB  ~33.57%

7 GB Windows node memory reservations.

0.75 + (0.25*4) + (0.20*3) + 2 = 0.75GB + 1GB + 0.6GB + 2 GB = 4.35GB ~62.14%

Previously we saw that there can be drivers running inside the node that will also require memory. Below is a screenshot from the running node with a decent amount of drivers in a windows node.

7 GB Windows node memory reservations

Please note the memory requirement for same drivers is less in Linux nodes in the same cluster. Also the number of drivers varies by workload requirement

Let us do math again.

7 GB Windows node with some drivers running

Considering only the request as that is the minimum required.

Driver memory = 300+200+600+50+120+120+120+512 = 2.02GB

Total memory = 4.35GB + 2.02 GB =  6.37 GB ~91% of memory is reserved

The resource reservation calculation for 32GB windows nodes

Without drivers 0.75 + (0.25*4) + (0.20*4) + (0.10*8) + (.06*16) + 2 = 0.75GB + 1GB + 0.8GB + 0.8GB + 0.96GB + 2 GB = 6.31GB

With drivers 6.31GB + 2.02GB = 8.33GB ~26.03% reserved 

The resource reservation calculation for 128 GB windows nodes

Without drivers 0.75 + (0.25*4) + (0.20*4) + (0.10*8) + (.06*112) + 2 = 0.75GB + 1GB + 0.8GB + 0.8GB + 6.72GB + 2 GB = 12.07GB

With drivers 12.07GB + 2.02GB = 14.09GB ~11.00% reserved => 113.91GB to containers

All these calculations are done with default recommended values

In case there is any calculation mistake found, please respond in the comments section.

Lessons learned

• Do not get into the windows container if possible.
• If windows containers are inevitable, use bigger nodes such as 128GB.

References

• Windows and Linus
• Whoa. Microsoft Is Using Linux to Run Its Cloud
• What is Microsoft doing with Linux?
• Microsoft loves Linux
• Kubernetes resource reservation
• https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable
• https://kubernetes.io/docs/concepts/configuration/windows-resource-management/
• https://learn.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#resource-reservations
• https://kubernetes.io/blog/2021/11/26/qos-memory-resources/
• https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu

ASP.Net IIS AppPool Recycling is a hack

From the beginning it was known to me that the IIS AppPool recycling is a hack. If a technology is matured enough to handle web traffic, it doesn't need periodic recycling of processes. To me there is no difference between support teams asking to restart machines to solve issues and app pool recycling. In both these cases, the memory is cleaned up.

Whenever I was involved in IIS applications, I instructed the teams to disable app pool recycling from the development time itself. But it never happened. According to most of developers and their managers that is a feature from IIS and why should they change defaults and bring unnecessary issues. Whenever I had direct delivery responsibility, I ensured that the app pool recycling is not enabled. There was mistake from my side as well that I never invested time to find out a valid Microsoft link which says do not recycle app pool process. But yesterday the time arrived. There was a client question on why the app pool recycling is disabled on some applications where majority has recycling. Those were framework level applications and in my direct control during development. 

If you have a problematic application and you cannot easily correct the code that causes the problems, you can limit the extent of these problems by periodically recycling the worker process that services the application. Microsoft

Finally got the link where MSFT says about app pool recycle. But the scenario again complicated for us. 3-4 web apps only disabled the app pool recycling and the other 50+ web apps not. Were those created with no quality and difficult to fix?

Stateful v/s Stateless IIS web apps

Here comes 2 different types of web apps. Some apps will be just to receive request and send response back. If we recycle these stateless apps, there is no harm to the behavior of the system as every request and its response pair is independent of other request-response pairs. So it is 'ok' to recycle. In case there are something going wrong with memory and unnoticed during dev and QA, it won't cause trouble in production as well. Since extensive memory, load, long running testing are not needed, we can develop in low cost.

Other type of apps will be maintaining some sort of state in the memory and state might be evolving over time. Also their behavior will be controlled by the current state. If we recycle the process, the state will be lost and their behavior will be wrong after recycling. Yes there is the answer.

It is 'ok' to recycle the stateless web apps but never recycle the stateful apps.

When it is said 'ok' means, there are no ways to fix in production. Other scenario is when there was no chance to properly test for long hours and LOH fragmentation during development. Again it is not recommended to keep a big state in memory which may cause scaling issues.

Better write programs properly and avoid app pool recycling.

LOH and recycling

Without mentioning Large Object Heap, any article about .Net memory management is incomplete. As all knows LOH was never getting compacted in earlier .Net versions. That causes large objects to be fragmented and finally lead to OutOfMemoryException. In those days, it was not a choice to enable AppPool recycling but we have to. I agree that there are programmatic mechanisms to avoid LOH fragmentation. But those are all costly during development and testing. Also requires high skills.

Links

https://weblogs.asp.net/owscott/why-is-the-iis-default-app-pool-recycle-set-to-1740-minutes

Unable to debug 64bit .net app in VS 2010, because sos is not getting loaded

Recently I had to debug one of our research application for memory leaks. It is developed in Visual Studio 2010 as 64 bit .Net application with considerable amount of unmanaged (Win32 API) calls. Since I have Visual Studio 2010 Pro, thought of debugging in VS itself, for just a change.

How to debug in Visual Studio using sos extensions

Initially I enabled the unmanaged debugging in the project properties which is the first thing to do when starting sos debugging in VisualStudio. Then I tried loading sos as usual in Immediate window.

.load "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos.dll"

But it ended up as

Error during command: extension c:\windows\microsoft.net\framework64\v4.0.30319\sos.dll could not load (error 193)

Oh I forgot one thing, we cannot debug 64 bit applications in VS 2010 because VS 2010 process is running as 32 bit even the OS and machine are running in 64 bit. I can see devenv.exe is a 32bit process in Task Manager, but I am not sure about the reason but people says it.

Thats fine. I changed the application to x86 ie 32 bit. Then I tried to load sos. But surprised to see another error message.

.load C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll
Error during command: IDebugClient asked for unimplemented interface
Error during command: Exception c0000005 occurred at 2A5E7EAE

This is interesting. I had no other way except googling the same error message. The first result itself told that there is an issue reported to Microsoft on this

If there is .Net 4.5 installed we cannot load sos in VS 2010

http://connect.microsoft.com/VisualStudio/feedback/details/742882/after-installing-visual-studio-11-beta-load-sos-fails-in-visual-studio-2010

I have .Net 4.5 installed in my machine which I cannot un-install for this silly thing. Yes I had to go back to WinDbg, the command line world to do my job.

What,Why & How to resolve OutOfMemoryException in .Net

This is a normal exception happens in most of the .net applications. When ever developers see this they start thinking about the RAM of the system and divert their thoughts towards physical memory. But before going to that area there are some other things to be noted. 

What is OutOfMemoryException

• An Exception in .Net which will be fired when the .net runtime is not able to allocate memory for object
• The MSIL instructions such as newobj,newarr and box may throw this exception

Why OutOfMemoryException is occuring

• Its not actually because there is not enough memory in the machine.(Memory shown in Task manager) 
• Its because there is not enough continuous free memory in the process's address space to do memory allocation for objects

How to resolve

• Facts before you try to solve this issue.
• There is no silver bullet to solve this ie There is no single line solution
• You need to have a better understanding about operating system's memory management , its limitations and the working of .Net GC.
• The memory consumption showing in Task Manager is not the actual memory used by the application.
• Steps for quick fix.
• Convert the application to 64bit. This will remove the 2GB memory limit on process and largely increase the process address space so that there will be lesser fragmentation which provides more continuous free memory space.
• Steps for actual fix
• Use tools such WinDbg, .Net memory profiler etc... to find out the size of objects in the process memory. In most cases this will be very less than the memory shown in task manager.You may also use performance monitor counters to do the same.
• If the size of objects and task manager reading has less difference ,it means you actually uses so many objects or the object you created are not deallocated properly. Check for memory leaks and fix it.
• Check for LOH (Large Object Heap) fragmentation and fix it.
• If you really want to work with huge objects use MemoryFailPoint class

Reference
http://blogs.msdn.com/b/tess/archive/tags/memory+issues/
http://msdn.microsoft.com/en-us/magazine/cc163528.aspx
http://stackoverflow.com/questions/4767651/what-is-the-cause-of-outofmemoryexception-in-net-on-the-windows-service-under-a