Azure @ Enterprise - Why I recommend Kubernetes as app hosting platform?

I started writing this post 2-3 years back. Mainly when Apache Spark 2.3 started supporting Kubernetes (K8s) in 2018. It was obvious that Kubernetes is taking over app hosting space the same way virtual machines took over physical machines. All are expected to understand where the industry is moving and adopt. Hence I paused this post as there is nothing I need to endorse. But it's time to resume this post and publish it.

If you are already using container and K8s orchestration, feel free to skip this post and save time.

Basics

Containers

Please refer to my 2015 post about Software Containerization via Docker - First look and some thoughts for very low-level basics. The container can be understood as a micro virtual machine with the application(s) and runtime isolated from each other. It has a file system inside it, it can expose ports for networking. Mount external storage.

This mainly co-evolved with the microservice architecture where the huge systems are separately developed and deployed as small services that own their own data and release cadence.

Kubernetes

Officially 

"Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation." - Kubernets.io

In short, it is the container orchestration engine.

What is and why do we need container orchestration?

 In my own words, orchestration is  

• Where to run
• Horizontal scaling - Mechanism to use multiple nodes(VMs are called nodes) to run containers of the same system. Nodes are expected to fail at any time.
• Deciding where to run the container - The mechanism to specify what node executes what container. Eg: a particular container in the solution may need to run on the Linux node another may need a Windows node. Some containers may need GPU to be present on the node
• Can also include virtual nodes that even can use Azure Container Instances.
• Deployment and rollback
• Ability to rollback easily in case something goes wrong in deployment
• It is expected to have failures in deployments but the orchestration system should be able to roll back easily to the previous known state.
• Networking
• Determine how the containers are allowed to communicate with each other and what ports are exposed to outside.
• Also has load balancing capability
• Also has routing capability.
• Service discovery
• The services/components in the system should be able to discover the endpoints to other services as the services may be deployed to multiple nodes.
• Resource allocation
• Determines how much system resources need to be given to each container.
• Storage mounting
• Ability to mount external storage
• Scaling
• The mechanism to specify how many container instances are to be running at a time. Eg: Based on Azure queue messages it has to scale out / horizontal scale.
• Secret and configuration management
• Secrets can even be stored in Azure KeyVault and places like that. But that may not work on-prem. An orchestration engine needs to have secret management.
• Configuration management is another aspect of any software system. The containers need to be initialized with proper configuration wherever it runs.
• CRON
• Capability to execute a container based on CRON expression.
• Health monitoring & self-healing
• Monitor the state of the containers and make sure the state is true to the definition of the system. Eg: If a container crashes, the orchestrator makes sure it's started again.

Hope the above explains what and why orchestration?

Links

There are people out there who can explain orchestration better than me.

https://kube.academy/courses/kubernetes-101/lessons/why-container-orchestration

https://www.redhat.com/en/topics/containers/what-is-container-orchestration

https://www.ibm.com/cloud/architecture/content/course/kubernetes-101/dealing-with-many-containers/

https://docs.microsoft.com/en-us/learn/modules/intro-to-kubernetes/2-what-is-kubernetes

Why Kubernetes?

It is the currently available best container orchestration engine. It supports most of the capabilities mentioned in the above list of what and why orchestration. For further reading refer to K8s documentation.

What Kubernetes is not?

It is equally important to understand what Kubernetes is not for. They

• Don’t have native CI/CD concepts.
• Don't have a data storage engine. RDBMS Rollbacks are not supported out of the box. Better be backward compatible. 
• Don’t have a built-in blue-green deployment strategy
• Don’t have a built-in serverless function or event-driven execution
• Don’t have a built-in message broker, service bus, etc… we have to use external or run a container for message broker.

More in their documentation.

How and where to host

K8s can be run on top of virtual or physical machines or in the cloud.

On-premise

It can be installed on-premise on top of virtual machines or physical machines. The machines are called nodes as well. Control nodes need to be Linux, worker nodes can be either Windows or Linux. If your organization is a hard-core windows shop, no luck here.

Cloud

Major cloud providers provide an easier way to host K8s applications.

• Azure - AKS
• AWS - EKS
• GCP - GKE

Why Kubernetes is good for enterprises?

• Vendor-neutral and portable
• If cloud providers increase fees, easy to switch from one vendor to other
• Worst case start and maintain on-prem K8s cluster
• Never do own cluster if you are a start-up and hosting a single application.
• No vendor lock-in due to native PaaS model
• If we start using Azure Function, Lambda, etc... that has vendor-specific API, there is a high chance that we get stuck with it. Remember the Azure Cloud web and worker roles that are in extended support now.
• Environment parity
• Less 'it works in my machine' comments from developers
• High density
• Can pack more things in resources. Less wastage.
• Multi-tenancy
• Multiple apps of the enterprise can share a single cluster via namespaces-based isolation.
• Fewer operations cost
• Due to high density and multi-tenant nature less operation cost to make the cluster up and running.
• A clear view of architecture
• Just reviewing the YAML files will give a clear idea of how the system is designed. No more surprises because the settings and configurations are spread across many places.
• Repeatability
• Since the container is immutable, it can be used years later if we want to see how the app did run earlier.
• No need for private cloud software
• An on-premise cloud can be created on the K8s cluster without any private cloud software such as OpenStack. The K8s cluster is the cloud where applications are separated by namespaces. Oh yes, the storage needs to be managed.

Learning curve

It will not be an easy journey to the Kubernetes. Engineering teams (Dev and DevOps if you have separate teams) need to be upskilled as there are a lot of new concepts in K8s. Some below

• Namespaces
• Nodes - Control and worker nodes
• Controllers - ReplicaController, NodeController
• Deployment
• ReplicaSet
• Pod
• StatefulSet
• DaemonSet
• Service
• Job, CronJob
• Volume, Persistent Volume, etc...
• ConfigMaps
• Secrets
• Policies, Resource Quota
• Selectors
• YAML (Not that specific to K8s)
• Kubectl

I didn't get a comprehensive mind-map showing all so planning to prepare one. It is advisable to upskill the team by hiring external people who already have experience with K8s. Converting an existing team that doesn't know K8s to experts is tough unless you are ok to accept failures.

Adding Azure VM auto shutdown via ARM template

This post is intended to show how can we use a simple Azure feature via ARM template. The feature is 'scheduled shutdown of virtual machines'. It is useful when we play with machines. Seriously, never shutdown production machines unless there is a real business scenario such as your business is strictly running during a particular time and you don't want to keep the system open for hackers to play with. eg: day trading.

Before we go ahead, this article doesn't promote the use of virtual machines. Virtual machines which is in cloud terms 'Infrastructure as a Service' is the last to consider when we design for cloud. One, it is costly. It is not truly pay as use mode. Virtual machines have to run regardless there is workload or not. We have to pay for all these idle times. Also virtual machines take more time to scale out as the boot time is more. By the time, the spike in requests/workload might be over. But there would be scenarios where we have to use VMs. When we deal with older systems without enough budget to rewrite, we have to go for VMs. Also when our corporate IT is not believing in PaaS because of security, VM is the option.

Now coming to the point. Auto shutdown can easily be done from portal as shown in the announcement. There are some sites out there which explains how to setup auto shut down via ARM. This is my experience with the same.

The schema of ARM template can be found in the below link. It is the starting point.
https://docs.microsoft.com/en-us/azure/templates/microsoft.devtestlab/schedules

Sample ARM template to shutdown

Below goes JSON to set auto shutdown to a VM

{
      "apiVersion": "2017-04-26-preview",
      "type": "Microsoft.DevTestLab/schedules",
      "name": "shutdown-computevm-nanotest",
      "location": "[resourceGroup().location]",
      "properties": {
        "status": "Disabled",
        "timeZoneId": "Pacific Standard Time",
        "taskType": "ComputeVmShutdownTask",
        "notificationSettings": {
          "status": "Disabled",
          "timeInMinutes": 15,
          "webhookUrl": "http://joymononline.in"
        },
        "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', parameters('nanoName'))]",
        "dailyRecurrence": {
          "time": "1600"
        }
        //"hourlyRecurrence": {
        //  "minute": "1"
        //}
      }
    }

We can see webHookUrl is given to a dummy url and hourlyRecurrence is commented. Lets see the details.

Where to add this JSON fragment in ARM template

This is not a sub resource of VM mode. Rather this needs to be directly added into top level resources node in ARM template.

Can we shutdown VM hourly

If we are using an Azure subscription which credits 50$ per month, it is not good to set auto shutdown daily one. As soon as we are done with the VM, it should be shutdown. Even if we forget, it has to shutdown. But currently the feature doesn't support hourly recurrence though the schema supports. That is why it is commented.

Can we omit daily recurrence value?

Below is the error which will pop when we don't give value for daily recurrence. Notice that this error won't be displayed during validation. But throws when we execute ARM template with ARM manager in Azure.

17:48:25 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:48:25 - [ERROR]   "error": {
17:48:25 - [ERROR]     "code": "MissingRequiredProperty",
17:48:25 - [ERROR]     "message": "Missing required property DailyRecurrence."
17:48:25 - [ERROR]   }
17:48:25 - [ERROR] }'

What is big deal with the name

Below is the error receive when a pet name given to schedule. Again notice that validation don't show but execution throws. In other words its runtime error.

17:55:49 - [ERROR] New-AzureRmResourceGroupDeployment : 5:55:49 PM - Resource
17:55:49 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:55:49 - [ERROR]   "error": {
17:55:49 - [ERROR]     "code": "InvalidScheduleId",
17:55:49 - [ERROR]     "message": "The schedule should be created in subscription
17:55:49 - [ERROR] 11111111-1111-1111-8888-####ffffeeee, resource group <Storage>
17:55:49 - [ERROR] and with name shutdown-computevm-nanotest."
17:55:49 - [ERROR]   }

The nanotest is the VM name.

APIVersion and region

The sample template above shows apiVersion of schedules '2017-04-26-preview'. It is region specific. Not all regions supporting all API versions. This is again an runtime error. Validation will pass.

There is a URL which tells what are available in what region. There are some programmatic techniques to find out the API version and other details.

The good part is that the error is descriptive, telling us what API version to use. That is really useful.

Rolling back ARM deployment

As mentioned above some errors happen when we deploy/execute ARM template. If the auto shutdown setup fails, does it revert the VM deployment? As software engineers we assume that it will,  same as MSI deployment or there will be some flags to deploy ARM in transaction. But it never roll backs. It is our pain to clean up what is left after a partially failed ARM deployment. Feature request below. It is there for the last one year.

https://feedback.azure.com/forums/281804-azure-resource-manager/suggestions/13701420-rollback-template-deployment-if-anything-fail-ju

The suggestion is to make our ARM template deployment idempotent. Simply saying even if we redeploy, it should not cause any harm. It should run many times by producing same result as first run. Simple if we are just setting up architecture via ARM. Gets complex if there are custom scripts running as part of deployment.

Debug ARM template

For a simple feature to get working, we may encounter these many errors. As developers we might have started thinking how to debug the ARM templates. In general the error messages are meaningful, if we are familiar with Azure terms. Debugging ARM template itself has scope for separate post. Again, it gets complicated when ARM template uses custom scripts. Below are 2 links talking about troubleshooting ARM template deployments.

https://azure.microsoft.com/en-us/blog/debugging-arm-template-deployments/

http://www.jamessturtevant.com/posts/Debugging-ARM-Templates-with-Custom-Scripts/

Extensible Chaos Monkey

Chaos Monkey

Chaos Monkey is a great concept introduced by Netflix to create random issues in cloud server environments so that those issues can be addressed early and test against unexpected failures. That makes sure any system can recover from any kind of failure which can happen in production.

The good news is that they didn't keep it secret. They made that tool online including source in Github. It is written in Java so it was a little difficult for the .Net community to extend. There are many .Net ports of the same tool or tried to implement the same.

The problem

In our company also things are not different from any other production environment. There are issues which are first reported / only happens in production. We have an online audit system that contains many backend processing components. The backend queueing system is developed in such a way that any application track developer can add a new queue type and write it's associated handler. Though there are guidelines for the backend developers to roll back properly, there are times when developers are not caring it to meet the deadlines and it goes to production. In production, we end up with the wrong data states. Some of the reasons are unexpected IIS / AppPool recycling. Database timeout/outage etc...Since the application is not expecting those states, it will be clueless about how to recover from there. Finally, support has to manually run SQL statements in production servers to correct the state.

Ideally, QA is supposed to test scenarios such as IIS reset and all when the backend services are running. But they have limitations. It is very difficult to make sure that they cover all the scenarios.  Ensuring the randomness and tracing the abnormal event to reproduce in dev will be difficult. There are also difficulties in manually creating abnormal scenarios when the tests are running overnight.

Solution

All these things lead to a testing strategy where issues / abnormal scenarios that are expected in production needs to created in dev/QA environments so that we can identify how the system behaves to those abnormal events and sometimes change the application flow to recover those states.

This is the point we started looking towards already existing ChaosMonkey solution. It seems suitable for us. But we are not able to use it straight away because ChaosMonkey is targeted towards cloud but our production deployment is in house.

Why extensible

So we tried to extend the tool. But since its in Java and most of our developers are familiar with .Net ecosystem, we started looking for .Net port of Chaos Monkey. Unfortunately not able to find much.  Though there are some they are also targeted towards the cloud. To restart the local IIS server, had to write a lot of code. 

So finally decided to take one as a base and make it extensible to meet the scenario. Forked ChaosMonkey implementation from Githb by Simonmnro and added my own changes to make it pluggable.

My version of Extensible chaos monkey is available in the below location.

https://github.com/joymon/ChaosMonkey

What's Next

• Clean separation of plugin code and config
• Adding more plugins such as increase memory pressure.

Feel free to comment if there is a better solution to the issue and contact me if willing to contribute to the project.

Hosting WCF service in IIS and System.Web.WebPages.Deployment

As per MSDN the System.Web.WebPages,Deployment namespace contains "the classes that are used to confirm the deployment of web applications". It make sense for web applications which contains web pages created using .aspx or Razor. But what is the relation between this System.Web.WebPages.Deployment.dll and WCF service application which hosted inside IIS which is exposed via netNamedPipes ?

As we have seem in my previous post about SSMS and System.Transactions.dll, I got a machine crash and the machine started behaving in its's own way which resulted in problems. Now when I host WCF services inside IIS there are problems. The services, I was able to browse before the crash stopped working with below error message.

Could not load file or assembly 'System.Web.WebPages.Deployment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The module was expected to contain an assembly manifest

The call stack looks as follows

---

[BadImageFormatException: Could not load file or assembly 'System.Web.WebPages.Deployment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The module was expected to contain an assembly manifest.]
   System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +0
   System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +34
   System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +152
   System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection) +77
   System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) +16
   System.Reflection.Assembly.Load(String assemblyString) +28
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +38

[ConfigurationErrorsException: Could not load file or assembly 'System.Web.WebPages.Deployment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The module was expected to contain an assembly manifest.]
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +752
   System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai) +57
   System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig) +170
   System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies() +91
   System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath) +258
   System.Web.Compilation.BuildManager.ExecutePreAppStart() +135
   System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +516

[HttpException (0x80004005): Could not load file or assembly 'System.Web.WebPages.Deployment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The module was expected to contain an assembly manifest.]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +9873912
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +101
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +254

---

One possibility for BadImageFormatException is the mismatch of runtime. ie if we try to load x86 assembly in x64 application or other way we may get this exception. I tried enabling 32 bit support in the application pool by setting the "Enable 32-Bit Applications" flag in IIS. 

It did't help.

Then looked at all the config files possible such as aspnet.config and machine.config. Nowhere I was able to find any entry to the problematic dll. Then I decided to find out the assembly binding information which is as follows

=== Pre-bind state information ===
LOG: User = NT AUTHORITY\NETWORK SERVICE
LOG: DisplayName = System.Web.WebPages.Deployment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///<drive>:/TFS/TestDotNet/WcfService1/
LOG: Initial PrivatePath = <drive>:\TFS\TestDotNet\WcfService1\bin
Calling assembly : (Unknown).
===

---

Here its clear that the calling assembly is unknown which I assume any framework dll. May be it was earlier getting loaded from GAC and after the crash GAC registration might be broken. So I figured out the location of he dll and registered into GAC.

Location of System.Web.WebPages.Deployment.dll - <Install drive>:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v2.0\Assemblies

But the GACUtil failed with a message as follows.

The module 'System.Web.WebPages.Deployment.dll' was expected to contain an assembly manifest.

Then I tried open the dll in reflector and could see that, it was not able to open as its no more a .net dll and it displayed more detailed error message

File is not a portable executable. DOS header does not contain 'MZ' signature.

This leads to a conclusion that the System.Web.Webpages.Deployment.dll is corrupt. So next option is repair or re-install. So went to appwiz.cpl and re-installed "Microsoft ASP,Net Webpages 2".

It worked!!!

But why WCF hosted in IIS needs System.Web.WebPages.Deployment.dll which is for web page related :-(

.Net Assembly versioning in Plug-in framework

As everybody knows assembly versioning in .net is a good concept. It helps us to deliver upgraded versions very easily without affecting behaviour of old versions ie easier to maintain backward compatibility. It also helps us to sign our assemblies with strong name which means our delivered dlls are unique.

In normal cases, the assembly version contains 4 numbers separated by period "." . Those denote Major version,Minor version, Build number and Revision number respectively. When to change or increment the assembly version is a tough decision especially when it comes to applications which uses reflection extensively. As best practices some people changes the assembly version for the shipping builds only. Anyhow when we change the assembly version we need to make sure that all assemblies including dependent assemblies are loading correctly as those are signed and needs to be in same version.

In my current project we are changing the assembly version once in a year as we are using reflection in its maximum. Recently we had to deal with a versioning issue related to a plug-in architecture, we introduced last year. For simplicity I can explain the plug-in architecture using simple drawing application framework which uses IShape and its implemented classes.

Problem

The aim is to have a plug-in architecture in place where there will be a framework which will be always latest. There will be an interface IShape with a method Draw().The framework creates the object of IShape implemented classes based on configuration using reflection and calls its Draw() method. We released it in 2013 and the structure was as follows

• Core.dll {Version:1.0.0.0}
• Contains IShape interface
• Framework.exe{1.0.0.0}
• The controller class which creates the object of IShape using reflection and call Draw()
• Impl10.dll{Verion:1.0.0.0}
• Contains Circle class which implements IShape.Draw() method to draw circle shape.

In 2014 we are not supposed to release Impl10.dll as there is no change. But the Framework.exe might have feature improvements and it's version needs to be incremented. Similarly the version of Core.dll needs to be incremented as its always expected to be latest.

The problem starts from here. If we deliver the new version of Core and Framework with incremented assembly version 2.0.0.0 for Core.dll and Framework.dll, the Impl10.dll will not get loaded as its in old version.

Dlls present after new release.

• Core.dll {Version:2.0.0.0}
• Contains IShape interface
• Framework.exe{2.0.0.0}
• The controller class which creates the object of IShape using reflection and call Draw()
• Impl10.dll{Verion:1.0.0.0}
• Contains Circle class which implements IShape.Draw() method to draw circle shape
• Impl20.dll{Version 2.0.0.0}
• Contains Rectangle class which implements IShape.Draw() method to draw rectangle shape

Possible solutions

1. Assembly binding redirection 
   Using this technique we are forcing the .net runtime to use latest version of Core.dll even for Impl10.dll which is pointing towards previous version of Core.dll .But needs to maintain list of all old versions.
2. Manual Assembly resolution
   
   This technique loads the assembly by writing the assembly loading code in the AppDomain.AssemblyResolve event.
3. Constant version for Core.dll
   Simple one. Don't change the version of Core.dll even there are changes for new features. So that always the classes are implementing same interface. The assembly version of implementing classes can be incremented.

We opted the 3rd solution as its simple. Attached a sample which explains the plug-in scenario.

Things to remember

When we select any of these options we need to make sure below points

1. Never remove any method or change the method signature in interface methods. If we do so .net runtime verification will fail and it will throw method not found exceptions if we try to load old assemblies.
2. If we want to add more methods to the interface, we need to create new interface inheriting from IShape and have new methods there. When creating objects in framework.exe make sure we cast to respective interfaces to invoke operations. Since deployed framework.exe will be always latest, it can be done easily.