Azure @ Enterprise - Connecting to Management API using Azure AD App certificate

Introduction

When we develop large multi-tenant applications, we often face requirements to dynamically provision infrastructure resources, which we never need to do for simple applications. For example, assume the enterprise security practice or business requirement demand to isolate tenant's data into separate databases. Then we will have to dynamically create databases on the fly, when a tenant joins the application. This is drastically different than working on a simple application where only one database is storing everything. The maximum we have to deal with are the indexes and partitioning in database level and load balancing at front end level.

When we create or provision the resources from application, there will be so many security related questions to be answered. What if someone hack into the application and delete the databases? How to handle the noisy neighbor problem? The list is large.

To handle security, the on-premise solution is to have separate service accounts. Those have permission to create databases and isolate that service from web services exposed to clients machines. Expose that only internally either by authorization or by exposing via netNamedPipe binding in WCF.

Managing resources in Azure

Cloud computing is expected to solve all the infrastructure provisioning issues. Azure does that well. When enterprise meet Azure, all the security related things mentioned above gets reevaluated. In cloud it gets important otherwise an attack or poor code may create large number of highly priced resources which directly affect the financials. Or resources can be deleted which brings the entire system down. In on-premise systems there is very limited or no way an attack can delete a virtual machine. But in cloud its not.

How to secure a component which does infrastructure provisioning? This problem can be solved in Azure many ways. We can have a service which is secured using Azure AD and only be exposed inside enterprise's own virtual network(vNet) in Azure. But then the question comes how to secure the Azure AD? Azure AD supports different types authentications and enterprise like the MFA and certificate based auth. The latest in the series is Managed Service Identity.

MFA - Multi Factor Authentication helps secure something exposed to users who can look at the security token and enter the same in web page or device. But for service to service communication, a scheduled job or queued operation to service communication, MFA is not suitable. The certificates helps there.

Securing in the world of Microservices - Automation RunBook?

In a large enterprise, there could be so many applications which are multi-tenant and need infrastructure provisioning. Enterprise may have only one Azure subscription for all these. In such scenario giving the certificates which have the privilege to create Azure resources to all those apps will not be feasible or those apps cannot be run with that level of high privilege.

One solution in Azure is to use Azure Automation. Automation Runbook can run in high privilege and can create Azure resources. That can be exposed via Webhooks to the applications. Applications can invoke the Webhook with some kind of application identity or developer key in request header. Once the runbook starts it can check for the application key and do actions if allowed. Please note that Webhooks don't have security mechanism built in. URL has a secret token and whoever knows the URL can invoke. The runbook can check for header and validate.

Writing RunBooks is easy and there are lot of tutorials available how to get it right. 

But there is a problem remaining. Webhook return a JobId. How do the applications check status of the Job?

Callback?

We will end up again in certificates, if we need to use the Azure Management API. But it is easy to do status reporting if the automation runbook accept a callback URL and invoke that on Job completion.

Webhook accepting another Webhook on completion may make things complicated but that is good solution without polling.

Unfortunately if we end up in polling, below are the code snippets which can be used to get the Automation Job status using .Net SDK. There are so many code snippets available in internet but very difficult to get working code which uses cert to auth into Azure Management API. 

Since the authentication APIs accept strings and the names are confusing, its gets complicated easily.

Code snippets

Below is the entry point which accepts the necessary inputs to locate an Azure Automation Job

---

private static async Task<JobGetResponse> GetJobResponse( string subscriptionGuid,string resourceGroupName, string AutomationAccount, string JobId)
{
            AutomationManagementClient client = await AutomationManagementClientFactory.Get(subscriptionGuid);
            return client.Jobs.Get(resourceGroupName, AutomationAccount, Guid.Parse(JobId));
}

---

The return ed JobGetResponse has Job property which exposes most of the properties of Job.

In order to get the code working, we need a valid AutomationManagementClient. How to properly give the string values into the flow is the trickiest part. .

---

internal class AutomationManagementClientFactory
{
    internal static async Task<AutomationManagementClient> Get(string subscriptionGuid)
    {
        string token = await TokenFactory.GetAccessToken("https://management.core.windows.net/");
        TokenCloudCredentials tcc = new TokenCloudCredentials(subscriptionGuid,token);
        return new AutomationManagementClient(tcc);
    }
}

---

This depend on the TokenFactory. But before going there the catch here is on the hard coded URL. The URL is to the management end point. Lets see the TokenFactory class

---

internal class TokenFactory
{
    /// <summary>
    /// Get Access Token
    /// </summary>
    /// <param name="resource"></param>
    /// <returns></returns>
    internal static async Task<string> GetAccessToken(string resource)
    {
        var context = new AuthenticationContext($"https://login.windows.net/{Configurations.TenantId}", TokenCache.DefaultShared);
        var assertionCert = GetClientAssertionCertificate(Configurations.AzureADApplicationId);
        var result = await context.AcquireTokenAsync(resource,assertionCert );
        return result.AccessToken;
    }
    internal static IClientAssertionCertificate GetClientAssertionCertificate(string clientId)
    {
        string certIssuerName = Configurations.GetConfigValueByKey("CertificateIssuerName");
        X509Certificate2 clientAssertionCertPfx = CertificateHelper.FindCertificateByIssuerName(certIssuerName);
        return new ClientAssertionCertificate(clientId, clientAssertionCertPfx);
    }
}

---

The responsibility of the class is to get authentication token towards a resource and the resource here is the Azure Management end point. The authentication context using the Azure AD tenant Guid to get token. The TenantId is not the Azure AD Application id. 

It uses certificate which is found using the issuer name. The criteria to find certificate can be anything but the rule here is that the certificate should be same as the certificate used for Azure AD Application. The Azure Application's id is obtained from the config. It has to be Application Id not the object id of Azure AD App.

The signature may confuse us. The client assertion certificate uses application id, but the parameter name is client id to make it generic.

The last thing is the certificate helper. As mentioned above, how we get the cert is not relevant as long as its the right certificate. Adding the code for that as well.

---

public static class CertificateHelper
{
    /// <summary>
    /// Find Certificate By Issuer name
    /// </summary>
    /// <param name="findValue"></param>
    /// <returns></returns>
    public static X509Certificate2 FindCertificateByIssuerName(string findValue)
    {
        using (X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser))
        {
            store.Open(OpenFlags.ReadOnly);
            X509Certificate2Collection col = store.Certificates.Find(X509FindType.FindByIssuerName,
                findValue, false); // Don't validate certs, since the test root isn't installed.

            return col.Count == 0 ? throw new CryptographicException($"Certifcate not found") : col[0];

        }
    }
}

---

Prerequisites / Environment setup

• Azure AD Application which has permission to resource group where the automation account reside.
• The above Azure AD application to accept certificate to get token back. So install the certificate to the proper store. In this case it search in Current User's personal store. If this code runs from IIS web application using service accounts, the store can be different.

Why the code snippet is important?

When we get the snippet working and look at the code we feel its simple. But when we get a situation when this doesn't work, we cannot understand anything such as what is client id, what is resource id etc...

The hard coded strings are applicable in public Azure cloud. When the code runs in Azure Government or other tenants the values will differ.

Exceptions

Below are some exceptions which may occur during the development

Access token from wrong audience

The below exception may occur if the token is obtained from TokenFactory is not associated with the right resource. 

"The access token has been obtained from wrong audience or resource ’https://management.core.windows.net'. It should exactly match (including forward slash) with one of the allowed audiences ‘https://management.core.windows.net/’,’https://management.azure.com/’"

Enjoy...

Interception in Microsoft Unity framework for .Net

Introduction

This article is to demonstrate how we can combine dependency injection framework with logging og low level execution flow of application.

Prerequisites

• Knowledge about dependency injection.
• Knowledge about unity from Microsoft for dependency injection

Background

If we develop any application, logging is an essential thing to troubleshoot issues which comes in  later stages. Ideally there should be no bugs in production, if we test well. But it is not practical 99.99% of times. Sometimes even if there are no defects we may need to understand how the system is getting used. We call it as instrumentation, telemetry etc...Whatever we call, we have to make sure our application is capable of telling us what it did. Writing log lines inside our business logic has more efforts/costly, makes it longer and very difficult to read. 

Instrumentation technology considerations

Below are the factors to be considered when we select any logging and instrumentation technology for software.

• Performance - Due to the logging operations our application should not be performing slow. Logging is for application not the other way.
• Controlling behavior - We should be able to turn the logging on and off without restarting the application. Also change the level of logging without restarting.
• Compatibility - If our system spans across different technologies better choose one which works everywhere.

Options

Traditional options include inserting log lines at the beginning and end of every methods and additional lines if the logic inside complicated. Another option is to inject the logging code at runtime. Both has their own pros and cons.

Few words on ideal composed application

If we look at SOLID principles we could see that all it tells us is to develop apps in a modular way where modules can be composed easily. In other words single responsible, closed components connected together to become an application. When we say modules, components in general they all translate to classes or functions in code.

How we compose the application by connecting the components is via dependency injection framework.

Combining DI framework and logging

Now most of the people might have got the click of idea. If the application follows SOLID principles religiously, why can't our DI framework log the method calls as it knows how application is composed. When we say how the application is composed it is same as the application flow which we need to log.

This is supported in Microsoft unity framework which provides DI capabilities for .Net. There are other DI frameworks available, one other called MEF which is native to .Net itself. This post is not to discuss what is the best DI framework but on the possibilities of combining DI with function level logging.

Steps in general

• Have a interceptor mechanism which can tell us what is happening in the DI framework
• Let the interceptor log the details into appropriate medium using best technology.
• Register the interceptor into the DI framework.

Understanding sample

Coming directly to the sample which is available in the below location.

https://github.com/joymon/dotnet-demos/tree/master/patterns-practices/DI/unity/Interception

class LogCallHandler : ICallHandler

This is the interceptor we have written. It has Invoke() implemented which came via ICallHandler. Invoke() will be called by the unity DI framework when things happen there.

This is one mechanism unity supports. There are other ways to hook interceptors.

UnityConfiguration.xml

This is the configuration file which tells the unity framework about the components and how they composed. In simple words when someone request object of x interface give object of y class which implemented x. This file also has options to register interceptor.

This configuration can be specified via code too.

class Logger

This is a simple logger class which logs via .Net trace. .Net tracing is configured in the sample to write to file.Refer app.config file. We have to consider intercepting the method calls and writing the logs separate. If our log writing takes more time the performance of application may get affected. Choosing log writing mechanism is not in the scope of this article. In short, recommended mechanisms are ETW for on premise and Application insights for Azure environments.

References

https://msdn.microsoft.com/en-us/library/dn178466(v=pandp.30).aspx
http://mohtasebi.com/patterns/2013/10/24/using-unity-for-aspect-oriented-programming-and-interception-part-1.html

Handle once for many events via sliding timer.

As a developer we might have encountered many situations where we have to do certain things when something triggers but it triggers so frequently which makes our handler code run many times and it affecting performance. The functionality might be achieved by responding to last trigger. In .Net way, we can take events as triggers and if we want to handle event which is firing frequently it affects the performance.

Coming to an example for more clarity. Suppose we need to do something when the network is connected or disconnected, we can hook into the NetworkAvailabilityChanged or NetworkAddressChanged and write the code. But the network sometimes can get fluctuations, and these events may fires frequently but we don't need to respond for each event but to the last event. This is same if we want to show the MousePosition. We should only handle the last mouse move event. Not all. 

The event is firing rapidly. Need to handle the last one.

The major problem here is to identify which is the last one. In other words how much time we need to wait without any event firing to determine the last one.

This is purely the decision of the developer based on the situation. In some situations we can easily conclude that, if there is a 1 second delay the previous one was the last event. Sometimes that delay may be 5-10 seconds or can be small in milliseconds.

In short, many event firings happened but no need to handle once for all by determining a last point. This can be done using sliding timer mechanism or simple timer monitor.

Sliding timer

There is no  timer class in .Net framework called SlidingTimer. This is just a mechanism to reset the time based on triggers. If no event triggered, for a specified time, the timer ticks and the operation can be done.

When this should not be used

 If every event needs to be handled we cannot use this sliding timer mechanism. For example a sensor event or data feed where each and every event brings different data and needs to be processed.

Sample code

Below is a sample WPF implementation. This sample is trying to show the location of a window in a textblock whenever the window is moved. Since the LocationChanged event fires rapidly it uses SlidingTimer to determine when the user stops dragging the window. In other words, if the LocationChanged firing started and stops firing for 2 seconds, it the best time to update the location in textbox. First lets take a look at the consuming side.

private readonly int IdleTimeInMilliSecondsRequiredToProcessEvent = 2000;
SlidingTimer slidingTimer;
private void InitializeAndStartSlidingTimer()
{
    if (slidingTimer == null)
    {
        slidingTimer = new SlidingTimer(IdleTimeInMilliSecondsRequiredToProcessEvent);
        App.Current.MainWindow.LocationChanged += MainWindow_LocationChanged;
        slidingTimer.Tick += (sender, args) =>
        {
            this.Dispatcher.BeginInvoke(new Action(() =>
            {

                string loc = $"{App.Current.MainWindow.Left},{App.Current.MainWindow.Top}";
                this.LocationTextBlock.Text = loc;
                Console.WriteLine(loc);
            }));
        };
    }
}

private void MainWindow_LocationChanged(object sender, EventArgs e)
{
    Console.WriteLine("Event fired restart sliding timer");
    slidingTimer.StartOrRestart();
}

The above code can be in any WPF UI component's code behind file. Prerequisite is to have a TextBlock called LocationTextBlock in the UI class. InitializeAndStartSlidingTimer() can be called in the beginning or at any suitable time to initialize the SlidingTimer. Since the Window.LocationChanged is subscribed after initialization, its fine here. Those can vary depends on our scenario. 

Here the SlidingTimer is initialized with an idle timeout which required to tick after it is started or restarted. In the MainWindow_LocationChanged event handler, the slidingTimer is restarted. If there is no StartOrRestart() called continuously for 2000 milliseconds, the Tick event will fire. The Tick event handler uses dispatcher to manipulate the UI as the event fires in different thread.

Now lets see how the SlidingTimer class looks like

internal sealed class SlidingTimer
{
    Timer _internalTimer;

    private void OnTick(EventArgs args)
    {
        if(Tick !=null)
        {
            Tick(this, args);
        }
    }
    /// <summary>
    /// Fires in separate thread, when the there is enough delay to execute task.
    /// </summary>
    public event EventHandler Tick;
    int _intervalInMilliSeconds;
    public SlidingTimer(int intervalInMilliSeconds)
    {
        _intervalInMilliSeconds = intervalInMilliSeconds;
        _internalTimer = new Timer((state) =>
        {
            OnTick(EventArgs.Empty);
        });
    }
    internal void StartOrRestart()
    {
        _internalTimer.Change(_intervalInMilliSeconds, Timeout.Infinite);
    }
}

Here a thread timer is used for timing purpose. After it is started, the invocation of the callback event handler is controlled by the StartOrRestart(). Suppose the interval is 2 seconds, if internal timer starts and the StartorRestart() called after 1 seconds the total time slides to 3 seconds to get the event fired. This is simple sliding mechanism.

Monitoring timer 

The original use case can be done by using a normal timer as well. The event handler can just set a flag to indicate the processing is required. When the timer ticks it can check the flag and do the operation. After timer operation it needs to reset the flag.

Happy coding.

Architecture of www.Joymononline.in - Context diagram

It is very important as Application Architect to make sure the architecture diagrams are in sync with what is there in the code. Often the code and architecture will be going in 2 different directions. If we try to map a box in architecture diagram in to code, sometimes we will not able to locate, it might be spread across many components or one component may be doing duties of one or more boxes. This is because architects and developers talk 2 different languages. Architects follows their own diagram conventions and tools where developers have only one tool Visual Studio or Eclipse to code. One famous solution is to generate the code from the architecture modeling tools. But in the age of agile, people are getting away from this mechanism.

One of the solution to this is to adopt a diagraming strategy which make sure both the sides are in sync. Simon Brown has put forward one mechanism in his book "Software Architecture for Developers" to solve this issue. We can summarize it as "C4 Architecture Model" where the entire system is documented as 4 different diagrams which we arranged in a fashion that we can zoom in / out to get more details or holistic view. There are many videos about the modeling. Its easy as preparing 4 diagrams, context diagram, container diagram, component diagrams and class diagrams respectively

Below is an attempt to document my personal site using the C4 architecture model. At least diagrams

Joymononline.in - Context Diagram

Context diagram usually is a high level diagram to get a bird eye view about the system. It can be shared to business people and they can easily understand.

This is my personal portfolio site. This is not just another collection of HTML files. There are some integration points with other systems as well. Lets see how the context diagram looks.

Users

There is only one set of users hitting into the site as it don't have authentication or authorization. All public users. The content is edited via ASPX files or by modifying in other systems

External Systems

This represents what at the external applications JoymonOnline.in is connected to. At this point there are 4 external systems for different purposes.

• Blogger - used to pull the recent blog posts via RSS feed
• StackOverflow - Used to show the badge in Resume page. Just an img src pointing to SO url.
• Github - for storing content in git pages. This helps to update the content without redeploying the site. Earlier it was Evernote. Then moved to http page as there are limitation on the API usage.
• LinkedIn- Used for showing badge. This is just an img src to LinkedIn

This gives a context about the system ie here its JoymonOnline.in site. In the C4 architecture model the next level is container diagram. That is nothing but a zoom-in of JoymonOnline.in web site.

My lead doesn't like commented code

Boss, look at the code. I didn't comment it. Why did you reject the check in?

---

        public void Foo()
        {
 
#if false
            Console.WriteLine("This is a sample which will never get executed");
#endif
            Console.WriteLine("Current code goes here");
        }

---

I had hard time with people who doesn't understand the problems with the commented code. According to them it needs to be there because that functionality might be requested by business in future.

My answer to them as a software craftsman is to use any source control system, if they are not using. But now a days there will be no company in the world which doesn't use source control for development. At least they can host in free Github account. Sometimes I had to show dev leads, how to get history from TFS.

Another argument people put forward is the reusability. Don't confuse with real reusability of code. This is about going back to previous logic after some time period. They argue that if the code is commented,it can be simply uncommented. If we delete, we have to go back to history and take it back. Sometimes, new developers will not be knowing that the same had been deleted in the past and they will write the old logic again. My answer is to use dependency injection. This still feels too technical to most of the people. But its very simple to those who had tried it at least one time. Follow the single responsibility pattern religiously to split up the code to small units and compose the application by injecting dependencies. if there is a change requested, have that logic in a new class by implementing required interface and change the dependency injection catalog so that the new change will take effect. Let the old class be there unused. We can use it in future.

  

What is Microservice Architecture to me?

Microservice is a buzzword in current Software engineering world. Everyone wanted to say we are building microservices or eagerly want to adopt it.

Reasons for micro services

Microservice - Bring Agile to Architecture

If we look at definition, we can see its an architectural style, where system is divided to small autonomous processes which are coupled with programming language/platform independent API. But we had similar kind of architecture in SOA (Service Oriented Architecture). They were also communicating kind of stable independent APIs. But what is the main difference or advantage.

Frequent releases

Yes the ability to release  more frequently is the key. We cannot deny other advantages that system is clean with small services which knows how to do its duty well, system can easily be built with different programming languages, scale only required components/services instead of whole system. etc...

If we remember the introduction of agile, there were so many advantages advertised. Working code over documentation, customer collaboration, responding to change were some to mention. But what is the real key? The key was 2-3 week based sprints which delivers something. As of my understanding most of the service companies adopted agile only because there will be a release in 2-3 weeks and developers will work night and weekend every 2nd week. In waterfall, the night and weekend parties were once in 6 months. Most of the places only the concept of sprint was taken but other concepts like maintaining backlogs, avoiding documentation over code were discarded. There would be no release happened in the world without documentation saying that code is working. 

What is there in Microservice? Independent teams who take ownership of small services. The size of the service, kind of relates to the size of the team who delivers it. And deliver daily, twice a day or if possible hourly.

Yes most of the places we can see an emphasis on TDD when adopting Microservice. But that is something might be discarded to rush to release.

What is the bottom line? The architecture will be agile. A Microservice could be introduced at any time in the development phase. Servers are going to be flooded with services. The complexity of the system is going to be in the orchestration of Microservices.

Design Microservice

Whatever we think, the Microservices architecture is a reality and soon most of our apps are going to be Microservice based. As the software engineering is still evolving, only fittest will survive. Currently it seems Microservice is a fittest one. Of course there are some problems to which Microservices approach is not the best solution.

Here lets see what are the characteristics of Microservices? 

Independent release

If a service can be released independently of the other parts of the system, it can be considered as Microservice. Sometimes the new API methods we released in the Microservice might not have any consumer components as the consumers can release only after the service release. 

The concept of independence is really important, if we want to release too often. If it depend on other components we cannot release fast & often.

Independent data store

If we want to have independent release of a service, it's data store should be independent. If there is a database table and a component release upgrade the schema causing other components to fail, we cannot consider that component as a Microservice. In short Microservice should be the only one to own it's data store. If other components in the system want to write something to the store they should call service.

Independent development team

If we want independence, probably it should be done by independent team. This is something not with the technology. Organization structure needs refactoring and is tough. 

The speed of delivery is prone to errors as well and it is advised to have development team supporting the service. If the team is shuffled after each release or members change often, they cannot support service as their own baby and the micro service architecture might feel a failure.

Independent framework/platform/language

This is not a mandatory thing. If there is a pressing need for adopting a particular technology which is different from regular technology stack, Microservice is recommended. But only if the above 3 criteria are met. Even we can have Microservice based systems with one language and technology.

Adopt only if

Do not jump just into Microservice without proper assessment and planning. If everything is independent who coordinates all these to work as a complete system which cater business need? Below are the things to make sure before jumping into Microservice world.

Defined integration guidelines between components. http(s)

If you don't have defined integration strategy of Microservices, it will simply fail. The recommended integration if through http(s) protocol if the interaction is synchronous. If the integration if asynchronous, adopt message queuing, service bus etc... But make sure there are enough design sessions and agreements before adoption.

Upfront design on the API

The real world systems / business problems are always complex. In monolithic system, we were handling the complexity in big components. But in Microservice era, the individual components are simple, but the complexity is shifted towards the interaction of services.

This requires carefully designed APIs for each Microservice. API is always like Mumbai underworld. Once we enter, there is no coming back. Once we publish an API, we don't know who are all calling it. Correcting the API and communicating the same to all consumers to redirect to new API is always costly.

Deployment is easy and automated.

We have seen one of the major advantage of Microservice is the frequent releases. Is the installation going to be done manually every time? If so we will immediately feel that the Microservice is a bad idea.

Deployment should always be automated. A continuous integration and delivery pipeline is always essential.

Strong automation testing. Prefer 100% test coverage

There is no human required for day to day deployment. They just setup the CI & CD in the beggining and system will do deployment automatically. What about testing? Are we expecting a manual tester to test all the changes before we release daily?

That's again should be automated. Developers should write proper test cases for the API. Most of the companies are employing a separate team / person for testing, is on the assumption that a developer doesn't want to break his own code or psychologically they cannot find out errors in the system what they build . When developer test, he tests only positive or usual scenarios because he knows how the system works. But here in Microservice world. we are not entertaining separate manual testing team,So how to tackle it?

Its very easy to tackle. Some other developer needs to write the tests. If the existing testing team knows how to code API tests, integration tests, they can also do. Please note we are against manual click QA. Automation QA is always welcome.

Service monitoring & management

We are talking about explosion of services. There are chances that each class/component in your existing system such as logging, caching will become a service. Some services will be versioned services, some will be cumulative/in place update. We need to add more service instances and load balance when services show load. Some might not be even used. A machine which hosted a service might go down and we need to relocate the service without zero or a little downtime. How do we deal with all these scenarios? 

The answer is a strong service monitoring and management infrastructure. Easy way is to host in cloud, where this is free with cloud provider. We don't need to develop infra management application for increasing number of machine and deploying the service to it. Or to move service from one machine to another machine etc...

Versioning, dependency management, retirement strategy

V2 of Service A may require V4 of Service B to be present in the system where V1 of Service C require V3 of Service B. This means V3 of Service B cannot be retired until Service C is retired.

Why we need to retire a service. Over the time some services will be legacy and there will be nobody calling it. But it just stays in the production environment. It is the decision of the stakeholders to retire or not. If we plan to retire everybody should be notified that, it is retired so that no new service call these retire ones.

Another interesting scenario is whether the a service say Service X should know the location of other service Y? Though the Location has many things to consider, here let's take as the url of service. If we decide to keep the location of Y in X, we cannot move the Service Y to other machine or environment. Hard coded location will cause trouble. Suppose we started with all the services on premises and due to load, if we decide to move Service Y to cloud Service X, will simply fail in calling Y.

This requires a service registry which keeps track of all the services and their dependency. This can be another service but in fixed location of course.

If you have access to production

If the development is for a product your company owns and is server based, Micro services works great. If the product is delivered as an installer to your clients, it would be really difficult to manage, as they may be still installing the morning release when you give evening release.

So basically better choose, if the CI & CD pipeline can reach till production. We can refactor the other systems to Microservice based but will never get the advantage of frequent release.

What to do with UI

Till now, we were talking mainly about services, what about UI? Are we going to deliver UI twice a day? That is something interesting and will discuss in a future post.

---

Disclaimer - I work in a project which have 100+ services, though we officially not adopted the Microservices architecture. It just came like that and many services are sharing same data store. We are not doing independent release of services due to data coupling. So the above view is not from a real time successful Microservie based project rather from reading and sufferings of service management.

10 years in software industry - What I learnt?

What's this post about

As of  7 Nov 2015, I have completed 10 years in the software industry. I have started as developer support engineer and now working as application architect. Currently in my third company, working places range from Chennai, Kochi in India and New Jersey, US. Worked in products and projects. Lacks pure consulting. Technologies are mainly Microsoft, little bit of Android and HTML,JavaScript & CSS based web.

Just thought of scratching my learning and some advise to other fellow developers. These are solely my views. You may agree or disagree. Sometimes will have entirely different opinion.

• Programming
• Design - every programmer should create/have minimum design before typing.
• Hardest thing in programming is naming things. Spend some time during design at least for public classes, methods etc...Private things we can change at any time to a meaningful name.
• Do no think when coding. Do not code when thinking.
• But these activities needs to happen in short cycles. eg: Think and design for 15mins and code for next 15 mins and commit your changes.
• We can never build a generic generic program. If it was possible world does't had these much programmers.
• Think about generic mechanism and patterns, if something is repeating 3 times.
• Design API by first writing client/consumer apps. Better test from 3 different clients/scenarios before finalizing.
• Problem solving
• If struck on a problem for 30 mins, take a break. It will clear our bias and help to think out of the box.
• If the above is not helping explain the situation to some one who may or may not understand software. 90% of time, during the explanation new idea will pop up in our brain. Brave people can try rubber duck debugging also.
• Defect fixing
• Fix code by eliminating code not by adding code.
• More code, more defects.
• When testing team reports a new failing combination, try to add that in unit tests when fixing.
• There are some defects which will never get reproduced in development. But had to fix by looking/reviewing the code.
• Always write down debugging steps, so that you can restart from where you stopped after attending call from wife or girl friend.
• Code quality
• There is no perfect code. After spending days to finalize one class, a new person seeing that first can suggest improvements. Need to balance to meet the release time.
• Do not refactor for the sake of refactoring. Refactor as part of new features or defect fix. But make sure those are different commits to source control.
• Achieving SOLID is difficult. At least make sure your functions are doing one thing and write pure functions.
• If possible get the review done via writing unit tests.
• During day job, often the priority goes to meeting deadlines and developer will be forced to write bad code. So develop personal projects or participate in open source projects to write perfect code where there is no deadline.
• It is not possible to meet a deadline without any hacks. The quality should be maintained by fixing it immediately after the release. In case, its single delivery, try to fix before completion.
• Coding practices
• Assembly language era is over. JavaScript has got many minification utilities. So let's program for fellow developers to understand. They are going to maintain the code in future
• There was a misunderstanding that writing unreadable, complex code will keep the job safe. But that is not true as the new generation developers are writing code way earlier than us and they are smart to take over. 
• Keep state in one place. Make 90% of classes / code stateless.
• Do not keep different scenarios, you thought during coding in your head. Document as test methods. Else after some months you cannot understand why you wrote.
• Communicate what code does via proper variables,functions, classes and component(dll) names. 
• No comments for explaining logic. This is because when next person changes the code there is no guarantee that he will update comments.
• API documentation is fine. Eg: xml comments in .Net
• Source control, build & deployment.
• Commit your changes within maximum of 30 mins. If you are not able to complete something after 30 mins, go back to thinking/planning and split the tasks appropriately. First step can be creating the source folder structure / test classes and committing those.
• Do not comment the code for using in future by uncommenting. Its there in source control. If your company is not using source control quit immediately.
• Architecture
• Tough task is to balance between delivery timeline and quality. Should never go to extremes.
• 90% of the time it is not built. Its evolved.
• Do not stop coding. If you do so your design will be just boxes to developers.
• Spend time to validate whether code is following arch. Sometimes change arch to the way code is going.
• Try to avoid third party libraries as much as possible because each library is adding one more dependency on your plan.
• Have an idea/capabilities about all the known technologies at high level. Dig deep only when required.
• Think how architecture is going to be coded, tested, deployed and maintained before finalizing. At any point a developer, tester, infrastructure or support guy may come and ask how to do something in detail.
• Learn buzz words / technical jargon to talk to different people.
• Communicate to different level of people in different levels of details. To a client/stakeholder a holistic approach, to business don't use technical words, to managers only about deliverables and developers only in terms of programs.
• Project management
• Follow the below principle in Agile
• Get it done - Demo
• Get it right - Put into QA
• Get it optimized - Production and maintenance.
• Do not plan more than one sprint.
• Create a new branch in source control start parallel development only when its really necessary.
• Setup a CI & CD at any cost. Manual build and installation is old story.
• If there is no test cases, do not push to production via CD.
• Try to get code freeze at least 1 day during the build & release days.
• Let the developers fix the hacks during the code freeze. But do not let them check in to main branch on which the build activities are going on.
• When every project is starting customer will have a fixed goal. Its us who should throw new ideas,improvements towards them and keep continuing the project.
• Make sure those ideas are business related and towards cost cutting for client to improve their profit. When client's profit increases our profit will also increase.
• As a team member
• Ask others to review your code and accept the good parts. Even ask juniors to review your code.
• Do not just say its not working to your technical lead without a proper description or screenshot of exception or some links saying its defect. They will think that you don't know, didn't do debugging and google.
• If you are taking a hack or shortcut, add the details with web links as comment in source code. So that future developers will not blame you. Do not keep it in mails, because the new members into your team will not get the mails.
• Always give suggestions to business analysts requirement documents, proposed user interfaces. Most of the time they may not be knows what are the capabilities of technology.
• Try to be truthful to testing team. When they come up with inconsistent help them to reproduce by understanding the code.
• Team management
• Always have one meeting in the morning and let the team explain what went yesterday and what is the plan for today.
• Try to reduce the time required for your team to send reports. Get the details from task/project/work item management systems such as TFS.
• Always look at the code your team is producing. To your superiors its your code.
• Keep quick access to release dates, number of features and defects total and pending. if you can remember some showstopper/high priority defect ids, you will be a star.
• Do not let project depend on one person. The show must go on at any cost.
• Give ownership of tracks/components to single person or teams for fixed time say 6 months. Rotate the members to different teams in every 6 months. But never replace everyone in track in single shot.
• Quality in software engineering
• If you are clicking everyday as part of testing app, try to bring automation or look for next company.
• Enterprise Application Development
• Most normal things won't work in enterprise. Many things done there doesn't have any common sense. But have to do to keep the project moving.
• It is very difficult to bring a change in enterprise.
• Self learning and carrier improvements
• Read others code.
• Debug for others. 
• There are many who can type code. But very less who can debug code.
• Try to look into third party libraries. If its unmanaged, it will be difficult but in all the other cases its possible to view the source code of third party libraries. Obfuscation makes it little difficult though.
• Learn one new language every year
• Read at least one technology book every year.
• Improve searching/googling skills. A best programmer needs to be a best googler.
• Never waste your brain memory for language syntax. Google can give it in seconds from internet. Instead free brain to keep the capabilities and think freely.
• Do not spend much time on learning short living frameworks such as Silverlight. Instead spend time on long living concepts such as http 2. 
• Try to create and publish reusable libraries. There are very good library management infrastructures now a days such as nuget, npm etc...At least contribute to libraries. If there is no time to fix defects report the defects in the libraries.
• Learn how things are working end to end. From business analysis, design, coding unit testing, functional testing, business testing, build systems, continuous deployment, application support and how to work with patches.
• If you want to become a good business analyst you should be knowing what are the developers problems and how developers interpret the requirements document. For that best way is to work as developers. Similarly if you want to be good developers, you should know how QA tests your app. Again better way is to know is to work as QA. QA may not catch all the defects but its caught in application support. So to do better testing you should be knowing how the users use the system and best place is to work in support. To me an ideal career path is Application support->QA(manual + automation)->Development->Business Analyst. Join as support and climb the ladder, If there is no chance to climb in current company change the company.
• When to leave company
• Evaluate your activities in your company at least once year other than performance appraisal. If you feel you got only salary in return, leave the company.
• If you feel you are the ultimate authority of something and everybody depends on you as you are the king, leave the company, because your growth is over there.

Leveraged Evernote in my personal site http://www.JoymonOnline.in

What we need in personal sites

If we look at any personal profile sites, we can see, its bunch of information about a person or group. Information like what they like, what they do? what is their recent advancements etc...

Problems with HTML / ASP.Net only sites

In the first glance we may think that it can be done via simple HTML pages. That is OK for the first release. Once its released there will be changes requests. It seems human nature that they always want to improve. If the change request is too low, it can be managed by manually editing HTML. But if its more manual editing is not a good option. 

Obviously the option is to build dynamic pages and serve. It brings non functional requirements as below

• A database and its maintenance.
• Need to get a good web based HTML editor. 
• Need to implement security
• May be a hidden login page as I don't want my viewers to see the login option in my personal site.

Or better use a CMS already available and customize it to suit to me.

Problems 

Maintaining database

Database creation and maintenance seems simple in the beginning. But once we do 10-15 releases, it will be nightmare. Mostly we will end up in so any schema modifications and data upgrades.

Making sure there are periodic backups is again a headache. Another thing is if we want to buy decent database hosting space its not cheap.

HTML Editor

There are good web based ie Javascript based HTML editors available in free. But still I need to spend some time selecting which is the best suited for me and integrate the same.

Security

ASP.Net has good security framework. Without coding much we will get a basic site with login page. But we need to always make sure its secure by applying patches. Sometimes we won't have access to the servers especially in shared hosting.

CMS and customization

I first used still using "Personal Web Site template" from Microsoft which is ASP.Net Web Forms. At that time I felt that is simple and will help me to launch site easier. 

Now No way I can easily upgrade to MVC. Slowly I am removing the framework features and making it simple web form application. There is nobody creating free themes for it now.

If I had chosen Joomla or other heavy CMS, I can get free templates now also.

If we consolidate all of the above what I want is

• A web site. 
• Design changes less frequently.
• Should be cheap to host.
• Sections are pretty static but content changes are frequent.
• There should be easy way to edit without worrying about HTML syntax

Solution - Outsourcing content for easy editing

The solution which I feel best suited is 

• Outsource content management.
• 
  
• Pull that content to our site and display it.
• Need to make sure there is API which gives content in HTML.
• Login to third party portal to change content

The one I found suitable is Evernote. I can keep my site content (text,list,table) in Evernote and they have API to pull the data.

Like any other solution there are pros and cons.

Pros

• Its free.
• No need to worry about database maintenance.
• No worries about security.(Need to remember the Evernote password :))
• This is trust/belief based. They can secure the contents than I can.
• Site contents can be simply edited from mobile on the fly.

Cons

• If Evernote stops free account, need to migrate all the contents to the web site.
• Need to properly abstract (class EvernoteContentProvider:IContentProvider {})
• There is monthly quota for free Evernote account. Need to cache the data in site.
• If the site has huge hits don't go this route.

Updates

2020-06-17 

After using some time JoymonOnline.in stopped using Evernote as Evernote made collaboration in premium paid tier. Currently it stores the data along with the source code in GitHub

API Versioning & its cost

Versioning everywhere in engineering / Manufacturing

Lets look at the below bridge analogy to understand versioning in other engineering disciplines.

There was a river and people lived in both the shores. They used small boats and rowed manually to reach other side. All were living happily.

One day some people came claiming that they are civil engineers and they told that they can make the  river crossing easy by introducing something called a bridge. People were convinced on the argument that they don't need to spend money on buying boat or boat tickets. They can use their legs which is proven to travel in the land. Some people told why we need new things as the boats are serving the needs. But eventually more people agreed, raised fund and engineers built a wooden bridge which is 1 feet wide and only allows travelling in one direction at a time. Yes they got Bridge V1.

Knot versioning

Engineers started seeing their unemployment and started saying cons about what they built recently. They complained that it supports traffic in one direction at a time. Also they complained that people cannot take their cattle through it. They advertised that it, if people can riase more money they can widen the bridge so that traffic can be bidirectional and it can support their animals such as cows. Some people who didn't support bridge earlier told why we need bigger bridge as we can use the existing by waiting for some extra time. But this time as well, more people thought of wider bridge and they raised fund to extend it. Then they got Bridge V1.1. This is the knot versioning model where every user uses same thing which is upgraded

There were small routine maintenance needed in the bridge and engineers were called to fix it. Each time they called it Bridge V1.(x+1)

Compatible versioning

After one year cars started running in the roads and the rich people bought cars first. They felt it difficult to cross the river as the bridge is not supporting cars. They came to engineers and asked them to have some way to take the cars to the other side. This time engineers asked more money and declared that the base of bridge, they had built is not strong enough for cars so they need new bridge. They also told that the same bridge they are building for cars can be used by people who don't have cars too. This time there was not much resistance from the people. They raised more money and built a 6 meter wide concret bridge. This time the fund raising was lead by local city authority and engineers were dealing with authority. After the new bridge is built, they declared that the old bridge will never be maintained. They never demolished the old bridge. Engineers named it Bridge V2.0.
This is compatible versioning where new version support old customers where older version is still there but discouraged to use and not maintained further.

Engineers got the maintenance work on the new bridge as well. After each maintenance they increased the last number in the bridge name.

Point to point versioning

Next year authorities decided to start train service between the shores and they contacted the engineers. This time they told that it needs to be new bridge because of below reasons

• Its difficult to upgrade the existing bridge which is already used for vehicles and people. At each and every steps of construction they need to test whether it can support cars and people as it is supporting right now.
• If the new bridge needs to handle single responsibility of trains they can build it efficiently.
• They also told that if they expand existing bridge for trains they may need to close it for 1 month which will be difficult for the people who are already using it.
• They prepared a cost comparison chart between building a new bridge as well as upgrading the existing one. It was clear that new bridge is cost saving.

Authority agreed on new bridge for trains. As usual they raised fund from people and gave to engineers to build the train bridge. Engineers built it using iron bars and directly placed the rails on top of bars.There were no flat surface like in other bridges. They called it Bridge V3.0. This is point to point versioning where each consumer stick with their own way instead of using latest.

Engineers got maintenance contract for new bridge as well.

The local authority was running out of funds they raised for bridges. So they declared wooded bridge is not required as most of the people have cars. They called same engineers to disassemble the old wooded bridge and auctioned the wood.

Versioning in Software engineering

In software engineering, we call the access points as API. Versioning concept is applicable here as well. It is very difficult to decide which versioning method is the way to go. Sometimes it will be cost effective to follow knot versioning where server and client applications can be upgraded together. But it cannot sustain for long and at some point we will be struck. Compatible versioning is little better in terms of cost but after long time, we will not be able to support backward compatibility.

Point to point doesn't held us from stopping new versions. But the cost is always proportional to the versions

The problem here is we don't have a concrete costing/estimation mechanism to compare the versioning strategies. Since we don't have facts it would be always difficult to convince the stake holders on the strategy.

But below is a good link, which explains mathematically how cost can be calculated for versioning API.

http://www.ebpml.org/blog2/index.php/2013/11/25/understanding-the-costs-of-versioning#

Happy designing...

IPC via Standard In and Out in C#.Net

When to communicate via console instead of service based IPC

IPC is an essential feature of large software systems. We cannot build large systems with just process. Technically we can but it will not scale to the larger audience. Another problem is stability. There are many ways to achieve inter process communication which is listed in wiki. Normally people prefer service based communication as it will scale easily to more machines.

If we are sure that 2 processes are 

• Always going to run on same machine.
• They may not be available always. Sometimes the destination process might be spawn by source. Sometimes source may need go down after initiating the destination after getting the 'take over'signal.

we can use simple mechanisms to establish the communication.

Command line arguments

If the communication is

• From source to destination only
• One time 
• Destination is spawn by source process 
• With small data

We can use command line argument passing mechanism from the source to destination.

Using Process.StandardInput.WriteLine

If the communication payload is big passing via the command line arguments is not the good option. We can use standard input and output streams instead. Below is the C# code snippets for source as well as destination. Even it can be 2 instances of same executable.

Source code for destination exe There are multiple ways to write to the standard in stream such as using the StreamWriter and all. If we use writer in source and ReadLine in destination there are chances that programs may wait on ReadLine statements unless we explicitly flush. So better use simple console write and read line methods.

Using unique/temp files and pass the file name via commandline args

If the communication payload is enormous or its binary and mainly one way from source to destination, we can use temp files to pass data. Things to remember

• Better generate a guid and use that as file name. 
• Close the file after writing at the source.
• Keep it in a temp location.
• Let the destination file delete the temp file once its read
• Still have cleanup routine to delete the temp files if anything remains
• Do not use this mechanism for 2 way communication with FileSystemWatcher or passing the file name via Standard in/out streams.

Why to subscribe to OutputDataReceived event

There are now 2 parties involved in communication and there are chances that 2 of those may wait for the input from others. So better use OutputDataReceived event to avoid such deadlocks.

References

http://stackoverflow.com/questions/21270842/reading-writing-to-a-command-line-program-in-c-sharp
http://stackoverflow.com/questions/285760/how-to-spawn-a-process-and-capture-its-stdout-in-net
http://stackoverflow.com/questions/26199441/redirect-standard-output-for-n-level-of-child-processes

Determining health of system in distributed queueing system to dequeue next message

Recently we are in to the business of building a distributed system where the long running operations to be offloaded to processing machines. Processing machines are pulling the messages from the queue instead of having a manager allocating tasks. 

Initially we were assigning throttling numbers to each of the processing machines according to their configuration such as number of processors, RAM etc...But soon we could see that the machines are either not utilized or over utilized. So decided to introduce a mechanism where the real time machine load needs to considered before taking new message from the queue.

We considered some techniques for sensing the load but those were not feeling better than analyzing the Windows performance counters. So decided to go with looking at the appropriate performance counters before dequeue a new message from the queue.

What are the appropriate performance counters? Its always debatable. We are in the initial stages of the implementation. Hopefully can update soon.

Another challenge we faced was how the performance counter data is translated to a boolean value saying system is healthy or not. We evaluated PAL which reads the .blg files and produces report. But finally reached to a conclusion of saving the performance counter values into database and running own rule engine there which replaces PAL.

Some links on how to work with performance counters from perfmon.exe below

https://technet.microsoft.com/en-us/library/cc722414.aspx
http://www.windowsnetworking.com/articles-tutorials/netgeneral/Scripted-Networt-Defense-Part2.html

Orchestrator pattern in .Net - Using TPL

This is continuation of my previous posts to find out the best mechanism to sequence or orchestrate operations in .Net. Running in parallel is added advantage in the library.

1. State pattern v/s Orchestrator
2. WWW - Passing arguements to custom activity inside a sequence container in coded workflow
3. Orchestration pattern in .Net - Using TPL

Using TPL to run tasks sequentially

Though the word TPL stands for Task Parallel Library, we can use it to execute tasks in sequence. One example is given below.

---

    class TPLTest
    {
        internal void Test()
        {
            Task.Factory.StartNew(new Action(Step1))
                .ContinueWith(new Action<Task>(task => Step2()))
                .ContinueWith(new Action<Task>(task=>Step3()));           
        }
        private void Step1()
        {
            Console.WriteLine("step 1 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 1 end");
        }
        private void Step2()
        {
            Console.WriteLine("step 2 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 2 end");
        }
        private void Step3()
        {
            Console.WriteLine("step 3 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 3 end");
        }
    }

---

Passing context object into steps

In normal business applications, when we deal with this type of sequential operation, there would be one or more contextual objects which needs to be modified as part of the execution. Either the result of one step needs to be passed as parameter to next step or one contextual state object will be shared by all the steps.

All steps sharing same state object

Its very simple to achieve using TPL. Just keep the state object in the class level so that all methods can access it.

---

    class TPLTest
    {
        //State kept as int for demo purpose only. In actual applications the state object will be more complicated
        int state = 0; 
        internal void Test()
        {
            Task mainTask = new Task(new Action(Step1));
            mainTask.ContinueWith(new Action<Task>(task => Step2()))
            .ContinueWith(new Action<Task>(task => Step3()));
            mainTask.Start();
        }
        private void Step1()
        {
            Console.WriteLine("step 1 start. state ="+state.ToString());
            Thread.Sleep(2000);
            state = 1;
            Console.WriteLine("step 1 end. state =" + state.ToString());
        }
        private void Step2()
        {
            Console.WriteLine("step 2 start. state =" + state.ToString());
            Thread.Sleep(2000);
            state = 2;
            Console.WriteLine("step 2 end. state =" + state.ToString());
        }
        private void Step3()
        {
            Console.WriteLine("step 3 start. state =" + state.ToString());
            Thread.Sleep(2000);
            state = 3;
            Console.WriteLine("step 3 end. state =" + state.ToString());
        }
    }

---

Executing in pipeline mode / Output of one step is fed as input to next step

Below is the code snippet which shows how the TPL can be used to execute tasks sequentially by feeding the output of one step as input to next step.

---

   class TPLTest
    {
        internal void TestPipeLine()
        {
            //int is used as input for demo purpose only. 
            //In actual application the input object will be more complicated class
            int input = 1;
            //There is no typed way to pass input to the pipeline.It accepts object and explicit conversion is required to get int value.
            Task<int> result = Task.Factory.StartNew<int>((arg) => Convert.ToInt32(arg),input)
                .ContinueWith(antTask => StepA(antTask.Result))
                .ContinueWith(antTask => StepB(antTask.Result))
                .ContinueWith(antTask => StepC(antTask.Result));
            Console.WriteLine("Result " + result.Result.ToString());
        }
        private int StepA(int arg)
        {
            return arg + 2;
        }
        private int StepB(int arg)
        {
            return arg * 2;
        }
        private int StepC(int arg)
        {
            return arg - 1;
        }
    }

---

We can see the result is printed as  5 (First step convert the object 1 to int 1 then + 2 then *2 then -1).

Honor SRP - Moving actions to separate classes

In the above examples we can see that the steps are staying in one class. This violates the SRP(Single Responsibility Principle). There are more than one reason for that class to change. So re-factoring little bit to honor SRP by moving the steps to separate classes.

---

    class TPLTest
    {
        int state = 0;
        internal void Test()
        {
            Task.Factory.StartNew(()=>new Step1().Execute())
                .ContinueWith((task)=> new Step2().Execute())
                .ContinueWith((task => new Step3().Execute()));
        }
    }
    class Step1
    {
        internal void Execute()
        {
            Console.WriteLine("step 1 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 1 end.");
        }
    }
    class Step2
    {
        internal void Execute()
        {
            Console.WriteLine("step 2 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 2 end.");
        }
    }
    class Step3
    {
        internal void Execute()
        {
            Console.WriteLine("step 3 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 3 end.");
        }
    }

---

Now the step classes will change, only if there is change in the step logic and the main class will change only if there is change in the step order.

Now we have seem how the TPL can be used to fulfill the requirements of orchestration. Below are the pros and cons of using TPL for sequential execution.

Pros

• Its inbuilt in .Net. We don't need to invent or refer other assembly.
• The Task objects can be reused in case we want to run in parallel.

Cons

• Difficult to organize each step in different class in type safe way as there is no base class for steps to inherit from.
• Out of the box it does't have the support to accept typed context object into the steps. ie we need to convert the object into the context / state object in first step.
• From the development team perspective, its little difficult to use if all needs to follow same orchestration method. We can see there is very little control in the framework. Look at the first 2 snippets. The Test() 7 TestInPipeline() method is implemented differently but works properly. So there are chances that, we can see 10 different implementations in same code base after 6 months if coded by 10 different developers.

Conclusion

This can be used if the development team is disciplined to follow same mechanism else we will end up in different methods to achieve orchestration. Also there is no object oriented support to organize or control the step class coding. I am continuing my research to find better library for orchestration.

Developing for developers - Designing and developing APIs

Couple of months backs I got a chance to be part of an initiative in our project, where we have to develop APIs for our server platform. The server platform is evolved with one of the project we are doing for one particular client. Now client want to extend that platform to be usable to their other projects. When we say its server platform, it basically manages servers, helps to provision new servers to scale horizontally, provide some platform data such as where some particular data is located. The data is expected to come in terabytes and it has some data storage mechanism which is multi-tenant. It really feels to me like we are reinventing the cloud and I tried to express here and there. Sometimes, I need to be quite, else who is going to pay me:-) Obviously there are reasons from other end also. There are many legacy code to be carried to this platform and client is having security concerns on hosting in the client. If we use Windows Azure, we might be ending up in limitation. Anyway its really interesting to be part of it especially when it is coming to API design and development.

My experience with API development starts with my first company which sells custom .Net libraries .There I was supporting .Net controls and fixing defects occasionally. At that time nobody told me about the importance of API development. But I realized myself, that API is something very important because we are developing for developers. If we didn't do it right, its very costly to change as all of our developers need to change their code as well. Think about the .Net library. Most of  the classes remains same for years. Now we cannot think about Microsoft renaming the MessageBox.Show() method to MessageBox.Alert(). I believe its same for other languages as well. I was really wondering in my early stages of carrier that, how they are coming up with these framework classes, its inheritance hierarchy, members and the access modifiers of each member. If something went to faulty, its very costly to recover.

In my second company, I got very little chance to work on critical APIs, but there were chances to code for their WPF custom control library. But in third and present company, there were no chance  still last 5 years. All I was doing is to help developers to fix their code which they written only to meet the release. Obviously there were some scope of drawing boxes as design. Most of the developers (including me sometimes because of pressure) were not programming, but gathering code to make a release.

When I got the new opportunity, I thought of doing some great things. I already targeted some  developers who are passionate in programming. But unfortunately, I got a completely different team who were gathering code for delivery as I mentioned above. After 2-3 weeks it came to notice that though they were great delivery guys, they are not much aware of SOLID principles or any other software engineering aspects of programming. It was really fun how I came to notice it. As the server platform components are already available in existing application,we started extracting those components. After extracting the code, we decided to re-factor it to look like a platform code.Class name ,Namespace changes etc... The problem started there.

As of my understanding, if somebody who already practiced SOLID principles, can re-factor any code by analyzing the technical aspects / code flow. But here, the team was struggling to re-factor and the reason they told was they don't know the business of application. The timers were really hard to understand for them. I was really wondering at this statement because I had re-factored big amount of code by understanding the logic by looking at the existing code. I had no clue what is the business requirement of those code or nobody gave me requirement documents. I think this is the case with all the service companies who are delivering from offshore. So arranged training on SOLID principles.

This leads to setting up one more training on API design. Or how to design APIs in a developer friendly and long lasting, backward compatible manner. For me too, it was really a learning experience. Got names for practices which we are already doing and of course obtained the experience of other API developers. Embedded the presentation below. Hope fully I can upload the video too.

References

http://wiki.qt.io/API_Design_Principles

Edits

9Oct2015 - Added references section.