Tuesday, April 26, 2016

Web App or Web Site ie Generating HTML at server v/s client

Most of the tech people know what is what and when to use what. But there are many developers out there in delivery companies who don't know what is the difference and they all call same thing web site sometimes and web app otherwise. This is just to consolidate my understanding about the difference between these 2

Web Site

This is collection of HTML web pages linked together. The pages may be residing as .html page in the web server or web server generates the pages based on the request. For generation of html pages dynamically server side technologies uses html templates which contains tokens. The tokens will get replaced based on the incoming request. Sometimes it reads database for the token value or calls another service for the same. The driving factor is the incoming request.

There were islands of more interactive components using embedding technologies such as Flash, Silverlight or native ActiveX components. JavaScript was used mainly to do simple nice to have things such as validation. The aim was even if the JavaScript is disabled in the browser the app should function properly.

These web sites were catering application needs as well. Eg: we can develop a Employee payroll application using this model.

Technologies
  • PHP
  • ASP.Net

Pros

  1. The site will function even if the JavaScript is disabled in browser
  2. Easy to develop with little js as js was behaving differently in different browsers

Cons

  1. Web Servers have to fetch data from database and mix it with html templates to produce final html which is directly viewable in browser. This overloads the web servers.

Web App

Here mode of accessing the application doesn't change from the end user side. They have to navigate to a provided URL. The content seen in browser is still html. But the difference is, the html template and data are downloaded separately from the server and the mixing happens in the browser. Sometimes if only needs data and instead of html templates it can use JavaScript programs to generate the viewing html.

Here JavaScript plays the key role. If the browser don't have JavaScript support, the application will not function.

Gmail Web interface is a classic example of a Web App.

Technologies

The same server technologies can be used. Just tell them we need only data. Preferably in JSON format. Don't mix with html template as we know how to display your data in browser.

There are technologies such as ASP.Net Web API which are more tailored to web apps.

At client side
  • JavaScript - This is essential
  • Client side frameworks such as AngularJS, knockout js, Bootstrap etc...

Pros

  • Web Servers act as just data provider. Their throughput increases. ie with same no of servers we will be able to serve more users
  • Easy to handle the browser specific formatting such as different view for mobile, tablet etc...Other words responsive UI.
  • Since the server only gives data, it can be consumed by desktop and native mobile apps.

Cons

  • Development team needs to have strong understanding of JavaScript. JavaScript is the tricky language as it contains unwanted things from the beginning. Also this is the language which has a book explaining good parts
    • There is a new trend of transpilation to get rid of this language problem. We can write in different language and then converted to JavaScript for execution.
  • Client side technologies are not stabilized yet. Everyday there are new frameworks getting introduced.

How to select?

We should always choose right technology to right solution. Most of today's apps demand Web App methodology. But there are scenarios where we can use WebSites. For example, simple sites created for personal / business profiles can follow web site methodology. This is because there are only 5-10 pages expected and those can be generated by web servers quickly.

Conclusion

WebSites and Web Apps differ mainly on where the html is generated. There is no change happened to browser during this shift except they increased the speed of JavaScript engine. HTML is HTML always. Most of today's client side machines are capable of generating html in their browsers without any performance lag. Choose the approach appropriately especially when choosing client side JavaScript framework.

References


Tuesday, April 19, 2016

Deserialize JSON string to C# dynamic type

Introduction

This is the age of APIs in computer science. To be more specific web APIs. Don't get confused with the specific technology of .Net Web API. Any methods or resources which are exposed via URL in a structure is an API. From our apps we can call them to do achieve the functionality.

Resources are now a days exposed via ReST protocol and the format is JSON. It works well with JavaScript. But when coming to .Net world, its little difficult to keep the JSON as is in our app. Often we need to convert those resources into entity classes. Creating entity class for each API response is a tricky tasks. It puts more time to complete the API integration tasks. Often after using one API for months we may need to switch to another API providing same service. At that time we need to rewrite it. Creating classes to get exact replica of response is really difficult to manage.

But its always suggested to have entity classes which are for our application regardless of the API we are using. As soon as we got the response from the web API, we should convert the response to our model and that should go on from that point on wards.

Dynamic object to hold JSON

In the beginning people were programming without type safety. Then the strong typed languages were introduced and everybody went behind that. Now its the time everybody again going to dynamic typing. May be its because of JavaScript's success.

VB always gave us the option to have dynamic typing. But in C# its introduced with .Net 4.0. The dynamic keyword is introduced to denote the variables which are not type safe. In other words a type which we don't know at compile time, what it is going to contain. Kind of key value store / dictionary.

Enough theory. Below is the code to convert a GitHub API response. The API url is 

private Project GetProjectFromJSON(string json)
{
    JavaScriptSerializer ser = new JavaScriptSerializer();
    dynamic projectNode = ser.Deserialize<dynamic>(json);
    return new Project()
    {
        Name = projectNode["name"],
        Description = projectNode["description"],
        SourceUrl = projectNode["html_url"],
        Url = projectNode["homepage"]
    };
}

JavaScriptSerializer is a .Net framework class. We don't need to add any dependency to third parties. There are other .Net libraries which are helping to convert JSON to dynamic object but those will add unwanted dependencies. Hope this is self explanatory.

Important thing to note here is the usage of dynamic. Do not ever return dynamic to outer world. This method should hide the complexities of getting the project from GitHub. The consumers of this method then don't need to change when the logic of GitHub API interaction changes.

More details about usage in real application can be seen in this file.

Tuesday, April 12, 2016

Caution JavaScript Ahead - atob() and btoa() to deal with base64 encoding

Setting the ground

JavaScript is always tricky. Because it has many garbage with it, its dynamic etc...This post aims towards how to work with base64 encoding in JavaScript.

History

In JavaScript if we want to work with base64 the standard methods are 
  • atob() - Used to decdode
  • btoa() - Used to encode to base64
It seems there are not much about this naming. Some people says it stands for Binary To ASCII when encoding and vice versa. Another thought is it was following C function naming. We can also think that when this was developed the aim was to write small JavaScript programs. If the programs are big it will take more time to transmit from the server to client. Language designers were concerned about that.

Usage

Simple

It's just a function which can be called normally.

var output=btoa("Joy");

Output
Sm95

var output=btoa("Sm95");

Output
Joy

Unicode

The above works well if the input is not unicode. If the input is unicode there will be an exception and the message goes as follows
In Chrome

InvalidCharacterError: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.

In IE (11)
InvalidCharacterError

Base64 with unicode

The fix to this problem goes as follows for encoding


var encoded = btoa(unescape(encodeURIComponent("ജോയ് മോൻ")));


Output

4LSc4LWL4LSv4LWNIOC0ruC1i+C1uw==


To decode use the below


decoded = decodeURIComponent(escape(atob("4LSc4LWL4LSv4LWNIOC0ruC1i+C1uw==")));

Output
ജോയ് മോൻ

Problem

The problem with this is same. It works differently in different browsers. If the encoding and decoding happens in same place, it works. But suppose if the server encodes the data and JavaScript gets it via Ajax,what will happen?

Ideally atob() should decode it and it does in Chrome and Firefox. But its not working in IE. Even in IE 11.

To be more precise the scenario is to decode the content coming from Github API. Below is a typical Github API url and it gets context in base64.

https://api.github.com/repos/joymon/joyful-visualstudio/readme

When we try to decode the base64 using atob() it will work well in Chrome and Firefox. But throw the the same error message in IE 11.

Solution - Use custom base64 library

The solution to this is to use a custom base64 library. Below is one url to such a library.


Refer this library in to the site and use below line to encode to base64

encoded = Base64.encode(inputText);

To decode

encoded = Base64.decode(encodedText);

Refer the source code of library to get more understanding of how its done.

Sample

Sample code is in jsFiddle. Url below
http://jsfiddle.net/joygeorgek/tv3b7feL/10/

Tuesday, April 5, 2016

What is Microservice Architecture to me?

Microservice is a buzzword in current Software engineering world. Everyone wanted to say we are building microservices or eagerly want to adopt it.

Reasons for micro services

Microservice - Bring Agile to Architecture

If we look at definition, we can see its an architectural style, where system is divided to small autonomous processes which are coupled with programming language/platform independent API. But we had similar kind of architecture in SOA (Service Oriented Architecture). They were also communicating kind of stable independent APIs. But what is the main difference or advantage.

Frequent releases

Yes the ability to release  more frequently is the key. We cannot deny other advantages that system is clean with small services which knows how to do its duty well, system can easily be built with different programming languages, scale only required components/services instead of whole system. etc...

If we remember the introduction of agile, there were so many advantages advertised. Working code over documentation, customer collaboration, responding to change were some to mention. But what is the real key? The key was 2-3 week based sprints which delivers something. As of my understanding most of the service companies adopted agile only because there will be a release in 2-3 weeks and developers will work night and weekend every 2nd week. In waterfall, the night and weekend parties were once in 6 months. Most of the places only the concept of sprint was taken but other concepts like maintaining backlogs, avoiding documentation over code were discarded. There would be no release happened in the world without documentation saying that code is working. 

What is there in Microservice? Independent teams who take ownership of small services. The size of the service, kind of relates to the size of the team who delivers it. And deliver daily, twice a day or if possible hourly.

Yes most of the places we can see an emphasis on TDD when adopting Microservice. But that is something might be discarded to rush to release.

What is the bottom line? The architecture will be agile. A Microservice could be introduced at any time in the development phase. Servers are going to be flooded with services. The complexity of the system is going to be in the orchestration of Microservices.

Design Microservice

Whatever we think, the Microservices architecture is a reality and soon most of our apps are going to be Microservice based. As the software engineering is still evolving, only fittest will survive. Currently it seems Microservice is a fittest one. Of course there are some problems to which Microservices approach is not the best solution.

Here lets see what are the characteristics of Microservices? 

Independent release

If a service can be released independently of the other parts of the system, it can be considered as Microservice. Sometimes the new API methods we released in the Microservice might not have any consumer components as the consumers can release only after the service release. 

The concept of independence is really important, if we want to release too often. If it depend on other components we cannot release fast & often.

Independent data store

If we want to have independent release of a service, it's data store should be independent. If there is a database table and a component release upgrade the schema causing other components to fail, we cannot consider that component as a Microservice. In short Microservice should be the only one to own it's data store. If other components in the system want to write something to the store they should call service.

Independent development team

If we want independence, probably it should be done by independent team. This is something not with the technology. Organization structure needs refactoring and is tough. 

The speed of delivery is prone to errors as well and it is advised to have development team supporting the service. If the team is shuffled after each release or members change often, they cannot support service as their own baby and the micro service architecture might feel a failure.

Independent framework/platform/language

This is not a mandatory thing. If there is a pressing need for adopting a particular technology which is different from regular technology stack, Microservice is recommended. But only if the above 3 criteria are met. Even we can have Microservice based systems with one language and technology.

Adopt only if

Do not jump just into Microservice without proper assessment and planning. If everything is independent who coordinates all these to work as a complete system which cater business need? Below are the things to make sure before jumping into Microservice world.

Defined integration guidelines between components. http(s)

If you don't have defined integration strategy of Microservices, it will simply fail. The recommended integration if through http(s) protocol if the interaction is synchronous. If the integration if asynchronous, adopt message queuing, service bus etc... But make sure there are enough design sessions and agreements before adoption.

Upfront design on the API

The real world systems / business problems are always complex. In monolithic system, we were handling the complexity in big components. But in Microservice era, the individual components are simple, but the complexity is shifted towards the interaction of services.

This requires carefully designed APIs for each Microservice. API is always like Mumbai underworld. Once we enter, there is no coming back. Once we publish an API, we don't know who are all calling it. Correcting the API and communicating the same to all consumers to redirect to new API is always costly.

Deployment is easy and automated.

We have seen one of the major advantage of Microservice is the frequent releases. Is the installation going to be done manually every time? If so we will immediately feel that the Microservice is a bad idea.

Deployment should always be automated. A continuous integration and delivery pipeline is always essential.

Strong automation testing. Prefer 100% test coverage

There is no human required for day to day deployment. They just setup the CI & CD in the beggining and system will do deployment automatically. What about testing? Are we expecting a manual tester to test all the changes before we release daily?

That's again should be automated. Developers should write proper test cases for the API. Most of the companies are employing a separate team / person for testing, is on the assumption that a developer doesn't want to break his own code or psychologically they cannot find out errors in the system what they build . When developer test, he tests only positive or usual scenarios because he knows how the system works. But here in Microservice world. we are not entertaining separate manual testing team,So how to tackle it?

Its very easy to tackle. Some other developer needs to write the tests. If the existing testing team knows how to code API tests, integration tests, they can also do. Please note we are against manual click QA. Automation QA is always welcome.

Service monitoring & management

We are talking about explosion of services. There are chances that each class/component in your existing system such as logging, caching will become a service. Some services will be versioned services, some will be cumulative/in place update. We need to add more service instances and load balance when services show load. Some might not be even used. A machine which hosted a service might go down and we need to relocate the service without zero or a little downtime. How do we deal with all these scenarios? 

The answer is a strong service monitoring and management infrastructure. Easy way is to host in cloud, where this is free with cloud provider. We don't need to develop infra management application for increasing number of machine and deploying the service to it. Or to move service from one machine to another machine etc...

Versioning, dependency management, retirement strategy

V2 of Service A may require V4 of Service B to be present in the system where V1 of Service C require V3 of Service B. This means V3 of Service B cannot be retired until Service C is retired.

Why we need to retire a service. Over the time some services will be legacy and there will be nobody calling it. But it just stays in the production environment. It is the decision of the stakeholders to retire or not. If we plan to retire everybody should be notified that, it is retired so that no new service call these retire ones.

Another interesting scenario is whether the a service say Service X should know the location of other service Y? Though the Location has many things to consider, here let's take as the url of service. If we decide to keep the location of Y in X, we cannot move the Service Y to other machine or environment. Hard coded location will cause trouble. Suppose we started with all the services on premises and due to load, if we decide to move Service Y to cloud Service X, will simply fail in calling Y.

This requires a service registry which keeps track of all the services and their dependency. This can be another service but in fixed location of course.

If you have access to production

If the development is for a product your company owns and is server based, Micro services works great. If the product is delivered as an installer to your clients, it would be really difficult to manage, as they may be still installing the morning release when you give evening release.

So basically better choose, if the CI & CD pipeline can reach till production. We can refactor the other systems to Microservice based but will never get the advantage of frequent release.

What to do with UI

Till now, we were talking mainly about services, what about UI? Are we going to deliver UI twice a day? That is something interesting and will discuss in a future post.

Disclaimer - I work in a project which have 100+ services, though we officially not adopted the Microservices architecture. It just came like that and many services are sharing same data store. We are not doing independent release of services due to data coupling. So the above view is not from a real time successful Microservie based project rather from reading and sufferings of service management.