Tuesday, October 27, 2015

Manipulate HTML string using jQuery


We have a HTML string which may or may not have tables. We need to make sure the width of all tables and table columns are empty.

We are in a web page which already have reference to jQuery. We don't have access to manipulate the original web page. The html string is present in one known TextArea in the loaded page. 

Our Javascript code is going to run from browser address bar.


So many constraints. But this was one of my requirement to modify width of tables present in my Evernote web page. Evernote don't have support to edit the table width once its created.

The technique is simple as follows
  • Get / parse the string into jQuery object
  • Use find() to select the objects
  • Once the objects are selected apply the changes.
See the jsFiddle below

Tuesday, October 20, 2015

Hosting Evernote .Net SDK in Godaddy or other hosting provider

There is no doubt that Evernote is a simple but great product. It was simple desktop application in the begining and moved to cloud at the right time. Cloud storage helps our notes to be synced across technology, platform and devices.

I was a proud user of Google notes. But Google shut it down. From that moment, I moved to Evernote cloud.

Evernote SDK for .Net - Using local file system

Evernote has exposed APIs to interact with out notes and there is SDK for programming models which help us to use the API easily. But if we try to host our application which the Evernote SDK in Godaddy just to read the notes, we will  get errors as Godaddy is not providing file system access. It will happen in all the shared hosting providers where our code don't have access to full file system. First it will ask to give full trust. After giving full trust we will get more specific exception. Details below

Exception:System.UnauthorizedAccessException Message:Access to the path '\Evernote\EvernoteSDK' is denied. Exception:System.UnauthorizedAccessException 

Message:Access to the path '\Evernote\EvernoteSDK' is denied. 

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) 
at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost) 
at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost) 
at System.IO.Directory.CreateDirectory(String path) 
at EvernoteSDK.Advanced.ENPreferencesStore.Save() 
at EvernoteSDK.Advanced.ENPreferencesStore.SetObject(Object objectToStore, String key) 
at EvernoteSDK.ENSession.PerformPostAuthentication() at EvernoteSDK.ENSession.Startup()
at EvernoteSDK.ENSession..ctor() 
at EvernoteSDK.ENSession.<.cctor>b__10() 
at System.Lazy`1.CreateValue() 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Lazy`1.get_Value() 
at EvernoteSDK.ENSession.get_SharedSession()

We can clearly see that its trying to access local file system.

If the hosting is by renting a virtual machine this issue will not happen as we can give our code full access to the file system.

Root cause

If we look at the Evernote SDK source in Github, we can see it is caching the data/preferences in local file system. Currently there is no way to control that behavior. There is not even a way to provide our own provider.


Fix is simple. Need to get the Evernote SDK for .Net source code and refactor to have preference provider interface. Move current code into default file system based preference provider so that existing code will not break. For our special case of web hosting into Godaddy kind of servers, write new preference provider which is based on memory. 

Yes the performance may get affected due to app pool recycling and rebuilding of preference cache, but it will work in the web.

The fix is available in the below repository. A pull request is created, Evernote is yet to merge into their code base.

Tuesday, October 13, 2015

API Versioning & its cost

Versioning everywhere in engineering / Manufacturing

Lets look at the below bridge analogy to understand versioning in other engineering disciplines.

There was a river and people lived in both the shores. They used small boats and rowed manually to reach other side. All were living happily.

One day some people came claiming that they are civil engineers and they told that they can make the  river crossing easy by introducing something called a bridge. People were convinced on the argument that they don't need to spend money on buying boat or boat tickets. They can use their legs which is proven to travel in the land. Some people told why we need new things as the boats are serving the needs. But eventually more people agreed, raised fund and engineers built a wooden bridge which is 1 feet wide and only allows travelling in one direction at a time. Yes they got Bridge V1.

Knot versioning

Engineers started seeing their unemployment and started saying cons about what they built recently. They complained that it supports traffic in one direction at a time. Also they complained that people cannot take their cattle through it. They advertised that it, if people can riase more money they can widen the bridge so that traffic can be bidirectional and it can support their animals such as cows. Some people who didn't support bridge earlier told why we need bigger bridge as we can use the existing by waiting for some extra time. But this time as well, more people thought of wider bridge and they raised fund to extend it. Then they got Bridge V1.1. This is the knot versioning model where every user uses same thing which is upgraded

There were small routine maintenance needed in the bridge and engineers were called to fix it. Each time they called it Bridge V1.(x+1)

Compatible versioning

After one year cars started running in the roads and the rich people bought cars first. They felt it difficult to cross the river as the bridge is not supporting cars. They came to engineers and asked them to have some way to take the cars to the other side. This time engineers asked more money and declared that the base of bridge, they had built is not strong enough for cars so they need new bridge. They also told that the same bridge they are building for cars can be used by people who don't have cars too. This time there was not much resistance from the people. They raised more money and built a 6 meter wide concret bridge. This time the fund raising was lead by local city authority and engineers were dealing with authority. After the new bridge is built, they declared that the old bridge will never be maintained. They never demolished the old bridge. Engineers named it Bridge V2.0.
This is compatible versioning where new version support old customers where older version is still there but discouraged to use and not maintained further.

Engineers got the maintenance work on the new bridge as well. After each maintenance they increased the last number in the bridge name.

Point to point versioning

Next year authorities decided to start train service between the shores and they contacted the engineers. This time they told that it needs to be new bridge because of below reasons
  • Its difficult to upgrade the existing bridge which is already used for vehicles and people. At each and every steps of construction they need to test whether it can support cars and people as it is supporting right now.
  • If the new bridge needs to handle single responsibility of trains they can build it efficiently.
  • They also told that if they expand existing bridge for trains they may need to close it for 1 month which will be difficult for the people who are already using it.
  • They prepared a cost comparison chart between building a new bridge as well as upgrading the existing one. It was clear that new bridge is cost saving.
Authority agreed on new bridge for trains. As usual they raised fund from people and gave to engineers to build the train bridge. Engineers built it using iron bars and directly placed the rails on top of bars.There were no flat surface like in other bridges. They called it Bridge V3.0. This is point to point versioning where each consumer stick with their own way instead of using latest.

Engineers got maintenance contract for new bridge as well.

The local authority was running out of funds they raised for bridges. So they declared wooded bridge is not required as most of the people have cars. They called same engineers to disassemble the old wooded bridge and auctioned the wood.

Versioning in Software engineering

In software engineering, we call the access points as API. Versioning concept is applicable here as well. It is very difficult to decide which versioning method is the way to go. Sometimes it will be cost effective to follow knot versioning where server and client applications can be upgraded together. But it cannot sustain for long and at some point we will be struck. Compatible versioning is little better in terms of cost but after long time, we will not be able to support backward compatibility.

Point to point doesn't held us from stopping new versions. But the cost is always proportional to the versions

The problem here is we don't have a concrete costing/estimation mechanism to compare the versioning strategies. Since we don't have facts it would be always difficult to convince the stake holders on the strategy.

But below is a good link, which explains mathematically how cost can be calculated for versioning API.


Happy designing...

Tuesday, October 6, 2015

Azure WebJobs - Internals and how tos

This is my attempt to index the features of Azure WebJobs in single place. Mainly for my reference.

What is Azure WebJobs

It is a feature of Windows Azure cloud more specifically WebApps to execute long running operations agnostic to platform / language. It can run executable files continuously, on demand or using a schedule. We can give .bat, .ps1, .exes and more as executable files to the Azure WebJob which it will execute as per the configuration. It uses an engine called kudu to achieve the functionality.

WebJobs does't know how to execute a job/function on events such as when a message arrives in Azure storage queue or a blob is uploaded. We need to write our own listening logic to these events in the executable script and execute corresponding actions on event.

If we are writing .Net exe as WebJob and want to listen to queue, blob etc..., the WebJobs SDK will help us to write it better. This is nothing but a collection of assemblies (.dll files) which our .Net project can refer.

What is Azure WebJobs SDK

.Net based framework which we can use to write triggerable functions. The SDK calls our functions on the trigger. It does many useful actions before and after our function execution. Some example of such actions are

  • Binds the values of queue or blob to the variables which our function has.
  • It mark the message as completed if our function didn't throw any exception. (Delete from the queue)
  • If the execution fails for 5(configurable) times it moves the message to poison queue.
The important thing to note here is This SDK is not coupled with Azure as it is just a listening mechanism and we can extend it to listen to our own custom events and invoke the handler functions.


How the .exe or script deployed in Azure machines

The WebJobs is a feature of Azure WebApps and it owns the job executable. Executable(s) are stored inside \\app_data\jobs\ folder separated by the type of the job and folder name as job name.

Using Azure WebJobs SDK outside Azure

Running outside of Azure and still listen to Azure


Listening to our custom events and invoking functions

There is support for invoking the functions manually using JobHost.Call() method. But there is no option for adding our own polling/listening logic. The polling needs to be done out side of SDK in our code and we can just call the function

Ideally if we are writing our own polling logic which is not polling the Azure properties, we don't need WebJobs SDK. But if we are migrating from another queue system to Azure queues, as part of step by step migration, we can first migrate the functions to SDK format and listen to old queue technology. Then in next phase we can start listen to Azure queue.

2 types of trigger

There is triggering mechanism on Azure WebJobs. Also there is different triggering mechanism on invoking a function written using WebJobs SDK.

Triggering Azure WebJobs

Even if we forget about WebJobs SDK this is still applicable as WebJobs is a feature of Azure WebApp. This feature is not tied with WebJobs SDK. It uses kudo engine to trigger Azure WebJobs. We can write Jobs in technologies other than .Net. I am really thankful to MSFT for opening up their eyes to see the whole world.

There are mainly 3 different type of invocation.
  • Continuous - Azure will make sure one instance of our WebJob exe or script is always running
  • OnDemand - Azure will run our exe whenever a request is received. Once its started it can run for long. Azure will kill it if its idle for configured time.
  • Run on Schdule - Azure will trigger execution of WebJob (.exe or supported script file) on the schedule. It uses Azure Scheduler to invoke the WebJob.

If we are writing .Net console exe application and use for WebJob task, the WebJob SDK comes into picture. We can even write our own logic in exe to monitor the queues and start corresponding function to process.

Triggering functions inside .Net exe written using WebJob SDK

If we use SDK, SDK provides options to just write functions which will be called by SDK on different triggers. SDK can listen to changes in Azure Storage Queue,Blobs etc... and based on the function registration, it will call our functions.

If the exe is not running, there is no way our function will get executed regardless of what ever SDK we use.

Azure WebJob SDK Magics

The WebJobs SDK does so many magic to make the life of developer easier so that they can concentrate on their business than the hosting and plumbing works.

Function based

Even if we don't know about class concept in OOP we can write web jobs as webjob SDK simply needs a function which will be executed on certain conditions.

Attribute based

We need to tell the SDK about us to do the magic and that is done via attributes. The queue, blob names to listen the object which is stored in the queue data all needs to be specified in attributes.


The SDK serialize the queue data for us so that we simply gets the object via parameter when our function gets called.

Token replacement

We can use tokens inside the attributes and those will be automatically replaced by SDK. If we want to have a new blob for logging based on messageId property in our custom message object just use log\{messageId}. Then SDK will try to find if there is any messageId property in the message object and replace the token with actual value.

Storage Management

We just need to tell the blob name to the SDK via attributes. It will make sure the blob is there for us. 


It is really expecting 2 config entries. AzureWebJobsDashboard & AzureWebJobsStorage. These are used by SDK to connect to our Azure account.

Things to remember

Multiple function executions in WebJobs SDK

There are chances that WebJob SDK may call our functions multiple times for same queue item as part of retry and handling failures. So we need to make sure that even if the function execute twice it should not produce any damage.
Recommended approach would be to keep our own state for the queued item, update it when execution completed.Have a check in function beginning to make sure we are not redoing the operation.

Do not use static variables

When we register our functions with Azure WebJobs SDK, it invokes our functions in different threads. But its in same AppDomain. So its same as how ASP.Net handles the requests using different threads. If we use static variables there are chances that different threads will manipulate same value and we will end up in deep trouble.

Do not use .Net lock

As everybody knows the lock keyword works in process. It doesn't have any relevance in other process or other machine. In Azure it is easy to scale to multiple instances ie different machines. 

So short cuts such as lock, taking while migrating legacy code into WebJobs might not work well when scaled. The issues may pop up inconsistently so better avoid static variables and lock 

Don't write intermediate state to local files

This is the basics of any scalable background processing mechanism as there are chances that after failing the process may be picked by another machine. Then it will not resume from where it failed last time.

How tos - WebJobs SDK