Tuesday, February 9, 2016

Automated integration testing in ASP.Net MVC & Forms without IIS

Background

I have wrote about testing ASP.Net web forms in one of my previous post. This the era of MVC and I am still writing about web forms. Mainly I am working with web forms only for my web site http://joymononline.in. I have started it before MVC days and still I don't see any reason for converting to MVC. Its serving its purpose using the web forms technology? Why should I change? Another reason I am keep it there in web forms just to realize that technology should not drive our architectural decisions. Every time before I suggest MVC for any project, I think about my own web site and think twice whether we should go to MVC or not? None in my company agree or do in old web forms but its still worth considering and making sure there is specific reason for selecting ASP.Net MVC. According to them we should never do anything in web forms.

Recently I tried to implement Continuous Integration and Continuous Delivery to my web site. Details of doing it via AppVeyor can be found in another post. It was very easy to run unit tests in the integration server.

Problem

But the one of the issue I found during that exercise is on running Integration tests. What I wanted was to make sure all the pages in my site is browsable after each check and and deploy to staging. I was using [HostType("ASP.NET")] attribute to run the tests against IIS. This is mentioned in my earlier post about testing web forms. This works well in development machine as I have created IIS web application from Visual Studio and the project was pointing to that web application(Project Properties->Web->Server->Local IIS in VS 2013). But in build machine, this hosting will not happen automatically. We need to write script to host the application to IIS which is according to me extra task.

Possible approaches

This problem is not new. Many people have solved it in many ways. There are free solutions such as using tools like Selenium or can use hosting libraries such as WatiN, MVCInterationTestFramework etc... Some more options can be seen in the below link.
http://stackoverflow.com/questions/118531/what-is-the-best-way-to-test-webforms-apps-asp-net

My Solution

The requirement is clear. We need to host the site in a web server to run integration tests. So why can't we start a web server before the test begins and the test request site page. This lead to MVCIntergationTestFramework library. This .Net library helps us to host any folder as IIS web application. It also has some features to test results from MVC response.

No need to get confused with MVC prefix. As mentioned earlier, we can host any folder as IIS web application. So web forms also works. The main git project has many clones and one of those provide nuget as well named FakeHost. So its easy to include into VS test project.

Continuous integration

Since it doesn't need upfront IIS web application, it is easy to run tests from CI server provided it has IIS in it. AppVeyor has IIS in it and it runs smoothly there.

Once again this is my solution towards a small problem of integration testing of my personal web site. It may not be applicable always. If anybody wants to take a look at production code how its done, have a look at my personal web site code in Github. Now I am really enjoying the free CI & CD life. I don't know how to write down the joyful feeling of getting our source compiled after each check in, automatically testing the same and publish to staging and production if everything goes well. Everything free.

Happy CI & CD

Tuesday, February 2, 2016

What is BrowserLink and how to get rid of that?

We were on a Javascript debugging session which involved Web Workers, FileAPI and jQuery in an http(s) enabled page. There were so many things in the F12 console window which increased the difficulty in debugging the same. When the Web Workers seems not accepting the messages, we started putting so many console.logs to understand where it breaks actually. Soon we notice that there is some .Net exception stack coming in the browser console as follows

[19:45:14 GMT-0400 (Eastern Daylight Time)] Browser Link: Exception thrown when trying to invoke Browser Link extension callback "madskristensen.editorextensions.browserlink.unusedcss.unusedcssextensionfactory.GetIgnoreList":
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at MadsKristensen.EditorExtensions.BrowserLink.UnusedCss.UnusedCssExtension.GetIgnoreList()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.VisualStudio.Web.BrowserLink.ClientMessaging.DispatchMessage(BrowserLinkConnection connection, MessageArgs obj)

The debugging happened in a dev machine. At first we thought there is some VS extension failing. But why should it come to browser? Sometimes when we debug js inside browser it enters strange file other than jQuery which took to the height of frustration. The debugging was kind of over after some time. But this wired call stack struck in my mind.

Immediately after the debugging, I googled for what is this browser link. I could see that its a feature from Microsoft from VS 2013 on wards, enabled by default for dealing with more browsers. More specifically refreshing multiple browsers associated with Visual Studio. If this is enabled Visual Studio establish links between browsers an can refresh them. It uses modules to inject javascript etc...

Wow...I immediately disabled it by following the below link.

http://blogs.msdn.com/b/webdev/archive/2013/06/28/browser-link-feature-in-visual-studio-preview-2013.aspx

Tuesday, January 26, 2016

AppVeyor a free cloud CI - First experience

Background

Though open source doesn't mean free, most of the people working in open source area wants to get things free which ever is required for their projects. Github as free source control repositoty, Google hangout for meetings etc...AppVeyor is one of the tool which should be added to this list especially its .Net project. If we google we can get lot of information about AppVeyor. This post is going to be discussed about my experience with AppVeyor. I am really late into this open source world.

When we consider starting of any software project, there are many things other than writing code. Most of those things needs minimum research before starting. Else it will take more time to setup later. As seen earlier there should be some place to host the code. We cannot have keep the code in each and every developer's machine and pass it in zip files. Github now became one stop destination for hosting code free. So at least that piece doesn't require any more research. Another thing is Continuous Integration.

Continuous Integration

The need for CI

If we have experience more than 10 years, probably we might not have heard this word in college or during the initial years inside software industry. At that time we were considering a release as something special like Kumbamela which occurs in predefined intervals. Intervals were in years and in some places months. Nobody saw even in dream that they can give something bi-weekly to customer. But things changed with the adoption of a magic word called Agile. People started to give releases in 2 weeks to customers or at least forced to do so. That automatically bring the demand for continuous integration especially in automated ways.

In the earlier days when the release is approaching one person/team will be allocated for creating builds. All he needs to do is to compile the project and create MSI. Install it. Run basic tests and give to QA for detailed testing. Some teams opted for rotational policy for this build person / team. 

Software engineers are expected to solve others problems especially repeatable and involves human power. Who solves their problems? Of course themselves. In order to save a build person/team's pain the Continuous Integration is born.

What is continuous integration

As the name says integrate the works of developers continuously instead of waiting till the release date. Integration should mean getting the code changes of all developers, compiling the same and generating binaries/installers. Nice to have features are running unit tests, deploying to test machines and running integration tests. Advanced practitioners can even push to production if all the tests are passed. Everything is expected to be automatic. Better integration should start based on a souce code check-in.

This ensure that the defects are caught early instead of waiting till the release date.

CI Platforms

We saw that it needs to happen automatically on check-in. Where all these operations can run? They need dedicated one or more machines because developers cannot just sit idle during this time by running it in their machines.

In the initial days of CI people used separate systems for doing integration. Obviously when we automate integration we need software to do the process. Some people wrote their own programs to do all these operations and clever people created products out of it idea and made money. There were open source CI systems also emerged.

Even if buy or get OSS, it required a machine to run CI. Open source people at least who are working as individuals and small teams were struggling in this area. A machine is not free. Even if we keep old system for CI, we need to invest for internet, power etc...Enterprises also suffered a little as they need to maintain the CI machines.

Advances in the cloud technology solved this problem. Cloud allowed anybody to have a machine in cloud for short span of time. The SaaS model ie Software as a Service made the things more simple. They started giving CI as a Service. In other words, anybody can out soure the CI activities to cloud based systems without worrying about setting up a machine in cloud. All we had to do is to connect our source code with CI as a Service system. They have machines in cloud in which they download our source compile it, package it, run unit tests and even deploy it to production.

What is there for open source. There started some hosted CI providers who give free CI to open source projects along with their paid service. Wow...Anybody can do CI free in cloud. There are many major players. Lists can be found in many places in internet. One of those is AppVeyor.

AppVeyor experience

I started my research when I continuously saw a "Build Passing" flag in many open source projects in Github. Most of those lead me to travis-ci. When I started looking for a CI for my open source projects, I had no confusion to go to travis-ci. But when I read their .Net support, I got little confused. They have Linux servers only which points to mono. My first project which needs CI is not just an app. Its visual studio extension which directly depend on VS SDK. I started research again for CI which has native .Net support. The below links gave me confidence to try with AppVeyor.

Nuget restore

When I started AppVeyor, first stop was on nuget restoration. I know there is nuget restore command. But I was not sure where to enter the command. First google lead me to AppVeyor page where they explains how to deal with nuget packages

It didn't help me in first shot. It was telling that solution was not found

sln not finding

The exact error goes as below

MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
This folder contains no solution files, nor packages.config files.
Command exited with code 1

Nowhere it tells its due to nuget except 'nor packages.config'. Googled and immediately found the below link which explains if the solution is in inner folder in github src, we need to explicitly mention the .sln file path in nuget restore command as parameter.
http://help.appveyor.com/discussions/problems/1446-yml-file-isnt-finding-my-solution-file

VS Extension issue due to AWS

AppVeyor's console output for build is really awesome. It shows real time status about build. It showed below error message.

Setting up Visual Studio for debugging extensions. This one-time operation may take a minute or more.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe /RootSuffix Exp /ResetSettings General.vssettings /Embedding /Command File.Exit
log4net : error XmlHierarchyConfigurator: Could not create Appender [RollingFileAppender] of type [log4net.Appender.RollingFileAppender]. Reported error follows. [C:\projects\joyful-visualstudio\src\VS2013\JoyfulTools\VSExtension\VSExtension.csproj]
System.IO.FileNotFoundException: Could not load file or assembly 'AWSToolkit.Util' or one of its dependencies. The system cannot find the file specified.
File name: 'AWSToolkit.Util'
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type)
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName)
at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.SystemInfo.GetTypeFromString(Assembly relativeAssembly, String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.SystemInfo.GetTypeFromString(String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.TypeConverters.TypeConverter.ConvertFrom(Object source)
at log4net.Util.OptionConverter.ConvertStringTo(Type target, String txt)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ConvertStringTo(Type type, String value)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.CreateObjectFromXml(XmlElement element, Type defaultTargetType, Type typeConstraint)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.CreateObjectFromXml(XmlElement element, Type defaultTargetType, Type typeConstraint)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)

I am not at all using AWSToolkit or log4net. This is the time the impression about AppVeyor went down. My initial google didn't give me proper answers so I created a support ticket in their forum

http://help.appveyor.com/discussions/problems/3348-vsix-project-build-fail-on-starting-integration-test

The response to the question came in amazing speed which I never experienced in any other support systems. He really knows the system well he clearly told the issue is related with an extension which is in the Visual Studio installed in their CI machines. Disabling that will solve the issue. He gave PowerShell command to disable the same.

I was not much clear on where to apply this script. This issues was already faced by others as well and they tried the same workaround. So I looked there and tried the normal command to disable the AWSToolkit.
http://help.appveyor.com/discussions/questions/1275-build-failed-log4net-error-xmlhierarchyconfigurator-could-not-create-appender-rollingfileappender

Yes it rocks. Now there is integration happening for each and every check in and I can get the artifact in their site.
https://ci.appveyor.com/project/joymon/joyful-visualstudio/build/1.0.0.27/artifacts

Note : I really don't have any relation with AppVeyor except using their free service. I cannot just stop me from writing about their free service and excellent support for free customers. May be in future they change their policy. But still then they are great.

Tuesday, January 19, 2016

What if there are no looping constructs in your language?

Being a professional software engineer, we cannot say that we are done we cannot program without looping control flow. Developers who started their career in machine or assembly language don't get scared of this. Similarly who understood and practiced machine and assembly language in college in. They will simply use 'go to' along with if statement and get the things done.

Forgot to tell in the question 'go to' is banned as always.So how can we tackle it without goto.

Lets come to code. Here we are going to see the scenario in 2 famous languages C# and Javascript. Below is the code to display numbers 1,2,3...10 in descending order which is written using loops.

JavascriptC#
function LoopWithWhileKeyword(loopCount) {
    while (loopCount > 0) {
        logLine(loopCount);
        loopCount -= 1;
    }
}
void LoopWithWhileKeyword(int loopCount)
{
    while (loopCount > 0)
    {
        logLine(loopCount);
        loopCount -= 1;
    }
}

This is straight forward. The code to call these functions is omitted to get focus, Now lets think how we can avoid this loop.

What is happening here

If we step back and think we can see there are mainly 2 code blocks. 
  • One block determines the continuation of execution.
  • Second is the actual code to execute multiple times.
Some idea might have triggered in everybody's brain who had done some programming. Nothing but extracting those separate blocks to functions.

We can use 'if' statement to decide whether to continue execution or not based on the return value if we move the condition logic to function. Then how do we execute the repeatable code block again without go to?

Execute the same function again. All of us has tried this in college when dealing with Fibonacci series. If we call a function from inside same function we can simulate loops. Yes recursion!!!

Lets see the code below which is rewritten with recursion.
JavascriptC#
function LoopWithoutWhileKeyword(loopCount) {
    WhileLoop(function () {
        return loopCount > 0
    },

    function () {
        logLine(loopCount);
        loopCount -= 1;
    });
}

function WhileLoop(condition, action) {
    if (condition()) {
        action();
        WhileLoop(condition, action);
    }
}
void LoopWithoutWhileKeyword(int loopCount)
{
    WhileLoop(
        () => loopCount > 0,
        () =>
    {
        logLine(loopCount);
        loopCount -= 1;
    });
}
void WhileLoop(Func<bool> condition,
Action action)
{
    if (condition())
    {
        action();
        WhileLoop(condition, action);
    }
}

Good. Isn't it? For those who gets headache on seeing recursion please read the explanation.

Here the WhileLoop function accepts 2 functions. It checks whether I should continue or not. Similar to the evaluation inside the while loop condition check. If the result of condition check is true, it calls the action method which has the code  needs to be executed multiple times.

Then it calls the WhileLoop function again to repeat the cycle. Since the action function has the code to reduce the loop count, it will end when the loopCount becomes 0.

Higher Order Function

We learned a jargon in programming called Higher Order Function. Any function which accepts another function or returns a function can be called as higher order function. Thinking that its this simple refer wiki

Most of us might be thinking, why we learned the loops if functions can do the job of loops. Why there are many ways to do same thing in computer science?

Tail call optimization

Normally developers has a tendency to use new things learned in their day job because of excitement. They might have heard it or saw one demo which touched the good parts of the concept / feature / technology.

Before anybody apply this technique in your production app understand and test it properly. Let's test this code by given 100,000 to the function.

In C# your will encounter 
An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

In browser, you will get 
Uncaught RangeError: Maximum call stack size exceeded

In my machine I got javascript error when the recursion reaches 17911 and c# 4.0 failed after 12202 iterations.

Don't think that this issue will occur always. There are technologies and languages in which this issue will never occur. If we we are getting this, that tells us that our run-time or compiler does not do tail call optimization. Google for what is tail call optimization.

Tuesday, January 12, 2016

Joyful Tools for Visual Studio - Remove commented code

Added one more feature. Those who downloaded last week might have got this feature. The blog posts are scheduled so announcement comes very late.

Started using AppVeyor as CI tool. They are cloud hosted and give free CI for open source projects in Github. Support also good. More details of the CI experience will be posted separately.

Tuesday, January 5, 2016

Joyful Tools for Visual Studio

Background

There won't be any confusion on Visual Studio as the best software development tool available today. Earlier it was mainly to Microsoft technologies. But now its opened to other technologies as well. Being the best tool in the world, it cannot incorporate all the features. For example if we want to remove all the multiple blank lines there is no direct option in VS except find and replace using RegEx.
When we develop any application made for generic purpose hardware which is computer, it cannot incorporate all the needs. One thing is that the developers cannot foresee all the needs of users. Some needs may be specific to specific user/project groups.

Visual Studio Extensions

This problem is always solved by application by giving provision to extend the application via plugins. Visual Studio is not different it has option to incorporate extensions / plugins. We can inject our own menu items, tool windows and all into Visual Studio.

For adding a direct method to remove multiple blank lines also we can extend Visual Studio. I created a plugin some years back and hosted the code in codeplex. It was very much useful to me who is working on a project which is running for last 8-10 years. So many blank spaces just to increase the file size. Some files has 5000+ lines. When a new developer comes to the team, they are just scared of touching the code. Mainly because of the size of files. So I thought of at least removing the continuous blank lines so that the scariness can be reduced somewhat.

Moving to Git and rebranding

The earlier name of the extension project was removeblanklines which limits its area to one problem. I am planning to add more features to it such as removing commented code except xml comments. So a rebranding felt right to me. So renamed it to "Joyful Tools for Visual Studio".

Extension can be downloaded from Visual Studio Gallery or directly installed from Visual Studio

https://visualstudiogallery.msdn.microsoft.com/c9256992-8b91-4448-b31c-fa1cd6a27455

Source code

I don't think there is any need to tell why moving from codeplex to Github.
https://github.com/joymon/joyful-visualstudio

Tuesday, December 29, 2015

Published my first Google App Engine Python Application


This would have been a post if everything went easily. But it was not easy. Got one trouble. Ok let me start the story.

Introduction

Google App Engine (here after GAE) was there for long time at least from 2011 its no more preview. Unfortunately I didn't get a chance to play with it. Now since my family went to India during the winter season, I have enough time to play around.

I never thought it would be easy as publishing web site from Visual Studio to Azure in single click. But app engine is simple as one click publish.

First I ensured that it still has free quota. Because being in the Microsoft shop in my day job who embraces Azure a lot, there is no way I will be hosting an application into Google cloud. So its for my personal use and for personal use I never wanted to pay money. Oh except my domain and hosting space.
https://cloud.google.com/appengine/docs/quotas

First Python application targeting Google App Engine

Then I started with the below basics on Python based GAE development and how to publish a simple Python web app into GAE.
https://cloud.google.com/appengine/docs/python/
https://cloud.google.com/appengine/docs/python/tools/devserver
https://cloud.google.com/appengine/docs/python/gettingstartedpython27/introduction

Then setup the environment. I already had Python 2.7. So installed only the GAE SDK.
https://cloud.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python

Its very simple. Once we install the SDK. It will provide us an option to run our web site locally. Once we are satisfied we can push to app engine ie cloud

The helloworld sample in google help is very preliminary. So googled for little higher startup sample and I got the below link.

http://www.labnol.org/internet/host-website-on-google-app-engine/18801/

In the above link I was able to see some more better sample which deal with html files as well. I took that did some modification in html and tried to run from local app engine server. It worked fine.

Deployment

Then I tried to deploy to real GAE by clicking on the Deploy button in the GAE launcher. It opened the browser and I entered my google credentials. But unfortunately it didn't publish to my app engine. The window stayed as is with the below message

If your browser is on a different machine then exit and re-run this
application with the command-line parameter

  --noauth_local_webserver

Then I started googling and found that it happens sometimes and if we use the command based deployment it will work. I tried that and didn't succeed.

Then googled with error and language Python. Then I got another workaround to a known issue.

If the local development server is running we will not be able to deploy to GAE. I just stopped the local server and deployed. It worked.

The public url is as follows
http://joystestproject.appspot.com/