Tuesday, January 26, 2016

AppVeyor a free cloud CI - First experience

Background

Though open source doesn't mean free, most of the people working in open source area wants to get things free which ever is required for their projects. Github as free source control repositoty, Google hangout for meetings etc...AppVeyor is one of the tool which should be added to this list especially its .Net project. If we google we can get lot of information about AppVeyor. This post is going to be discussed about my experience with AppVeyor. I am really late into this open source world.

When we consider starting of any software project, there are many things other than writing code. Most of those things needs minimum research before starting. Else it will take more time to setup later. As seen earlier there should be some place to host the code. We cannot have keep the code in each and every developer's machine and pass it in zip files. Github now became one stop destination for hosting code free. So at least that piece doesn't require any more research. Another thing is Continuous Integration.

Continuous Integration

The need for CI

If we have experience more than 10 years, probably we might not have heard this word in college or during the initial years inside software industry. At that time we were considering a release as something special like Kumbamela which occurs in predefined intervals. Intervals were in years and in some places months. Nobody saw even in dream that they can give something bi-weekly to customer. But things changed with the adoption of a magic word called Agile. People started to give releases in 2 weeks to customers or at least forced to do so. That automatically bring the demand for continuous integration especially in automated ways.

In the earlier days when the release is approaching one person/team will be allocated for creating builds. All he needs to do is to compile the project and create MSI. Install it. Run basic tests and give to QA for detailed testing. Some teams opted for rotational policy for this build person / team. 

Software engineers are expected to solve others problems especially repeatable and involves human power. Who solves their problems? Of course themselves. In order to save a build person/team's pain the Continuous Integration is born.

What is continuous integration

As the name says integrate the works of developers continuously instead of waiting till the release date. Integration should mean getting the code changes of all developers, compiling the same and generating binaries/installers. Nice to have features are running unit tests, deploying to test machines and running integration tests. Advanced practitioners can even push to production if all the tests are passed. Everything is expected to be automatic. Better integration should start based on a souce code check-in.

This ensure that the defects are caught early instead of waiting till the release date.

CI Platforms

We saw that it needs to happen automatically on check-in. Where all these operations can run? They need dedicated one or more machines because developers cannot just sit idle during this time by running it in their machines.

In the initial days of CI people used separate systems for doing integration. Obviously when we automate integration we need software to do the process. Some people wrote their own programs to do all these operations and clever people created products out of it idea and made money. There were open source CI systems also emerged.

Even if buy or get OSS, it required a machine to run CI. Open source people at least who are working as individuals and small teams were struggling in this area. A machine is not free. Even if we keep old system for CI, we need to invest for internet, power etc...Enterprises also suffered a little as they need to maintain the CI machines.

Advances in the cloud technology solved this problem. Cloud allowed anybody to have a machine in cloud for short span of time. The SaaS model ie Software as a Service made the things more simple. They started giving CI as a Service. In other words, anybody can out soure the CI activities to cloud based systems without worrying about setting up a machine in cloud. All we had to do is to connect our source code with CI as a Service system. They have machines in cloud in which they download our source compile it, package it, run unit tests and even deploy it to production.

What is there for open source. There started some hosted CI providers who give free CI to open source projects along with their paid service. Wow...Anybody can do CI free in cloud. There are many major players. Lists can be found in many places in internet. One of those is AppVeyor.

AppVeyor experience

I started my research when I continuously saw a "Build Passing" flag in many open source projects in Github. Most of those lead me to travis-ci. When I started looking for a CI for my open source projects, I had no confusion to go to travis-ci. But when I read their .Net support, I got little confused. They have Linux servers only which points to mono. My first project which needs CI is not just an app. Its visual studio extension which directly depend on VS SDK. I started research again for CI which has native .Net support. The below links gave me confidence to try with AppVeyor.

Nuget restore

When I started AppVeyor, first stop was on nuget restoration. I know there is nuget restore command. But I was not sure where to enter the command. First google lead me to AppVeyor page where they explains how to deal with nuget packages

It didn't help me in first shot. It was telling that solution was not found

sln not finding

The exact error goes as below

MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
This folder contains no solution files, nor packages.config files.
Command exited with code 1

Nowhere it tells its due to nuget except 'nor packages.config'. Googled and immediately found the below link which explains if the solution is in inner folder in github src, we need to explicitly mention the .sln file path in nuget restore command as parameter.
http://help.appveyor.com/discussions/problems/1446-yml-file-isnt-finding-my-solution-file

VS Extension issue due to AWS

AppVeyor's console output for build is really awesome. It shows real time status about build. It showed below error message.

Setting up Visual Studio for debugging extensions. This one-time operation may take a minute or more.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe /RootSuffix Exp /ResetSettings General.vssettings /Embedding /Command File.Exit
log4net : error XmlHierarchyConfigurator: Could not create Appender [RollingFileAppender] of type [log4net.Appender.RollingFileAppender]. Reported error follows. [C:\projects\joyful-visualstudio\src\VS2013\JoyfulTools\VSExtension\VSExtension.csproj]
System.IO.FileNotFoundException: Could not load file or assembly 'AWSToolkit.Util' or one of its dependencies. The system cannot find the file specified.
File name: 'AWSToolkit.Util'
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type)
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName)
at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.SystemInfo.GetTypeFromString(Assembly relativeAssembly, String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.SystemInfo.GetTypeFromString(String typeName, Boolean throwOnError, Boolean ignoreCase)
at log4net.Util.TypeConverters.TypeConverter.ConvertFrom(Object source)
at log4net.Util.OptionConverter.ConvertStringTo(Type target, String txt)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ConvertStringTo(Type type, String value)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.CreateObjectFromXml(XmlElement element, Type defaultTargetType, Type typeConstraint)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.CreateObjectFromXml(XmlElement element, Type defaultTargetType, Type typeConstraint)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.SetParameter(XmlElement element, Object target)
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)

I am not at all using AWSToolkit or log4net. This is the time the impression about AppVeyor went down. My initial google didn't give me proper answers so I created a support ticket in their forum

http://help.appveyor.com/discussions/problems/3348-vsix-project-build-fail-on-starting-integration-test

The response to the question came in amazing speed which I never experienced in any other support systems. He really knows the system well he clearly told the issue is related with an extension which is in the Visual Studio installed in their CI machines. Disabling that will solve the issue. He gave PowerShell command to disable the same.

I was not much clear on where to apply this script. This issues was already faced by others as well and they tried the same workaround. So I looked there and tried the normal command to disable the AWSToolkit.
http://help.appveyor.com/discussions/questions/1275-build-failed-log4net-error-xmlhierarchyconfigurator-could-not-create-appender-rollingfileappender

Yes it rocks. Now there is integration happening for each and every check in and I can get the artifact in their site.
https://ci.appveyor.com/project/joymon/joyful-visualstudio/build/1.0.0.27/artifacts

Note : I really don't have any relation with AppVeyor except using their free service. I cannot just stop me from writing about their free service and excellent support for free customers. May be in future they change their policy. But still then they are great.

Tuesday, January 19, 2016

What if there are no looping constructs in your language?

Being a professional software engineer, we cannot say that we are done we cannot program without looping control flow. Developers who started their career in machine or assembly language don't get scared of this. Similarly who understood and practiced machine and assembly language in college in. They will simply use 'go to' along with if statement and get the things done.

Forgot to tell in the question 'go to' is banned as always.So how can we tackle it without goto.

Lets come to code. Here we are going to see the scenario in 2 famous languages C# and Javascript. Below is the code to display numbers 1,2,3...10 in descending order which is written using loops.

JavascriptC#
function LoopWithWhileKeyword(loopCount) {
    while (loopCount > 0) {
        logLine(loopCount);
        loopCount -= 1;
    }
}
void LoopWithWhileKeyword(int loopCount)
{
    while (loopCount > 0)
    {
        logLine(loopCount);
        loopCount -= 1;
    }
}

This is straight forward. The code to call these functions is omitted to get focus, Now lets think how we can avoid this loop.

What is happening here

If we step back and think we can see there are mainly 2 code blocks. 
  • One block determines the continuation of execution.
  • Second is the actual code to execute multiple times.
Some idea might have triggered in everybody's brain who had done some programming. Nothing but extracting those separate blocks to functions.

We can use 'if' statement to decide whether to continue execution or not based on the return value if we move the condition logic to function. Then how do we execute the repeatable code block again without go to?

Execute the same function again. All of us has tried this in college when dealing with Fibonacci series. If we call a function from inside same function we can simulate loops. Yes recursion!!!

Lets see the code below which is rewritten with recursion.
JavascriptC#
function LoopWithoutWhileKeyword(loopCount) {
    WhileLoop(function () {
        return loopCount > 0
    },

    function () {
        logLine(loopCount);
        loopCount -= 1;
    });
}

function WhileLoop(condition, action) {
    if (condition()) {
        action();
        WhileLoop(condition, action);
    }
}
void LoopWithoutWhileKeyword(int loopCount)
{
    WhileLoop(
        () => loopCount > 0,
        () =>
    {
        logLine(loopCount);
        loopCount -= 1;
    });
}
void WhileLoop(Func<bool> condition,
Action action)
{
    if (condition())
    {
        action();
        WhileLoop(condition, action);
    }
}

Good. Isn't it? For those who gets headache on seeing recursion please read the explanation.

Here the WhileLoop function accepts 2 functions. It checks whether I should continue or not. Similar to the evaluation inside the while loop condition check. If the result of condition check is true, it calls the action method which has the code  needs to be executed multiple times.

Then it calls the WhileLoop function again to repeat the cycle. Since the action function has the code to reduce the loop count, it will end when the loopCount becomes 0.

Higher Order Function

We learned a jargon in programming called Higher Order Function. Any function which accepts another function or returns a function can be called as higher order function. Thinking that its this simple refer wiki

Most of us might be thinking, why we learned the loops if functions can do the job of loops. Why there are many ways to do same thing in computer science?

Tail call optimization

Normally developers has a tendency to use new things learned in their day job because of excitement. They might have heard it or saw one demo which touched the good parts of the concept / feature / technology.

Before anybody apply this technique in your production app understand and test it properly. Let's test this code by given 100,000 to the function.

In C# your will encounter 
An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

In browser, you will get 
Uncaught RangeError: Maximum call stack size exceeded

In my machine I got javascript error when the recursion reaches 17911 and c# 4.0 failed after 12202 iterations.

Don't think that this issue will occur always. There are technologies and languages in which this issue will never occur. If we we are getting this, that tells us that our run-time or compiler does not do tail call optimization. Google for what is tail call optimization.

Tuesday, January 12, 2016

Joyful Tools for Visual Studio - Remove commented code

Added one more feature. Those who downloaded last week might have got this feature. The blog posts are scheduled so announcement comes very late.

Started using AppVeyor as CI tool. They are cloud hosted and give free CI for open source projects in Github. Support also good. More details of the CI experience will be posted separately.

Tuesday, January 5, 2016

Joyful Tools for Visual Studio

Background

There won't be any confusion on Visual Studio as the best software development tool available today. Earlier it was mainly to Microsoft technologies. But now its opened to other technologies as well. Being the best tool in the world, it cannot incorporate all the features. For example if we want to remove all the multiple blank lines there is no direct option in VS except find and replace using RegEx.
When we develop any application made for generic purpose hardware which is computer, it cannot incorporate all the needs. One thing is that the developers cannot foresee all the needs of users. Some needs may be specific to specific user/project groups.

Visual Studio Extensions

This problem is always solved by application by giving provision to extend the application via plugins. Visual Studio is not different it has option to incorporate extensions / plugins. We can inject our own menu items, tool windows and all into Visual Studio.

For adding a direct method to remove multiple blank lines also we can extend Visual Studio. I created a plugin some years back and hosted the code in codeplex. It was very much useful to me who is working on a project which is running for last 8-10 years. So many blank spaces just to increase the file size. Some files has 5000+ lines. When a new developer comes to the team, they are just scared of touching the code. Mainly because of the size of files. So I thought of at least removing the continuous blank lines so that the scariness can be reduced somewhat.

Moving to Git and rebranding

The earlier name of the extension project was removeblanklines which limits its area to one problem. I am planning to add more features to it such as removing commented code except xml comments. So a rebranding felt right to me. So renamed it to "Joyful Tools for Visual Studio".

Extension can be downloaded from Visual Studio Gallery or directly installed from Visual Studio

https://visualstudiogallery.msdn.microsoft.com/c9256992-8b91-4448-b31c-fa1cd6a27455

Source code

I don't think there is any need to tell why moving from codeplex to Github.
https://github.com/joymon/joyful-visualstudio