Tuesday, June 30, 2015

C# async and await programming model from scratch

Introduction

This is a brief introduction to async and await keywords to a normal developer who wants to understand the basics and little insights to internals of how stuff works.

Background

Asynchronous programming is now an essential thing when we develop any application because it avoids waiting in main thread on long running operations such as disk I/O, network operations database access etc...In normal case, if our program needs something to be done from the results of these long operations, our code is struck until the operation is done and we proceed from that point. 

Using async mechanism, we can just trigger long running operations and can do other tasks. Those long running operations does the job in different thread and when they complete it, they notify our main code and our code can do the post actions from here. When we refer our code, its our main thread which deals with user interface or the thread which primarily process a web request. Sometimes we ourselves write these kind of long running operations.

What is async and await

In simple sense these are 2 new keywords introduced in .Net 4.5 to easily write asynchronous programs. They work in the method level. Of course we cannot make classes work in parallel as they are not unit of execution. 

Are these keywords known to CLR, the .Net run-time or a wrapper over TPL Task Parallel Library ? If they are wrappers, it it good to have language depends on a library written using same language?

We will find out the answer to these questions in this article.

History of .Net async programming

Threads were there from the very beginning of the .Net framework. They were the wrappers on operating system threads and little difficult to work with. Then more concepts such as background worker, async delegate and Task Parallel Library came to ease the async programming model. Those came as part of class library. C# language as such doesn't had 'out of the box' support for  async programming until the async and await keywords are introduced with C# 4.0. Lets see how the async and await helps us in async programming by examining each of these methods.

Example

Lets take the below example of finding factorial of first N numbers if they are completely divisible by 3. We are using console application for simplicity. If we had used a windows application we could easily hook into async event delegate handler and demo the async features in easily. But that won't help us to learn the language features.

Synchronous code


We can see there is a loop runs from 1 to 5 using counter variable. It find whether the current counter value is completely divisible by 3. If so it writes the factorial. The writing function calculates the factorial by calling FindFactorialWithSimulatedDelay() method. This method here in sample is going to put delay to simulate real life workload. In other sense this is the long running operation.

Easily we can see that the execution is happening in sequence. The WriteFactorial() call in loop waits until the factorial is calculated. Why should we wait here? Why can't we move to next number as there is no dependency between numbers? We can. But what about the Console.WriteLine statement in WriteFactorial(). It should wait until the factorial is found. It means we can asynchronously call FindFactorialWithSimulatedDelay() provided there is a call back to the WriteFactorial(). When the async invocation happens the loop can advance counter to next number and call the WriteFactorial().

Threading is one way we can achieve it. Since the threading is difficult and needs more knowledge than a common developer, we are using async delegates mechanism. Below is the rewrite of WriteFactorial() method using async delegate.

Making it async using async delegates

One of the easier method used earlier was to use Asynchronous Delegate Invocation. It uses the Begin/End method call mechanism. Here the run-time uses a thread from thread pool to execute the code and we can have call backs once its completed. Below code explains it well which uses Func delegate.

No change in finding factorial. We simply added new function called WriteFactorialAsyncUsingDelegate() and modified the Main to call this method from the loop.

As soon as the BeginInvoke on findFact delegate is called the main thread goes back to the counter loop, then it increment the counter and continue looping. When the factorial is available the anonymous call back will hit and it will be written into console.

We don't have direct option to cancel the task. Also if we want to wait for one or more methods its little difficult.

Also we can see that the piece of code is not wrapped as object and we need to battle with the IAsyncResult object to get the result back. TPL solves that problem too, It looks more object oriented. Lets have a look.

Improving async programming using TPL

TPL is introduced in .Net 4.0. We can wrap the asynchronous code in a Task object and execute it. We can wait on one or many tasks to be completed. Can cancel task easily etc...There are more to it. Below is a rewrite of our Factorial writing code with TPL.

Here we can see that first task is run then its continuing with next task which is the completed handler which receives notification of first task and writing the result to console.

Still this is not a language feature. We need to refer the TPL libraries to get the support. Main problem here is the effort to write the completed event handler. Lets see how this can be rewritten using async and await keywords.

The language feature async and await

We are going to see how the TPL sample can be rewritten using async and await keywords. We decorated the WriteFactorialAsyncUsingAwait method using async keyword to denote this function is going to do operations in async manner and it may contain await keywords. Without async we cannot await.

Then we are awaiting on the factorial finding function. The moment the await is encountered during the execution, thread goes to the calling method and resume the execution from there. Here in our case the counter loop and takes next number. The awaited code is executed using TPL as its task. As normal it takes a thread from the pool and execute it. Once the execution is completed the statements below the await will be executed.
Here also we are not going to change anything in the FindFactorialWithSimulatedDelay(). 

This avoids the needs for extra call back handlers and developers can write the code in a sequential manner.

What is the relation with Task Parallel Library and async await keywords

The keywords async and await make use of TPL internally. More clearly we can say async and await are syntactic sugar in C# language. Still not clear? In other sense the .Net runtime doesn't know about async and await keywords.

Look at the above disassembled code of  WriteFactorialAsyncUsingAwait(). I used reflector to disassemble the assembly.

Should a language depend on a library/class created with it?

This is a old question. If we look at C or C++, the language was always independent and the libraries were fully depend on it. But if we look from introduction of yield keyword in C#, we can see there is a marriage between language features (keywords) and libraries. Here yield which is a language feature depends on IEnumerable interface created using the language itself. Then the compiler does the magic and replace the yield keyword with corresponding IEnumerable implementation making sure CLR doesn't know about yield keyword.

Another example is using keyword. Its tightly coupled with IDisposable interface. Since then there are many syntactic sugars added more details can be found in below link.

http://blogs.msdn.com/b/ericlippert/archive/2010/10/28/asynchrony-in-c-5-part-one.aspx

Personally I don't prefer mixing language features with libraries. Let language evolve its own and libraries depend on language. If we do the other way the compiler is forced to inject more code and we know adding more lines is not coming in free. But unfortunately we are in a world where coding needs to be fast not the execution of the code.

Should the compiler modify our code?

The main problem with compiler modifying our code is debugging. There are chances that we will see call stack of our application which contains symbols which are not written by us. Try to see the call stack by raising an exception in anonymous method. If there are many anonymous methods in the application, that's it. We are done in debugging.

Should the language know parallel programming and threading?

This is another area to discuss. Since the threading is managed by OS, should the language care about threading. Should the threading be a library or integrated language feature?

Now a days most of the hardware has multiple cores and if the language doesn't provide the integrated features, nobody will leverage multiple cores. This is because either development community is afraid of threading or it requires additional coding time. If the language gives easy way, developers can focus more on the functionality or business side of the app than threading which is infrastructure related.

So I really want my language and associated runtime know parallel and async programming. But please try to avoid tight coupling with class library and stop compiler altering my code.

When should we use it?

We can use async and await anytime we are waiting for something. ie whenever we are dealing with async scenarios. Examples are file IO, network operations, database operations etc...This will help us to make our UI responsive.

So go ahead and make sure your APIs are await-able.

References

https://msdn.microsoft.com/en-us/library/hh191443.aspx
http://stephenhaunts.com/2014/10/10/simple-async-await-example-for-asynchronous-programming/
https://richnewman.wordpress.com/2012/12/03/tutorial-asynchronous-programming-async-and-await-for-beginners/

Tuesday, June 23, 2015

Running Roslyn Analyzers from console application

This post require prior knowledge from my earlier post. In last post we saw how our C# program can compile other C# program via Roslyn compiler. Yes its without Visual Studio. If we are compiling without Visual Studio how a custom Roslyn DiagnosticAnalyzer can be invoked to do analysis on the code fragment which is getting compiled.

Here we are going to see how we can connect with Roslyn custom analyzers when our program is compiling another program using .Net compiler platform.
First we are defining our custom analyzer which is nothing but a check to make sure the type names are starting with capital letter. See the code below. Now lets move on how this analyzer can be integrated to our programmatic compilation using Roslyn.
  1. GetSimpleCompilation()
    This is same as what is there in previous post about simple compilation.
  2. GetAnalyzerAwareCompilation
  3. Here we connect our compilation unit with custom Roslyn diagnostic analyzer.
  4. GetAllDiagnosticsAsync()
  5. This is Roslyn API to execute our custom analyzers on top of source code input and get the results back. The important point here is the compiler platform returns same type ImmutableArray<Diagnostic> every time. ie Regardless of whether its simple compilation or compilation with analyzers.

    If the result from GetAllDiagnosticsAsync is empty list, proceed to actual compilation which produces the assembly file.
  6. ShowDiagnostics()
    This is too same as previous post.

Tuesday, June 16, 2015

Using Roslyn to compile C# source code

Below are the steps write a program to compile C# code snippet using Roslyn (Now .Net Compiler Platform)to produce an assembly. Simply saying a C# program compiling another C# program using Roslyn compiler.

The Roslyn nuget version used here is 1.0.0-rc2. This tutorial is organized in zoom in/out fashion.

There are mainly 3 high level steps and they are getting a compilation unit, compiling the same and processing result / diagnostics it produced. The code snippet shows the steps.

  1. GetSimpleCompilation() Now lets look at what is happening in GetSimpleCompilation. As seen in the code we are getting the source code, creating syntax tree out of that then prepare the references needed for the compilation. Finally Creating CSharpCompilation using all the information.
    1. GetSourceCode()
    2. This method simply returns sample source code to compile.Can be replaced by any other code snippet.
    3. GetSyntaxTree()
    4. This method parses the input code and creates the CSharpSyntaxTree.
    5. GetReferences()
    6. Compilation always needs what are the references needed. Here its very basic as we are not using any advanced features.
  2. Compile()
  3. Now its the time to compile. It needs to get the assembly file name to proceed. Since the GetOutpuFilePath() is just simply returning the dll file path, omitting it.
  4. ShowDiagnostics()
  5. This method shows any warnings in compilation.
The compiled assembly can be kept in memory and execute from there. Please refer this link for more details.

Happy coding...

Tuesday, June 9, 2015

WebAPI SelfHosting via OWIN v/s HttpSelfHostServer

WebAPI can be hosted via 2 ways. The traditional way using HttpSelfHostServer and the new OWIN based hosted. Below is an attempt to compare both techniques.

API Controller

Below is the controller class I am going to host using both the hosting techniques.

Differences between HttpSelfHostServer and OWIN Hosting

In HttpSelfHostServer we can see most of the things as configurable. But in OWIN, its by convention.

References

https://frendsrnd.wordpress.com/2014/02/03/httpselfhostserver-hosted-web-api-with-https-and-windows-authentication-enabled/

Tuesday, June 2, 2015

Invoking a method without parentheses () in C#

The biggest difficult faced by any programmer coming to VB from C#, will the lack of parenthesis '()' when they want to call functions. In VB its not required to put ( ) to call functions. But we can put if we want. If we need to deal with legacy codebase which is developed by classic VB developers, we cannot expect any brackets in the functions calls.

It really reduces the readability of program. Each time when somebody see a word, they need to spend some time understanding whether its a function or property. Also to make sure what are the exact parameters, it will take some time. If we assume it will take 30 seconds for a developer to understand the intention of function call and its parameters without brackets, it will take considerable amount of time if we think about project with thousands of lines and 30-50 developers.

We (at least I am) are thinking that C# doesn't has this problem. But is that true? See the below code snippet.
Here the char.IsLower method is being invoked 3 times. But there is no parentheses used.

Tuesday, May 26, 2015

Moq - How to know the number of times a method is called on mocked object

Moq seems a good mocking library for .Net unit tests. It has most of the features a developer needs and pretty much stable and many people are using it. Once I started using it, I never felt a need to another mocking library. Below is one if the scenario which I had to face related to Moq.

Scenario

There is a method which randomly executes operations (IOperation implemented classes) stored in an array. The operations are unit tested independently to make sure there is no problems with them. This method which is in test simply uses Random class to get random number and get an operation from array using that number and invokes the operation.

Ideally speaking we don't need to test the Random class which is part of .Net framework. But why should we avoid unit testing our logic inside the method ? Some tests like 'making sure random value is not above the length of array' etc... So whats the big deal?

Challenge

The unit tests works by Arrange,Act,Assert methodology. First we need to arrange the things, Then act or do the operation by calling the method under test. Finally assert.

When we do the act, we should be knowing what is the output. Then only we can write assert. If we don't know what is going to happen we don't know what to assert.

An Example

Recently I forked ChaosMonkey for .Net and started contributing to it. I made it configurable. Looking into that code might be useful to understand the example better.It contains a MonkeyKeeper and IMonkey interface. MonkeyKeeper is the external interface. When MonkeyKeeper.Unleash() is called it should randomly pick an IMonkey from its monkeys collection and call Unleash() method on it. IMonkey implementations can be easily unit tested to make sure they are doing their duties on Unleash. But how to unit test MonkeyKeeper.Unleash()?

In this scenario, we can assert that the mocked IMonkey objects at least got one call to unleash. But in order to get at least one random calls, we need to call the MonkeyKeeper.Unleash multiple times because Random can return same value multiple times.

Below is one implementation, where MonkeyKeeper.Unleash() is called 5 times which is holding 3 IMonkeys collection.

[TestMethod]
        public void WhenMonkeyListProviderGivesMultipleMonkeysAndUnleashMultipleTimes_UseThoseMonkeysAtleastOnce()
        {
            Settings settings = new Settings();
            ChaosLogger logger = new ChaosLogger("");
        //Arrange
            var mockMonkey1 = new Mock<ParentMonkey>(settings, logger);
            var mockMonkey2 = new Mock<ParentMonkey>(settings, logger);
            var mockMonkey3 = new Mock<ParentMonkey>(settings, logger);
 
            var mockMonkeyListBuilder = new Mock<MonkeyListBuilder>();
 
            mockMonkeyListBuilder.Setup(builder => builder.GetMonkeys(settings, logger))
                    .Returns(new List<ParentMonkey>() {
                        mockMonkey1.Object,
                        mockMonkey2.Object,
                        mockMonkey3.Object
                    });
 
            MonkeyKeeper keeper = new MonkeyKeeper(
                settings,
                logger,
                mockMonkeyListBuilder.Object);
            //Act
            keeper.UnleashRandomMonkey();
            keeper.UnleashRandomMonkey();
            keeper.UnleashRandomMonkey();
            keeper.UnleashRandomMonkey();
            keeper.UnleashRandomMonkey();
            //Assert
            mockMonkey1.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);
            mockMonkey2.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);
            mockMonkey3.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);
        }

Most of the times this test will succeed. But there is no guarantee that it will succeed always because Randon.Next() using inside MonkeyKeeper.Unleash can return same value 4 times and that's why its random. in such scenarios this test will fail. So whats the solution.

Solution

There is no solution to this. If we can predict random number and write asserts, that is not random. So what we can do?

At least make sure, there is some level of randomness in the selection of IMonkeys. Also we can make sure if there are 3 IMonkeys in the collection at least one of the IMonkey's unleash is called. If that is not called or called multiple times, there are some problems in the random selection logic of MonkeyKeeper.Unleash. Whats the problem in doing that?

We have 3 monkeys and we don't know which one will be selected by random logic. But we need to make sure that the unleash is called only once and on one monkey. This clearly tells that normal mechanism of asserting via Verify() and Times will not work. If there is any confusion just try writing that as below

mockMonkey1.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);
mockMonkey2.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);
mockMonkey3.Verify(monkey => monkey.Unleash(), Times.AtLeastOnce);

Why the above wont work? when we unleash it may call mockMonkey1. and that line works and test passed. But what about the next lines? They will fail. We cannot put a return statement because next time mockMonket3 might be the one selected. In that case first 2 statements will fail.

Now whats the method? We need to know the number of times a method is called on mocked object. There is no direct way to know that. So we need to use the callback mechanism available in Moq to achieve our aim. Callback function helps to get a hook in our unit test when a particular method is called on mock object. Code goes as below.
[TestMethod]
public void WhenMonkeyListProviderGives3MonkeysAndUnleashOnce_UnleashMustBeCalledOnOneMonkey()
{
    Settings settings = new Settings();
    ChaosLogger logger = new ChaosLogger("");
    //Arrange
    var mockMonkey1 = new Mock<ParentMonkey>(settings, logger);
    var mockMonkey2 = new Mock<ParentMonkey>(settings, logger);
    var mockMonkey3 = new Mock<ParentMonkey>(settings, logger);
 
    int callCount = 0;
    mockMonkey1.Setup(monkey => monkey.Unleash()).Callback(() =>callCount+=1);
    mockMonkey2.Setup(monkey => monkey.Unleash()).Callback(() => callCount += 1);
    mockMonkey3.Setup(monkey => monkey.Unleash()).Callback(() => callCount += 1);
 
    var mockMonkeyListBuilder = new Mock<MonkeyListBuilder>();
    mockMonkeyListBuilder.Setup(builder => builder.GetMonkeys(settings, logger))
            .Returns(new List<ParentMonkey>() {
                mockMonkey1.Object,
                mockMonkey2.Object,
                mockMonkey3.Object
            });
 
    MonkeyKeeper keeper = new MonkeyKeeper(
        settings,
        logger,
        mockMonkeyListBuilder.Object);
    //Act
    keeper.UnleashRandomMonkey();
    //Assert
    Assert.AreEqual(1, callCount, "Unleash Called multiple times");
}

http://stackoverflow.com/questions/16693078/when-using-moq-verify-method-invocation-count-have-failing-tests-error-messa

Tuesday, May 19, 2015

Unit testing ASP.Net Web Forms

Scenario

We have a simple legacy web application created using ASP.Net Web Forms. It works well and doing its duties without fail. Mainly it deals with simple pages not many transactions involving many pages. But rarely there are changes coming in. Even if the changes come frequently, the changes are small changes which are not affecting the core functionality of web site. Being a passionate software engineer, we wanted to bring innovation into the project such as refactoring to patterns and more object orientation.

Problem

We are confident on the refactoring we are doing in the site. But still there is a factor of fear that whether those changes will break the other pages in the site or not? Basically need to make sure all the links are working. Normally after a change we manually click on every link and make sure they are all loading.

That is a repetitive work. No software engineer should be doing repetitive work. We need to solve this in less cost. As usual since its legacy application, there is no budget to buy tools or automated testing service.

Approaches

1.Write a tool to validate links

We can write a tool which will parse the html of default.aspx and make sure all the links are working properly ie returning valid html contents. Can use XMLHttpRequest or WebClient class to accomplish it. Can configure the tool to run as post build action.

Challenges are in validation because there is no guarantee that HTML pages can be easily parsed with XML classes. There is a nuget called HTML Agility pack which we can try here.

2.Write unit tests on existing Web Forms app

Write unit tests for all the pages and at least make sure all the are passing.

After evaluating, I feel the unit tests are the way to do developer testing. But is my legacy application written as testable. Absolutely no. Now what we can do? One way is to rewrite as ASP.Net or use the Url based unit test. Refactoring the business logic and testing those is not applicable as the site doesn't have complex logic and more than that we need to make sure the pages are all working after our change.

3.Convert into ASP.Net MVC and unit test

Though this is the perfect solution most of the time this will not work out because.
  1. There may not be enough budget for maintaining legacy app or convert into ASP.Net MVC because from the business perspective, the site is doing its duty properly
  2. There might be links already bookmarked by users. Those will be invalid unless there is a rerouting mechanism in new MVC application.

Solution - Unit test ASP.Net Web Forms

The concept of unit test is applied to one level above the normal unit testing of classes and methods. Here we test the ASP.Net Pages itself. We can check particular properties of Page is set when rendered for a particular request url. Below are the steps to create unit tests for ASP.Net Web pages

1.Create UnitTest Project

Create normal unit test project from VisualStudio->New->Project dialog which we used to create for other scenarios such as class libraries or MVC.

2.Refer System.Web

Reference to Syste.Web.dll in UnitTest project is required if we want to hold the Page variable for assert.

3.Write test method with special attributes

Now comes the testmethod same as in normal test project. Here additions are 2 attributes HostType and UrlToTest. They specifies where the web page is hosted and what page corresponding test method is going to test. Below is one scenario where test method is testing a page hosted in Local IIS server.

string url = "http://localhost/joymononline/"
[TestMethod]
[HostType("ASP.NET")]
[UrlToTest(baseUrl+"links.aspx")]
public void WhenLinksPageIsRequested_ShouldReturn200AndContent()
{
    Page page = TestContext.RequestedPage;
    string url = Path.Combine(baseUrl, "links.aspx");
    //Assert.IsTrue(UrlIsValid(url));
    Assert.IsTrue(page.Title.Equals("Joymon Online | Links"));
}

If you are looking for how to test a web application running from development server, please have a look at the below links.


I strongly recommend developing by pointing to local IIS server, because IIS is the way your application is going to be served to end users. Otherwise there will be scenarios of  "works on my machine" which is really tough to deal with.

Debugging IIS hosted Web Forms Unit test

By default if we put a break point in ASP.Net Web Forms unit test where the site is hosted in IIS, it wont get hit. We need to attach to IIS process w3wp.exe.

So put a Debugger.Break or Debugger.Launch. This will open up an error dialog and from there we can attach to W3WP.exe.1

Also need to make sure that the dlls are compiled in debug mode.

Issues

Sometime there may be issues when running the unit tests. One error what I faced was below.

The URL specified ('http://localhost/joymononline') does not correspond to a valid directory. Tests configured to run in ASP.NET in IIS require a valid directory to exist for the URL. The URL may be invalid or may not point to a valid Web application

There are multiple reasons for this.

First we need to run Visual Studio in admin privilege. 

Second we need check whether the below entry is logged into event log viewer -> application  node

(QTAgent32_40.exe, PID 12920, Thread 14) WebSites.GetWebServer: failed to create AspNetHelper: Microsoft.VisualStudio.Enterprise.Common.AspNetHelperException: The website metabase contains unexpected information or you do not have permission to access the metabase.  You must be a member of the Administrators group on the local computer to access the IIS metabase. Therefore, you cannot create or open a local IIS Web site.  If you have Read, Write, and Modify Permissions for the folder where the files are located, you can create a file system web site that points to the folder in order to proceed. ---> System.Runtime.InteropServices.COMException: Unknown error (0x80005000)
   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.get_IsContainer()
   at System.DirectoryServices.DirectoryEntries.ChildEnumerator..ctor(DirectoryEntry container)
   at System.DirectoryServices.DirectoryEntries.GetEnumerator()
   at Microsoft.VisualStudio.Enterprise.Common.IISHelper.GetWebServerOrdinal(Uri site)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Enterprise.Common.IISHelper.GetWebServerOrdinal(Uri site)
   at Microsoft.VisualStudio.Enterprise.Common.IISHelper.get_WebServerOrdinal()
   at Microsoft.VisualStudio.Enterprise.Common.IISHelper.get_RootPath()
   at Microsoft.VisualStudio.Enterprise.Common.IISHelper.get_PhysicalPath()
   at Microsoft.VisualStudio.Enterprise.Common.AspNetHelperMan..ctor(Uri uri, BasicAuthCredential credential, Int32 frameworkMajorVersion)
   at Microsoft.VisualStudio.Enterprise.Common.AspNetHelperMan..ctor(Uri uri, BasicAuthCredential credential)
   at Microsoft.VisualStudio.Enterprise.Common.AspNetHelper.Create(Uri uri, BasicAuthCredential credential)
   at Microsoft.VisualStudio.TestTools.HostAdapters.WebSites.GetWebServer(String webServerName, WebServerType webServerType, String urlToTest, String pathToWeb, String webAppRoot, BasicAuthCredential credential, Context context, WebSiteConfigurationType webSiteConfigType, Origin origin)

If we can see the above we can try enabling IIS 6 compatibility. This is applicable if we are running on Windows 7 with IIS 7 or above.

Go to Control Panel->Programs and Features->Turn Windows Featuers On or Off

Then

Internet Information Services->Web Management Tools->IIS 6 Management Compatibility

Tick the "IIS Metabase and IIS 6 configuration compatibility" option and install it.

Some users says that we need to enable everything in IIS 6 Management Compatibility.2

References

1http://forums.asp.net/t/1246417.aspx?How+to+Debug+unit+tests+for+ASP+NET+web+application
2http://stackoverflow.com/questions/9065937/asp-net-unit-testing-windows7-iis7