Showing posts with label .Net 4. Show all posts
Showing posts with label .Net 4. Show all posts

Tuesday, October 3, 2017

Using .Net default value is trouble especially for APIs

Though there are features, they are not meant to be used. JavaScript atleast has a book named "JavaScript: The good parts" but others don't have one. Lets see one scenario in C# .Net.

Long long ago there was an API exposed to clients.

        class MyAPI
        {
            public void APIMethod(string s, int i = 10)
            {
                Console.WriteLine($"Inside foo with i = {i}");
            }
        }
internal void Test()
        {
            MyAPI apiClient = new MyAPI();
            apiClient.APIMethod("hi");
        }

Clients were happy using it. Later someone got added to the API team and he overloaded the method as follows thinking that keeping the same name will help the clients discover the API.

        class MyAPI
        {
            public void APIMethod(string s, int i = 10)
            {
                Console.WriteLine($"Inside APIMethod with i = {i}");
            }
            public void APIMethod(string s)
            {
                Console.WriteLine($"Inside APIMethod");
            }
        }

Clients happily adopted the new version of API. But soon they started feeling their calls are not working as expected. They escalated the issue and developers spent hours and days and finally they figured what went wrong. They corrected the code as follows.

        class MyAPI
        {
            public void APIMethod(string s, int i = 10)
            {
                Console.WriteLine($"Inside APIMethod with i = {i}");
            }
            public void NewAPIMethod(string s)
            {
                Console.WriteLine($"Inside NewAPIMethod");
            }
        }

Moral of the story

Do not use a feature only because it is available.

Tuesday, June 21, 2016

Orchestration library for .Net 4.0

Introduction

It was started with queued processing in one of the projects. We have a queue framework mechanism for long running processes. If anyone needs to use that queue, they should host a web service which will be called from queue framework after de-queue. The queue type and service url needs to follow some convention to get the connection.

Problem

If it is a technology shop where developers follow principles such as SOLID this problem will never occur. Here in the delivery shop, the problem was in the service method which does the long running operation people started writing 2000+ lines.

May be the developers were securing their jobs. Soon the code base became monopoly to corresponding devs. No other developers were able to touch it.

It went without noticed for couple of years. It became problem soon after people left the team and started getting failures on really long running (more than 1 hour) queue processes. It was supposed to resume but none of those resumed from the failure point on requeue. It became nightmare to makre it resumable. 

Why you developed it in sequential way? The developers answered this is the way we know. We need to do sequence of operation of the de-queued message. Output of one operation may affect the subsequent operation sometimes they are independent. If required we can separate to functions.

SRP

It is clear that developers were not following the Single Responsibility Principle. Separating steps to methods doesn't help here because some long running operations share steps. So its the time to enforce SRP.

Another problem is the way queue framework relies the control to actual long running program. It was a simple service call based on convention. It helped the developers to write a service and write the logic inside the implementation method.

The solution is to accept step objects by the queue framework and call those one by one instead of calling a service method. So after each step Orchestrator can store the state so that it can restart from the last step. The steps can be unit tested easily.

The Orchestrator born here.

Mutability - Pipelined operations on same object context

Another concern was to manage the state. These operations were sharing same state and some time one required output of previous. Not necessarily the immediate previous step.

Everywhere its evil to have mutability and people strive for immutable systems. But here it was little difficult to achieve the same. So continued with mutability.

But how to control the flow better?

Stateless steps

Why should a step be ever stateful if Execute(context) method gets the context via parameter? If the  Orchestration engine can pass the same context object to every step and step can modify the state and there is no parallel step execution we are good. Isn't it?

Yes we are good without parallelism. 

Concurrency & Parallel step execution

Parallel processing is always difficult to get right. One first factor is to decide who defines the what to run in parallel. Whether its the Orchestrator framework or the step authors? Its a debatable topic and most probably we will end up with step developers to tell what to run in parallel. What will happen if developers, who don't know SRP defines the unit of parallelism?

Most of the time stability is better than speed. So no concurrency as of now.

Source

Being an architect I can suggest the solution. But cannot enforce it as long as I don't have the rights to hold a release. But there is no problem in developing such a framework and open sourcing the same.

https://github.com/joymon/Orchestration

Note: This is for .Net 4.0 users. If the target is .Net 4.5, there is better library called Banzai.

Tuesday, April 19, 2016

Deserialize JSON string to C# dynamic type

Introduction

This is the age of APIs in computer science. To be more specific web APIs. Don't get confused with the specific technology of .Net Web API. Any methods or resources which are exposed via URL in a structure is an API. From our apps we can call them to do achieve the functionality.

Resources are now a days exposed via ReST protocol and the format is JSON. It works well with JavaScript. But when coming to .Net world, its little difficult to keep the JSON as is in our app. Often we need to convert those resources into entity classes. Creating entity class for each API response is a tricky tasks. It puts more time to complete the API integration tasks. Often after using one API for months we may need to switch to another API providing same service. At that time we need to rewrite it. Creating classes to get exact replica of response is really difficult to manage.

But its always suggested to have entity classes which are for our application regardless of the API we are using. As soon as we got the response from the web API, we should convert the response to our model and that should go on from that point on wards.

Dynamic object to hold JSON

In the beginning people were programming without type safety. Then the strong typed languages were introduced and everybody went behind that. Now its the time everybody again going to dynamic typing. May be its because of JavaScript's success.

VB always gave us the option to have dynamic typing. But in C# its introduced with .Net 4.0. The dynamic keyword is introduced to denote the variables which are not type safe. In other words a type which we don't know at compile time, what it is going to contain. Kind of key value store / dictionary.

Enough theory. Below is the code to convert a GitHub API response. The API url is 

private Project GetProjectFromJSON(string json)
{
    JavaScriptSerializer ser = new JavaScriptSerializer();
    dynamic projectNode = ser.Deserialize<dynamic>(json);
    return new Project()
    {
        Name = projectNode["name"],
        Description = projectNode["description"],
        SourceUrl = projectNode["html_url"],
        Url = projectNode["homepage"]
    };
}

JavaScriptSerializer is a .Net framework class. We don't need to add any dependency to third parties. There are other .Net libraries which are helping to convert JSON to dynamic object but those will add unwanted dependencies. Hope this is self explanatory.

Important thing to note here is the usage of dynamic. Do not ever return dynamic to outer world. This method should hide the complexities of getting the project from GitHub. The consumers of this method then don't need to change when the logic of GitHub API interaction changes.

More details about usage in real application can be seen in this file.

Tuesday, October 20, 2015

Hosting Evernote .Net SDK in Godaddy or other hosting provider

There is no doubt that Evernote is a simple but great product. It was simple desktop application in the begining and moved to cloud at the right time. Cloud storage helps our notes to be synced across technology, platform and devices.

I was a proud user of Google notes. But Google shut it down. From that moment, I moved to Evernote cloud.

Evernote SDK for .Net - Using local file system

Evernote has exposed APIs to interact with out notes and there is SDK for programming models which help us to use the API easily. But if we try to host our application which the Evernote SDK in Godaddy just to read the notes, we will  get errors as Godaddy is not providing file system access. It will happen in all the shared hosting providers where our code don't have access to full file system. First it will ask to give full trust. After giving full trust we will get more specific exception. Details below


Exception:System.UnauthorizedAccessException Message:Access to the path '\Evernote\EvernoteSDK' is denied. Exception:System.UnauthorizedAccessException 

Message:Access to the path '\Evernote\EvernoteSDK' is denied. 

StackTrace: 
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) 
at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost) 
at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost) 
at System.IO.Directory.CreateDirectory(String path) 
at EvernoteSDK.Advanced.ENPreferencesStore.Save() 
at EvernoteSDK.Advanced.ENPreferencesStore.SetObject(Object objectToStore, String key) 
at EvernoteSDK.ENSession.PerformPostAuthentication() at EvernoteSDK.ENSession.Startup()
at EvernoteSDK.ENSession..ctor() 
at EvernoteSDK.ENSession.<.cctor>b__10() 
at System.Lazy`1.CreateValue() 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Lazy`1.get_Value() 
at EvernoteSDK.ENSession.get_SharedSession()

We can clearly see that its trying to access local file system.

If the hosting is by renting a virtual machine this issue will not happen as we can give our code full access to the file system.

Root cause

If we look at the Evernote SDK source in Github, we can see it is caching the data/preferences in local file system. Currently there is no way to control that behavior. There is not even a way to provide our own provider.

Fix

Fix is simple. Need to get the Evernote SDK for .Net source code and refactor to have preference provider interface. Move current code into default file system based preference provider so that existing code will not break. For our special case of web hosting into Godaddy kind of servers, write new preference provider which is based on memory. 

Yes the performance may get affected due to app pool recycling and rebuilding of preference cache, but it will work in the web.

The fix is available in the below repository. A pull request is created, Evernote is yet to merge into their code base.

Tuesday, July 14, 2015

Not abel to create DTS.LogProviderTextFile.2 when running SSIS from .Net

Recently I was trying to execute a SSIS package from C# .Net code and it was not working as expected. So I decided to get the logs from the package. Unfortunately there was no logs. So I decided to inject the LogProvider and ConnectionManager from my C# code during execution.

Google lead me to a MSDN link which explains the same.
https://msdn.microsoft.com/en-us/library/ms136023(v=sql.105).aspx.

I selected SQL Server 2012 to get more accurate code snippet. The BIDS version I used was VS2010 based.

But I got below exception when I run the sample code.

A first chance exception of type 'Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException' occurred in Microsoft.SqlServer.ManagedDTS.dll

Additional information: The log provider type "DTS.LogProviderTextFile.2, {0A039101-ACC1-4E06-943F-279948323883}" specified for log provider "{652479A1-923C-452D-9328-5F703624B5FC}" is not recognized as a valid log provider type. This error occurs when an attempt is made to create a log provider for unknown log provider type. Verify the spelling in the log provider type name.

Googled again and was not able to get much help. So decided to debug myself. I added LogProvider from BIDS UI and checked its type in Visual Studio before executing the package. It says the type is "DTS.LogProviderTextFile.3". It clearly indicate that if we create log provider with .3 suffix, it will work. I tried and it succeeded.

So if the target SSIS run time is SQL 2012 change the provider type to DTS.LogProviderTextFile.3. 

Tuesday, November 11, 2014

Simple interface based programming

For some long I was searching for a good article which explains how to use interface in .Net programming. It is not for a developer who is passionate in programming, got good knowledge in object oriented programming and comes with computer science degree. But for people who came into the field because others compelled and not having computer science background. At some point I thought of writing one myself. But I was able to find a good one. Below are the link(s). I will be updating if I came across another one.

https://www.simple-talk.com/dotnet/.net-framework/designing-c-software-with-interfaces/

Tuesday, November 4, 2014

.Net Access modifiers - Revisited

This is intended mainly for beginners to enforce their understanding about access modifiers used in .Net. We are going to discuss it in a question answer fashion. Read the below MSDN link which explains the topic and try to answer the questions.

http://msdn.microsoft.com/en-us/library/6tcf2h8w.aspx
http://msdn.microsoft.com/en-us/library/ms173121.aspx
  1. Is there any method even a hack, we can access private members from other class?
  2. If we can access private member via some mechanism what is the meaning of all these access modifiers?
  3. We are using access modifiers to restrict the access to fields and functions. Do we need any user name and password to access the variables? 
  4. Can't we program if there is no concept of access modifiers in the language? What are the advantage of restricting access?
  5. Can we have same namespace in 2 different assemblies? If so can I access internal classes in other assembly which are in same namespace?
  6. Can we have a public method inside internal class? What is the specialty?
  7. Can a public method return an internal type?
  8. Can we have protected class in any scenario? Where all we can use it?
  9. There is a public class named A.Can I have a protected class in same name A inside class A?
  10. If we make the constructor as private, will that class be unusable for ever?
  11. There is a public method which returns a public interface. In the method body can we have 'return new ImplClass();' where the ImplClass is internal / protected / private?
  12. Can a public class inherit internal class?
  13. Can a class inherit a class which is declared inside the same class?
  14. Can a public class expose a property which is of internal class type?
  15. Are the access modifiers mutually exclusive in .Net? Can I have more than one access modifiers for single member?
  16. Does 'sealed' access modifier seal the class for ever? Will that class create dead code in the assembly?
  17. Can we have protected static methods in non static class?
  18. Can we have protected method inside static class?
  19. Can we have public static constructor in a non-static class?
  20. Can I declare static variable inside a method?
  21. Can we have a member variable in a class of same type? Will that cause infinite recursive loop?
  22. When I tried the above scenario, I got StackOverflowException? Why its so?
Initially I thought I should be writing 2 posts so that when trying to answer these questions you don't get the temptation to look at the answer. Later sometime, the thought process changed. We are all professionals and true professionals can never lie to his profession. We are not trying this to get marks or pass exam here. This is just for gaining knowledge or understanding our level of OOP knowledge. So answers are there in this post only.
  1. Yes there is. Using reflection we can access the private properties or call private methods of other class / object. 
  2. .Net is managed the run time knows what are the types. But object orientation started before the managed environments. In that world it has sense. Even in .Net there are some framework classes whose members cannot be invoked via reflection.
  3. This is really childish when I asked this during one interview, he really started explaining about ASP.Net membership. He was a developer born into ASP.Net and according to him .Net is for ASP.Net. The access modifiers control the callers / consumers of the class in object creation and usage of it's members through objects. When we say via objects, it might be in a method in same inheritance hierarchy using 'this' or 'base' keywords.
  4. We can program even if there is no access modifiers in the language. In that case everything will be public. Below are the advantages of access modifiers
    1. Refactoring - Suppose we need to re-factor a method, by adding one more argument. If the method is private we need to care about only one class to get successful compilation. In case you are working in huge legacy code base (I am now working in a code base which is 10+ years old,contains more than 300 projects) and the method is public, getting compiled in first shot is a nightmare.
    2. Library development - There is no access modifiers in language and we are developing a library which has a method WeatherAtLocation GetWeatherAsync(location) for getting the weather information from a web service asynchronously and returns a structure which has the parameter location and the temperature. The method temporarily stores the location in a class level variable. Its obviously public. By the time our async web request to the actual web service completes the caller can change the value in member variable. So when we return the result it has a different location. 
  5. No. Namespaces has nothing to do with internal access modifier.
  6. Yes we can. But effectively the access of that method will be internal as the contained type is internal.There is one blog post by Eric on this.
  7. No. The public method is supposed to call by classes in other assemblies. If it returns an internal type, which is restricted to classes in other assemblies, we are violating the internal 
  8. Yes if we are declaring a class inside another class it can be protected. We can create object of this protected class in members of same class and all its derived class and its members regardless of assembly.
  9. Yes we can have. If we create object from outside, it will create object of outer class A. If the object creation is inside a member function of outer class A it will be the nested/protected class A.
  10. No. Private means accessible to all the members in the same class. If the class is public / internal, we can have a public / internal static method which creates the object and returns the same. Singleton uses this technique.
  11. Yes. The public method is returning public interface type. The interface members are permitted to be called from classes in other assemblies.
  12. No. If we do so, we are expanding the reach of internal.
  13. No. A class cannot inherit nested class inside it.
  14. No. The property is public but type is internal. So there is no meaning in returning the Type via  property. Similar to Qn 7.
  15. Yes. Some are mutually exclusive. We can have protected internal as combination. It tells that the method can be accessed in any inherited class regardless of which assembly OR in any class inside same assembly.
  16. Sealed modifier for the declaration. If applied on class it means the class cannot be inherited.If on method it cannot be overriden.
  17. Yes we can have static protected members. Those will be accessible in same class or inherited classes.
  18. We cannot have protected instance methods. In fact no instance members. Also we cannot have protected static methods inside static class. There is no inheritance support.in static classes.
  19. No. Access modifiers are not allowed on static constructor as its not callable from our code. Static constructors are supposed to execute when that type is first used regardless of where from that call originated and to which member. So it doesn't make sense.
  20. No. This was supported in C. But not in .Net.
  21. Yes we can have a member variable inside the class which is of same type. That won't cause any infinite loop.Otherwise linked list would not have been possible.
  22. If we just declare, there wont be any issue. But if we try to instantiate that object in the constructor without any condition check or initiate the object in the same line of declaration, it will throw StackOverflowException.

When I started, I thought there would be limited questions. But there are many. May be the compiler team at Microsoft will be having near full list as their unit test cases. Once I get some time I will surely tag / group these questions as Static,Inheritance etc...

Tuesday, September 16, 2014

Overriding without virtual and override keywords in .Net

This is about a particular scenario where we can achieve overriding without virtual or override keywords in C#. Below is the scenario of an inheritance hierarchy.
Here we can see that both the base and derived class implemented the IDisposable interface. In this case we don't need to mention the keyword overrides but the Dispose() of DerivedDisposable class will be called when we invoke the Dispose() method on the IDisposable type variable, if it contains the DerivedDisposable class object.

            IDisposable id = new DerivedDisposable();
            id.Dispose();

This will invoke the method written inside the DerivedDisposable class.

Calling base class Dispose method

Normally in overriding scenarios, we may call the base class implementation by using base.OverriddenMethodName(). But in this case, base object will not have the Dispose method, if the IDisposable is implemented explicitly. Means the code below won't compile.


    class DerivedDisposable : Disposable,IDisposable
    {
        void IDisposable.Dispose()
        {
            base.Dispose();//This wont compile if the base class / class Derived in this example implements the IDisposable explicitly
        }
    }


Next idea what we can do is to cast the base to IDisposable and call the Dispose method. But that too won't compile.


    class DerivedDisposable : Disposable,IDisposable
    {
        void IDisposable.Dispose()
        {
            //DoWork();
            (base as IDisposable).Dispose();//Won't compile
        }
    }

Implicit interface implementation

Only way to call the base class Dispose method is to make sure the base implements the interface implicitly.Then we can call base.Dispose method().


    class Disposable : IDisposable
    {
        public void Dispose()
        {
            Console.WriteLine("Disposable.Dispose");
        }
    }
    class DerivedDisposable : Disposable,IDisposable
    {
        void IDisposable.Dispose()
        {
            //DoWork();
            base.Dispose();
        }
    }


If we are concerned about having additional public function, another workaround is to have separate protected function in base class (eg:DisposeWorker) which is called from the Dispose() of base class to carry out the task. So from the derived class.Dispose() method we can call that protected function.


    class Disposable : IDisposable
    {
        void IDisposable.Dispose()
        {
            DisposeWorker();
        }
        protected void DisposeWorker()
        {
            Console.WriteLine("Disposable.Dispose");
        }
    }
    class DerivedDisposable : Disposable,IDisposable
    {
        void IDisposable.Dispose()
        {
            //DoWork();
            base.DisposeWorker();
        }
    }

Real scenario - Overriding the ClientBase<TChannel>.Dispose

This is useful when overriding the Dispose method of ClientBase<TChannel> class related to WCF to have our own behavior when disposing. In our project as I mentioned in earlier posts, we are not adding service reference to call WCF service. Instead we create our own proxy class by implementing ClientBase<TChannel>. We have proxy pooling technique built into the system and it needs "return  of proxy to the pool" when Dispose is called on proxy object.


    class Client : ClientBase<IService1>, IService1, IDisposable
    {
        void IDisposable.Dispose()
        {
            Console.WriteLine("Disposable.Dispose");
        }
 
        string IService1.GetData(int value)
        {
            throw new NotImplementedException();
        }
    }


I am really not sure whether its overriding or 'feels like' overriding. Anyway I am open to discussions. This is one of the scenario I could see which can only be solved by using implicit interface implementation.We will face this same issue if we implement interfaces explicitly in VB.Net.

Happy coding

Tuesday, July 22, 2014

Not able to uninstall .net assembly from GAC

Occasionally when we try to uninstall .net dll from GAC by pressing delete button, it will show the below error message.
Unable to uninstall: assembly is required by one or more applications

Solution 1 - Run in admin mode

The major reason could be the user don't have permission to delete the dll from GAC. If we run the windows folder explorer in administrator mode it might be uninstalled. How to run the windows explorer as in administer mode? We cannot right click and "Run as administrator" on any item to get an explorer window in admin mode. So we need to run the cmd in admin mode. and use the GACUtil.exe to remove the dll. 

Most of the cases it will get uninstalled. Else try the below link to and uninstall as admin .

Solution 2 - Installer problem. Edit registry

There is one more reason why we cannot uninstall from GAC. Sometimes the the assembly might have installed into GAC by installers and they will hold the reference so that we cannot uninstall from GAC. If this is the case we need to remove some registry entries present inside the below registry path.

[HKCU\Software\Microsoft\Installer\Assemblies\Global]
[HKLM\SOFTWARE\Classes\Installer\Assemblies\Global]

This is already reported to MSFT and more details can be found in below links.
http://support.microsoft.com/kb/873195
http://blogs.msdn.com/b/alanshi/archive/2003/12/10/42690.aspx

Last week when I was trying to run a sample which I got from internet, it was loading wrong Microsoft.Practices.Unity.dll. I could see that there is one Microsoft.Practices.Unity.dll of different version in the GAC. So thought of deleting it from GAC and it ended up in the above mentioned error.

Tuesday, May 6, 2014

Danger of using ThreadStatic variables if the threads are from ThreadPool

What is static variable

We know there is static variables in .Net which are bound to class not to objects. ie there will be only one memory location for a static variable in our AppDomain. If the same variable is accessed from different app domain it will get a different memory location. What is mean by one memory location per one static variable?

Its simple. From where ever we change the value of the static variable, it will reflected throughout the .Net application / app domain.

         static void Main()
    {
        Adder a1 = new Adder();
        a1.Add(2, 3);
        Adder a2 = new Adder();
        a2.Add(4, 5);
        Console.WriteLine(a1.Result);
        Console.WriteLine(a2.Result);
    }
}
//Demo purpose - Never use in your apps
class Adder
{
    static int _Result;
    internal int Result { get { return _Result; } }
    internal void Add(int n1, int n2)
    {
        _Result = n1 + n2;
    }
}

You may easily understand that the result shown for a1 addition(2,3) is wrong. This is because the result of a2 addition (4,5) has overridden the previous static variable value. In other word the static variable have only one location to store its value.

Static variable v/s Instance variable

Its very simple. If the value of a static variable is changed from an instance method,that value will be available in all the other instance methods. Unlike the instance variable where there will be different memory location for storing values of the variable per instance.

The beginners in programming sometime experienced but not worked in multi-threaded world, tend use static variables as it is the easiest medium to transport a value between disconnected classes.


        internal static void TestStatic()
    {
        Adder a1 = new Adder();
        a1.Add(2, 3);
        Displayer.Display();
        Adder a2 = new Adder();         a2.Add(4, 5);         Displayer.Display();     } } //Demo purpose - Never use in your apps class Adder {     internal static int _Result;     internal int Result { get { return _Result; } }     internal void Add(int n1, int n2)     {         _Result = n1 + n2;     } } class Displayer {     internal static void Display()     {         Console.WriteLine(Adder._Result);     } }

The Adder and Displayer class has no relation.In these disconnected scenarios people will prefer static variable to transport the value. It will display the values properly as long as there is single thread and the methods are invoked in order.

 Static variable in threading scenario

The static variables will keep their values across threads as well. For example if one instance method of an object is executed inside the thread and it changes a static variable, the changed value will be available in another thread which is executing same instance method of another object. What is the problem if the value is reflected in other threads?


    static void Main()
    {
        new Thread(() =>
        {
            Adder a1 = new Adder();
            a1.Add(2, 3);
            Thread.Sleep(100);//This is to simulate processing delay
            Displayer.Display();
        }).Start();
 
        new Thread(() =>
        {
            Adder a2 = new Adder();
            a2.Add(4, 5);
            Displayer.Display();
        }).Start();
    }
}
//Demo purpose - Never use in your apps
class Adder
{
    internal static int _Result;
    internal int Result { get { return _Result; } }
    internal void Add(int n1, int n2)
    {
        _Result = n1 + n2;
    }
}
class Displayer
{
    internal static void Display()
    {
        Console.WriteLine(Adder._Result);
    }
}

Here we can see the result of a1 addition (2,3) is overridden by the a2 addition. Also the behavior is not predictable. Sometimes the a2 addition may complete before a1 addition and a2 result might be 5 which is wrong.

lock to avoid unwanted manipulations

As everybody knows we can use the lock keyword to sequence the execution of code blocks so even if there are multiple threads running in parallel only one thread will get the lock and execute the code block at a time. In other words, we can sequence the execution of locked code block.

static object lockObj = new object();
        internal static void TestStatic()
        {
            new Thread(() =>
            {
                Adder a1 = new Adder();
                lock (lockObj)
                {
                    a1.Add(2, 3);
                    Thread.Sleep(100);//This is to simulate processing delay
                    Displayer.Display();
                }
            }).Start();
 
            new Thread(() =>
            {
                lock (lockObj)
                {
                    Adder a2 = new Adder();
                    a2.Add(4, 5);
                    Displayer.Display();
                }
            }).Start();
        }
    }
    //Demo purpose - Never use in your apps
    class Adder
    {
        internal static int _Result;
        internal int Result { get { return _Result; } }
        internal void Add(int n1, int n2)
        {
                _Result = n1 + n2;   
        }
    }
    class Displayer
    {
        internal static void Display()
        {
            Console.WriteLine(Adder._Result);
        }
    }

Here we can see things are working proper.But we lost the benefits of threading as its sequenced.

ThreadStatic

The simple static variable share common memory location inside the AppDomain. What if there is a way to have memory location per thread. So that the value of static variable will be available if the code is executed in same thread? Yes its an easy way to control problems of static variables which are accessed across the threads. But is it the silver bullet to solve the issue always? Lets see below scenario as well where the threads are coming from ThreadPool


internal static void TestStatic()
     {
         new Thread(() =>
         {
             Adder a1 = new Adder();
                 a1.Add(2, 3);
                 Thread.Sleep(100);//This is to simulate processing delay
                 Displayer.Display();
         }).Start();
 
         new Thread(() =>
         {
                 Adder a2 = new Adder();
                 a2.Add(4, 5);
                 Displayer.Display();
         }).Start();
     }
 }
 //Demo purpose - Never use in your apps
 class Adder
 {
     [ThreadStatic]
     internal static int _Result;
     internal int Result { get { return _Result; } }
     internal void Add(int n1, int n2)
     {
             _Result = n1 + n2;   
     }
 }
 class Displayer
 {
     internal static void Display()
     {
         Console.WriteLine(Adder._Result);
     }
 }

This will keep keep separate memory location for static variables based on the thread. 2 Disconnected classes can access same value of static variable if their member function are executed in same thread. Else they will get different values. Here the application logic works fine.

ThreadPool

I don't think, I should explain what is thread pooling. In fact that is not the aim of this post. As mentioned in the previous section, the thread static values are stored based on the thread. So what if the same thread is coming back from the pool with a value which is set by us last time, when it was in use? Yes we are in trouble.

If we write code for taking thread from pool,we know this issue will happen.But what about ASP.Net where it internally uses the ThreadPool to take threads which are used to serve each request? We may not notice the fact and our code will fail during load testing / staging / production.


    static void Main()
    {
        ThreadPool.SetMaxThreads(1,1);
        for( int i=0;i<10;i++)
        ThreadPool.QueueUserWorkItem((state) =>
        {
            Adder a1 = new Adder();
                a1.Increment();
                Thread.Sleep(100);//This is to simulate processing delay
                Displayer.Display();
        });
    }
}
//Demo purpose - Never use in your apps
class Adder
{
    [ThreadStatic]
    internal static int _Result=0;
    internal int Result { get { return _Result; } }
    internal void Increment()
    {
            _Result += 1;   
    }
}

Here the class has altered to increment the value. We are always expecting the value 1 as in each thread the _Result is supposed to initialize to 0 due to its ThreadStatic behaviour. But we can see as the threads are reused, there will be values displayed which are higher than 1. So what is the solution?

ThreadStatic variables in ThreadPooling

Always make sure we are initializing the threadstatic variables where our thread handler is started. If possible use a using{} block to wrap our thread code so that in the constructor of using{} block object,we can initialize the thread static variables to their defaults.

Conclusion

So avoid static variables at any cost. If they are unavoidable use ThreadStatic and initialize their values in thread handler.

Tuesday, April 29, 2014

XSS attack into ASP.Net MVC 4 site via Javascript comment

Introduction

Till this moment I strongly believe that, software security can not be achieved. I am telling this within my 6 academic + 8 professional years of experience with computer systems. I did building computer systems as well as hacking the same. Whenever I architect a system most of the time I was able to forecast the holes too. But since timely release of the application takes more priority in most of the cases, I have no way except building the systems with security holes. 
"Computer software security is always a concept like perfection"
Hacking a desktop based software is more easy because we can always see the source code which helps to understand how its being executed. Sorry to the new generation developers who born into the managed world. When the old generation says source code its always assembly language which cannot be obfuscated or protected. Assembly language is enough to understand how the system works.

Web systems are easy to build as it requires primary knowledge of HTML and one scripting language. Now a days with the penetration of Node.JS, one can create web sites with only HTML & Javascript knowledge without even knowing how the web works on request-response mechanism. If a hacker finds out a security hole in a desktop system and make use of it, the impact will be normally less. For example if somebody hacks the serial key mechanism, only some people are going to use it, unless he puts that in internet. But think about a web system, if attacker succeeds in hacking a bank web site, the impact will be huge.

XSS is one type of the web attack.In short we can define that as below

"Injecting script into the system as if, it is input data and getting the same script executed on other user's browser"

I know this is not easy to digest at the first glance. If you need more details, visit the below links.
Below is one sort of XSS attack which uses the query string to inject script and makes it available inside a script block.
http://buffered.io/posts/xss-flaws-via-mvc-model-binding-and-request.querystring-inconsistencies/

One more special case is to exploit the javascript encoding and passes the javascript via innerHTML property of html elements.
http://weblogs.asp.net/jgalloway/archive/2011/04/28/preventing-javascript-encoding-xss-attacks-in-asp-net-mvc.aspx

XSS via commented JS which has ASP.Net MVC RAZOR token

Normally in MVC people may not use the old query string technique.Also the sites will be having the ASP.Net request validation enabled. The ASP.Net request validation scans all the requests for any possible script tags and throws exception on the same.

What if the script came through an Action method without script tag and renders with the help of a commented line and get executed without even an HTML event? Again, the Action method can be invoked both by get requests and post requests.

We are going to discuss how such a situation can occur.

The victim Web application / ASP.Net site

In most of the cases the security holes are opened by the programmers unknowingly. Lets take an example of a simple web site. Here the programmer wants to implement a contact page where the contact person's name comes via an action method.

@model MvcExceptionHandlingDemo.Models.Contact
@{
    ViewBag.Title = "Contact";
}
 
<hgroup class="title">
    <h1>@ViewBag.Title.</h1>
    <h2>@ViewBag.Message</h2>
    @using (Html.BeginForm())
    {
        @Html.LabelFor(s => s.Name)    @Html.TextBoxFor(s => s.Name)
        <input type="submit" value="Change" />
    }
</hgroup>

        public ActionResult Contact(string name)
        {
            ViewBag.Message = name;
            if (ViewData.Model == null)
            {
                ViewData.Model=new Contact(){Name=name};
            }
            return View();
        }
        [HttpPost]
        public ActionResult Contact(Contact c)
        {
            ViewBag.Message = c.Name;
            if (ViewData.Model == null) ViewData.Model = c;
            return View();
        }

One of his intention was to show the social profile of that person by using some web services which accepts name. So wrote a javascript function. But later he commented the js code because he was not able to find suitable web API which gives the social profiles.

<script type="text/javascript">
    function dispProfile() {
        //var s = '@ViewBag.Message';
    }
</script>

How to conduct attack

The base for this attack is the option of rendering text into javascript which is commented. The MVC framework doesn't know about the javascript comment. So it will do its duty which is nothing but rendering the value in @ViewBag.Message property.

What string we should inject as script in this scenario is little tricky.
  1. We need to close the existing js function. 
  2. Then we can have our script. Since it is outside of the function it will get executed when page loads.
  3. Finally we need to have a dummy function with only starting '{' to make sure the script is in correct syntax
Below is one sample javascript which we can inject

';
}
alert(2024);
function dummy(){
//

This will make sure dispProfile() function closed properly. and write the attack script (Here it is alert(2024);) and starts a new dummy function to make the syntax proper.Otherwise the dispProfile() function will be in open state. So after the injection the rendered script will be as follows.

 <script>
    function dispProfile() {
        //var s = '&#39;;
}
alert(2024);
function dummy(){
//';
    }
</script>

One more thing to remember is how to inject this script .One of the option is to exploit the Contact action method with msg parameter. If we normally inject any script tag the ASP.Net request validation system will throw error. But here we are not injecting the script tag as it is. Instead some fragments of script. The script as its cannot be added to URL. For that we need to use the URLEncoded string of the script. Encoded URL can be

http://localhost:50471/Home/Contact?msg=%27%3B%0A%7D%0Aalert(2024)%3B%0Afunction%20dummy()%7B%0A%2F%2F

There are so many online URL encoding and decoding sites. Those can be used to do the encoding.

How to prevent

In this case the problem is commented code. The intention of the programmer is to stop the functionality of dispProfile() but he forgot the fact that its containing RAZOR token. He should have commented RAZOR code as well. In other words here same program is handled by 2 execution engines. RAZOR & Javascript. So when commented it should be commented for both the engines.

Sample application can be downloaded from the below link.

Monday, March 31, 2014

Getting unique records based on property from list without using Distinct() in Linq

The scenario is very simple. We have a collection of Person objects. We want to know the unique PersonTypes from the collection. Below the Person class definition.

    public class Person
    {
        [Display(Name = "First Name")]
        public string PersonType { get; set; }
        public XElement Properties { get; set; }
    }
    public class PersonType
    {
        public string TypeName { get; set; }
    }

I don't think nobody will write a loop to achieve this. The normal solution we may think of is to get the distinct Person objects using the Distinct function in Linq. But it requires an IEqualityComparer derived object to handle the comparison which is employed to find the duplicates by the Distinct function. I have explained about IEqualityComparer in one of my previous posts related to merging 2 collections without duplicates.

        static internal IEnumerable<PersonType> GetUniquePersonTypes()
        {
            IEnumerable<PersonType> personTypesList = _persons.Distinct(new PersonComparer())
                                                                                                .Select(per=>new PersonType(){TypeName=per.PersonType});
            return personTypesList;
        }
   public  class PersonComparer:IEqualityComparer<Person>
    {
        bool IEqualityComparer<Person>.Equals(Person person1, Person person2)
        {
            return person1.PersonType.Equals(person2.PersonType);
        }
 
        int IEqualityComparer<Person>.GetHashCode(Person person)
        {
            return person.PersonType.GetHashCode();
        }
    }

GroupBy to find distinct objects based on entity property

Should we really need Distinct and another class which implements IEqualityComparer? If we group the list based on the property which is key to decide duplicate and build a collection out of the group keys, won't that be same result of Distinct with comparer? Lets try it.

        static internal IEnumerable<PersonType> GetUniquePersonTypes()
        {
            IEnumerable<PersonType> personTypesList= _persons.GroupBy(x => x.PersonType)
                                                                                            .Select(group => new PersonType() 
                                                                                                                { TypeName = group.Key }
                                                                                            );
            return personTypesList;
        }

Is it working? Happy coding

Monday, February 17, 2014

Using PerfView to catch eaten .Net exceptions

Most of the time eaten exceptions are nightmares to debugging guys. Almost all the developers are taught that never eat any exception but still there are chances that they may eat the exception as a short cut to hide the issues in their code. An exception means something  happened wrong and there should be some actions  to be taken to resolve or to inform the caller about that error. We cannot continue the program execution same way as if, there was no exception. When the developers hide exceptions they can continue for short period of time but eventually the consumers, who calls their APIs will get wrong data back and their tracks will fail. Usually by that time the application might be in business testing or other final rounds of testing. In the worst case it might be reached production. At this time if something goes wrong it would be difficult to find out as the original error is hidden somewhere other than the place where the error is visible.

There are many ways to capture these exceptions.Here I am explaining one of those ways to find out eaten exceptions using a free Tool called PerfView from Microsoft. You may download the tool from Microsoft downloads. Its a standalone executable. Below are the steps

  1. Run the tool.
  2. Start collection.
  3. Analyse the .etl file which is generated.
Here I am explaining in more detailed way by taking a scenario.

Sample scenario / application


        private static void DivideWhichNeverFails(int n1,int n2)
        {
            int r = 0;
            try
            {
                r = n1 / n2;
            }
            catch (Exception ex) 
            { 
            }
            Console.WriteLine(r);
        }

I don't think the above code needs any kind of explanation. The developer cleverly written to escape from the error. The caller will get a zero even if his operation didn't carried out. Again, this will happen only in the projects where the developers are working like type writers who just produce lines of code and code review is not at all followed.

If such a code enters into the code base, it may survive the testing as long as there are no calls into this method with n2 as 0. But it may fail once its in production on the label of an inconsistent issue. When we need to deal with such issues in production, the first thing should be to check for eaten exceptions. Perfview is a gift here.

Starting PerfView and collecting.

Just run the PerfView.exe and select the Collect menu -> Run.It will show a popup which has boxes to fill our application name and the log file path. Thanks to .Net CLR team for instrumenting everything using ETW. The PerfView collects the data using ETW providers and the file extension is .etl. Make sure in the 'Advanced Options' section '.Net' is checked.  'Run Command' button will run the application. Execute the use case which is reported as error. Then close the application. The data collection phase is over.

Analyse the .etl file

Now we need to analyse the logs collected. In the main window of PerfView.exe there is a file browser.Point it to the .etl file which is created as part of the data collection. Double click on the etl file to expand it. Select the 'Events' node and open it by double click. Then filter the 'Event Types' using Exception to see all the exceptions. It will give us the list of exceptions happened during the run.

To see the call stack, expand the 'Exception Stacks' node of the etl file. It will open the call stacks for all the processes happened in the system during the collection. Find out our particular process and get the exception.

Using PerfView to find out eaten exceptions of ASP.Net web apps.

Here instead of running particular application, use the global collection method. To collect everything, select Collect menu-> Collect option.Set the data collection etl file and click the 'Start Collection' button. Then execute the failed test case. Stop the collection and start analysis in the same way did for windows application.

Happy debugging.