Tuesday, December 30, 2014

Orchestrator pattern in .Net - Using TPL

This is continuation of my previous posts to find out the best mechanism to sequence or orchestrate operations in .Net. Running in parallel is added advantage in the library.
  1. State pattern v/s Orchestrator
  2. WWW - Passing arguements to custom activity inside a sequence container in coded workflow
  3. Orchestration pattern in .Net - Using TPL

Using TPL to run tasks sequentially

Though the word TPL stands for Task Parallel Library, we can use it to execute tasks in sequence. One example is given below.

    class TPLTest
    {
        internal void Test()
        {
            Task.Factory.StartNew(new Action(Step1))
                .ContinueWith(new Action<Task>(task => Step2()))
                .ContinueWith(new Action<Task>(task=>Step3()));           
        }
        private void Step1()
        {
            Console.WriteLine("step 1 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 1 end");
        }
        private void Step2()
        {
            Console.WriteLine("step 2 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 2 end");
        }
        private void Step3()
        {
            Console.WriteLine("step 3 start");
            Thread.Sleep(2000);
            Console.WriteLine("step 3 end");
        }
    }

Passing context object into steps

In normal business applications, when we deal with this type of sequential operation, there would be one or more contextual objects which needs to be modified as part of the execution. Either the result of one step needs to be passed as parameter to next step or one contextual state object will be shared by all the steps.

All steps sharing same state object

Its very simple to achieve using TPL. Just keep the state object in the class level so that all methods can access it.


    class TPLTest
    {
        //State kept as int for demo purpose only. In actual applications the state object will be more complicated
        int state = 0; 
        internal void Test()
        {
            Task mainTask = new Task(new Action(Step1));
            mainTask.ContinueWith(new Action<Task>(task => Step2()))
            .ContinueWith(new Action<Task>(task => Step3()));
            mainTask.Start();
        }
        private void Step1()
        {
            Console.WriteLine("step 1 start. state ="+state.ToString());
            Thread.Sleep(2000);
            state = 1;
            Console.WriteLine("step 1 end. state =" + state.ToString());
        }
        private void Step2()
        {
            Console.WriteLine("step 2 start. state =" + state.ToString());
            Thread.Sleep(2000);
            state = 2;
            Console.WriteLine("step 2 end. state =" + state.ToString());
        }
        private void Step3()
        {
            Console.WriteLine("step 3 start. state =" + state.ToString());
            Thread.Sleep(2000);
            state = 3;
            Console.WriteLine("step 3 end. state =" + state.ToString());
        }
    }

Executing in pipeline mode / Output of one step is fed as input to next step

Below is the code snippet which shows how the TPL can be used to execute tasks sequentially by feeding the output of one step as input to next step.

   class TPLTest
    {
        internal void TestPipeLine()
        {
            //int is used as input for demo purpose only. 
            //In actual application the input object will be more complicated class
            int input = 1;
            //There is no typed way to pass input to the pipeline.It accepts object and explicit conversion is required to get int value.
            Task<int> result = Task.Factory.StartNew<int>((arg) => Convert.ToInt32(arg),input)
                .ContinueWith(antTask => StepA(antTask.Result))
                .ContinueWith(antTask => StepB(antTask.Result))
                .ContinueWith(antTask => StepC(antTask.Result));
            Console.WriteLine("Result " + result.Result.ToString());
        }
        private int StepA(int arg)
        {
            return arg + 2;
        }
        private int StepB(int arg)
        {
            return arg * 2;
        }
        private int StepC(int arg)
        {
            return arg - 1;
        }
    }

We can see the result is printed as  5 (First step convert the object 1 to int 1 then + 2 then *2 then -1).

Honor SRP - Moving actions to separate classes

In the above examples we can see that the steps are staying in one class. This violates the SRP(Single Responsibility Principle). There are more than one reason for that class to change. So re-factoring little bit to honor SRP by moving the steps to separate classes.

    class TPLTest
    {
        int state = 0;
        internal void Test()
        {
            Task.Factory.StartNew(()=>new Step1().Execute())
                .ContinueWith((task)=> new Step2().Execute())
                .ContinueWith((task => new Step3().Execute()));
        }
    }
    class Step1
    {
        internal void Execute()
        {
            Console.WriteLine("step 1 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 1 end.");
        }
    }
    class Step2
    {
        internal void Execute()
        {
            Console.WriteLine("step 2 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 2 end.");
        }
    }
    class Step3
    {
        internal void Execute()
        {
            Console.WriteLine("step 3 start.");
            Thread.Sleep(2000);
            Console.WriteLine("step 3 end.");
        }
    }

Now the step classes will change, only if there is change in the step logic and the main class will change only if there is change in the step order.

Now we have seem how the TPL can be used to fulfill the requirements of orchestration. Below are the pros and cons of using TPL for sequential execution.

Pros

  • Its inbuilt in .Net. We don't need to invent or refer other assembly.
  • The Task objects can be reused in case we want to run in parallel.

Cons

  • Difficult to organize each step in different class in type safe way as there is no base class for steps to inherit from.
  • Out of the box it does't have the support to accept typed context object into the steps. ie we need to convert the object into the context / state object in first step.
  • From the development team perspective, its little difficult to use if all needs to follow same orchestration method. We can see there is very little control in the framework. Look at the first 2 snippets. The Test() 7 TestInPipeline() method is implemented differently but works properly. So there are chances that, we can see 10 different implementations in same code base after 6 months if coded by 10 different developers.

Conclusion

This can be used if the development team is disciplined to follow same mechanism else we will end up in different methods to achieve orchestration. Also there is no object oriented support to organize or control the step class coding. I am continuing my research to find better library for orchestration.

Tuesday, December 23, 2014

Developing for developers - Designing and developing APIs

Couple of months backs I got a chance to be part of an initiative in our project, where we have to develop APIs for our server platform. The server platform is evolved with one of the project we are doing for one particular client. Now client want to extend that platform to be usable to their other projects. When we say its server platform, it basically manages servers, helps to provision new servers to scale horizontally, provide some platform data such as where some particular data is located. The data is expected to come in terabytes and it has some data storage mechanism which is multi-tenant. It really feels to me like we are reinventing the cloud and I tried to express here and there. Sometimes, I need to be quite, else who is going to pay me:-) Obviously there are reasons from other end also. There are many legacy code to be carried to this platform and client is having security concerns on hosting in the client. If we use Windows Azure, we might be ending up in limitation. Anyway its really interesting to be part of it especially when it is coming to API design and development.

My experience with API development starts with my first company which sells custom .Net libraries .There I was supporting .Net controls and fixing defects occasionally. At that time nobody told me about the importance of API development. But I realized myself, that API is something very important because we are developing for developers. If we didn't do it right, its very costly to change as all of our developers need to change their code as well. Think about the .Net library. Most of  the classes remains same for years. Now we cannot think about Microsoft renaming the MessageBox.Show() method to MessageBox.Alert(). I believe its same for other languages as well. I was really wondering in my early stages of carrier that, how they are coming up with these framework classes, its inheritance hierarchy, members and the access modifiers of each member. If something went to faulty, its very costly to recover.

In my second company, I got very little chance to work on critical APIs, but there were chances to code for their WPF custom control library. But in third and present company, there were no chance  still last 5 years. All I was doing is to help developers to fix their code which they written only to meet the release. Obviously there were some scope of drawing boxes as design. Most of the developers (including me sometimes because of pressure) were not programming, but gathering code to make a release.

When I got the new opportunity, I thought of doing some great things. I already targeted some  developers who are passionate in programming. But unfortunately, I got a completely different team who were gathering code for delivery as I mentioned above. After 2-3 weeks it came to notice that though they were great delivery guys, they are not much aware of SOLID principles or any other software engineering aspects of programming. It was really fun how I came to notice it. As the server platform components are already available in existing application,we started extracting those components. After extracting the code, we decided to re-factor it to look like a platform code.Class name ,Namespace changes etc... The problem started there.

As of my understanding, if somebody who already practiced SOLID principles, can re-factor any code by analyzing the technical aspects / code flow. But here, the team was struggling to re-factor and the reason they told was they don't know the business of application. The timers were really hard to understand for them. I was really wondering at this statement because I had re-factored big amount of code by understanding the logic by looking at the existing code. I had no clue what is the business requirement of those code or nobody gave me requirement documents. I think this is the case with all the service companies who are delivering from offshore. So arranged training on SOLID principles.

This leads to setting up one more training on API design. Or how to design APIs in a developer friendly and long lasting, backward compatible manner. For me too, it was really a learning experience. Got names for practices which we are already doing and of course obtained the experience of other API developers. Embedded the presentation below. Hope fully I can upload the video too.

References

http://wiki.qt.io/API_Design_Principles

Edits

9Oct2015 - Added references section.

Tuesday, December 9, 2014

Unit testing - How to verify dependent object's methods are called?

Testing void methods

In one of previous post, we saw what is mocking in unit testing. If the function which we are going to unit test is returning a value also expecting some value from the dependency which we mocked, its easy to test. But how to test a method which doesn't return a value, but successful execution calls a method in the dependency. In other sense we need to know whether a particular method in mock gets called or not to determine the test result.

So this post is aiming at how to test void methods which calls a particular method in dependency. Hope the purpose on why we should check whether a method in dependency is called or not is clear.

If its not clear, lets take one scenario. Sorry my project mates felt it difficult to understand so I am thinking there would be similar kind of developers out there. Others can skip the below story and look at the code.

Take the scenario of saving a Person object into database. There is a PersonRepository class with Save() method. It needs to validate some business rules such as the phone number and email. When we test the void PersonRepository.Save() by writing unit test cases, we must be focused only on the logic inside the Save(), not about saving into database. That will be done by IDataAccess which is a dependency of PersonRepository. We need to make sure that the Save() method is calling IDataAccess.InsertOrUpdate() upon success

Below is the code snippet which is ensuring that a method of dependency is called from production code upon success. It uses Moq library.

    public class PersonRepository
    {
        IDataAccess _dataAccess;
        public PersonRepository(IDataAccess dataAccess)
        {
            _dataAccess = dataAccess;
        }
        public void Save(Person person)
        {
            ThrowExceptionIfInvalid(person);
            _dataAccess.InsertOrUpdate(person);
        }
        private void ThrowExceptionIfInvalid(Person person)
        {
            if(string.IsNullOrEmpty(person.EMail))
            {
            throw new ArgumentNullException("e-Mail should not be empty");
            }
            //More code goes here to validate email & phone
        }
    }

Unit test code

        [TestMethod()]
        public void WhenEverythingIsOK_CallIDataAccessSave()
        {
            Mock<IDataAccess> mockDataAccess = new Mock<IDataAccess>();
 
            Person person = new Person() { EMail = "joymon@gmail.com" };
            PersonRepository repository = new PersonRepository(mockDataAccess.Object);
            repository.Save(person);//Test only the logic written inside Save() method.

            //IDataAccess.InserOrUpdate() will be called on success
            mockDataAccess.Verify(mock => mock.InsertOrUpdate(person), Times.Exactly(1));
        }

Happy testing.

Tuesday, December 2, 2014

WWF - Passing arguements to Custom Activity inside a Sequence container in coded workflow

Why workflow?

According to me Workflow is kind of DSL which helps to separate the business logic from input output operations. It provides a domain which contains the objects which are involved in the system and the business logic / rules which can be easily modified. Its very much useful when the business rules are very much dynamic and putting developers for modifying the logic for ever is not practical. One of the workflow framework is Windows Workflow Foundation. We will be using WWF in rest of this post.

Why Windows Workflow Foundation?

The big advantage of WWF is its tooling. Microsoft is providing good tooling experience in their Visual Studio and the amazing factor is that the designer can be packaged in to our WPF applications easily. In other words we can easily give our users a graphical way to alter the program flow. Its really more than giving some configuration screens or files.

The WWF uses XAML to store the workflow definition. ie When we create new workflow , open it in visual studio editor, add activities (steps) and save, we get a .xaml file. We can open it in notepad and see how the logic is captured in the form of xml.

The operations what we can do with editor can be done using code as well. 

Simple coded workflow using C# .Net

As we saw above, workflow is a DSL which has all most all the features of normal programming language. It supports variables as storage locations and arguments as input/output mechanism. Refer this link to see how simple it is to invoke an workflow which have only one step/activity.

We can connect one or more activities using Sequence Activity just like how we can do from VisualStudio. Below is one example where we are connecting more than one activity using Sequence activity when we create coded workflows.

WorkflowInvoker.Invoke(new Sequence
            {
                Activities =
                        {
                                        new WriteLine {Text = "Workflow"},
                                        new WriteLine {Text = "World"}
                        }
            });

Custom Activity

WWF is really extensible. Apart from using its built-in activities we can have our own activities as well which are knows as custom activities. Development of custom activities are as simple as inheriting from Activity class. Similarly invocation of custom activity is too simple as calling the WorkflowInvoker.Invoke() method by passing the object of custom activity. It can accept arguments as well like built in activity. One example below.


    public class MyParameterizedActivity : CodeActivity
    {
        [RequiredArgument]
        public InArgument<string> Arg1 { get; set; }
        
        protected override void CacheMetadata(CodeActivityMetadata metadata)
        {
            base.CacheMetadata(metadata);
        }
        protected override void Execute(CodeActivityContext context)
        {
            string arg = context.GetValue(Arg1);
            Console.WriteLine("Param received.Value :" + arg);
        }
    }

Code to invoke goes below

WorkflowInvoker.Invoke(new MyParameterizedActivity(), new Dictionary<string, object>()
            {
                {"Arg1","argument value"}
            });

Passing arguments into Custom Activity in a coded workflow

Whatever we saw above are some background things. The real intention of this post comes now. We saw about Sequence and custom activities. If we host this custom activity inside a sequence activity whether we will get the argument value easily into the custom activity?

Below is the easiest way people think of. But this will not work

            //The below will compile but won't run as there is no arguement for Sequence Activity
            //The error message will be "'MyParameterizedActivity': Value for a required activity argument 'Arg1' was not supplied."
            Activity act = new Sequence
            {
                Activities =
                {   new WriteLine() {Text="Test" },
                    new MyParameterizedActivity()
                    {
                    }
                },
            };
            WorkflowInvoker.Invoke(act, new Dictionary<string, object>()
            {
                    {"Arg1","arg1 value"}
            });

Basically here nobody is telling that which argument /variable needs to be mapped with the Arg1 of custom activity 'MyParameterizedActivity'. So how to overcome this?

Here comes the DynamicActivity.Using that we can specify the mapping. Below is the code snippet. Hope its not much complicated :)

// There is a Property in DynamicActivity and that holds the value.
// That value is passed to MyParameterizedActivity
            InArgument<string> InArg1 = new InArgument<string>();
            DynamicActivity<int> wf = new DynamicActivity<int>
            {
                Properties =
                {
                    new DynamicActivityProperty
                    {
                        Name = "DynamicActivityArg1",
                        Type = typeof(InArgument<string>),
                        Value = InArg1
                    },
                },
                Implementation = () => new Sequence
                {
                    Activities =
                    {
                        new WriteLine() {Text ="Writing from first console activity "},
                        new MyParameterizedActivity()
                        {
                            Arg1 = new InArgument<string>((env)=> InArg1.Get(env))
                        },
                        new WriteLine() {Text ="Writing from second console activity "}
                    }
                }
            };

            WorkflowInvoker.Invoke(wf, new Dictionary<string, object>
                {
                    { "DynamicActivityArg1", "Value for arg 1" },
                });

Why this much complication to orchestrate / sequence 2 activities. Its because the WWF don't have support to pass value from one activity to other activity. So we need to use extra variable in between.

Getting return values from custom activity hosted inside DynamicActivity

We saw the things done to pass one variable to custom activity if its hosted as a step inside another activity. Below is the code for creating a custom activity which returns a value and  hosted inside DynamicActivity.
                

// Custom activity class which just multiplies and stored the result
public class MyMultiplyActivity : CodeActivity
    {
        [RequiredArgument]
        public InArgument<int> Operand1 { get; set; }
        [RequiredArgument]
        public InArgument<int> Operand2 { get; set; }

        [RequiredArgument]   
        public OutArgument<int> Product { get; set; }

        protected override void Execute(CodeActivityContext context)
        {
            int a = Operand1.Get(context);
            int b = context.GetValue(Operand2);
            int c = a * b;
            Product.Set(context, c);
            //Sum.Set(context, a + b);
        }
    }

Below is the code to invoke and print the result.


            // Dynamic activity accepts 2 arguments and pass to custom activity
            InArgument<int> Operand1 = new InArgument<int>();
            InArgument<int> Operand2 = new InArgument<int>();
            DynamicActivity<int> wf = new DynamicActivity<int>
            {
                Properties =
                {
                    new DynamicActivityProperty
                    {
                        Name = "Operand1",
                        Type = typeof(InArgument<int>),
                        Value = Operand1
                    },
                    new DynamicActivityProperty
                    {
                        Name = "Operand2",
                        Type = typeof(InArgument<int>),
                        Value = Operand2
                    }
                },
                Implementation = () => new Sequence
                {
                    Activities =
                    {
                        new MyMultiplyActivity()
                        {
                            Operand1 =new InArgument<int>((env)=>Operand1.Get(env)),
                            Operand2=new InArgument<int>((env)=>Operand2.Get(env)),
                            Product=new ArgumentReference<int>{ ArgumentName = "Result" },
                        },
                        new WriteLine() {Text="Console activity after custom activity" }
                    }
                }
            };
            int result = WorkflowInvoker.Invoke<int>(wf, new Dictionary<string, object>
                {
                    { "Operand1", 25 },
                    { "Operand2", 15 }
                });
            Console.WriteLine(result);


The 'Result' is kind of built in argument name which is mapped to the returned value of Invoke() method

We can think about more and more scenarios such as how can we return 2 values from the custom activity. Not about returning as class object, instead what about having more than 1 OutArguments and how to handle them from the invoker side?

Another interesting area is to use scripts instead of typed coding. If we are using VB / C# scripts instead of typed code we can avoid some amount of code in mapping variables to activity parameters.