Tuesday, April 21, 2015

Determining health of system in distributed queueing system to dequeue next message

Recently we are in to the business of building a distributed system where the long running operations to be offloaded to processing machines. Processing machines are pulling the messages from the queue instead of having a manager allocating tasks. 

Initially we were assigning throttling numbers to each of the processing machines according to their configuration such as number of processors, RAM etc...But soon we could see that the machines are either not utilized or over utilized. So decided to introduce a mechanism where the real time machine load needs to considered before taking new message from the queue.

We considered some techniques for sensing the load but those were not feeling better than analyzing the Windows performance counters. So decided to go with looking at the appropriate performance counters before dequeue a new message from the queue.

What are the appropriate performance counters? Its always debatable. We are in the initial stages of the implementation. Hopefully can update soon.

Another challenge we faced was how the performance counter data is translated to a boolean value saying system is healthy or not. We evaluated PAL which reads the .blg files and produces report. But finally reached to a conclusion of saving the performance counter values into database and running own rule engine there which replaces PAL.

Some links on how to work with performance counters from perfmon.exe below

https://technet.microsoft.com/en-us/library/cc722414.aspx
http://www.windowsnetworking.com/articles-tutorials/netgeneral/Scripted-Networt-Defense-Part2.html

Tuesday, March 17, 2015

Starting Scala & R Language

Its always good to learn at least one new programming language per year to sustain in software engineering industry as architect or in any development position. At least need to know what are the capabilities and some hands on experience on the new language we are learning every year.

Last year I was with PowerShell. It is a nice language. Less code more things. That is the beauty if it. Also piping is a great feature. Seamless integration with .Net programs etc...

This year I decided to learn something not much related to .Net. One is Scala and other is R language.

Why interested in Scala & R as a hard core .Net guy?

Let us see some code fragments in these languages.

//Scala
  def OddOrEven(a: Int) = {
    if (a % 2 == 0) "Even"
    else "Odd"
  }

There is not more things interesting here except the absence of return statement. Yes the language now can understand programmer's intention and can remove strict rules.

This is just a start. There are more things out there in scala which makes programming faster with less code.

After working in delivery oriented organization which maintains huge amount of legacy code, I feel the way to survive in intense delivery environment is to stick to the below principle

"Less code less defects"

Some of the features are already available in C# in the form of lambda expressions, automatic properties etc...But Scala seems more simplified and takes advantage of great JVM.

# R Program to increment all members in list
a=c(10,12,13)
a=a+1
print(a)

The same thoughts here. Less code as well as R is very good for statistical programming.

There are many more reasons why these languages are becoming popular. Just google for more features.

Setting up the environment for these 2 languages is very easy. Next time I will post it.

Tuesday, March 10, 2015

Using ref parameters in WCF service contract

Actually it didn't make sense to have ref parameter in service calls as most of the cases service runs in different process and client don't get reference to the server memory address. But still WCF supports it.

I never thought it will work until I saw this in one of the production applications. We were re-factoring a legacy code base and could see one operation contract with ref keyword. Initially we thought that service itself is not in use and can delete it. But my colleague was sure that it works. So decided to do a sample as running the app and finding out which functionality in that giant app is going to use it takes more time. Below is the PoC we did.

WCF Service contract

Created service contract with 3 operation contracts

    [ServiceContract]
public interface IMathServiceWithRef
{
    [OperationContract]
    int GetSum(int number1, int number2);
    [OperationContract]
    Result GetSumAndAverageAsResultObject(int number1, int number2);
    [OperationContract]
    int GetSumAndAverage(int number1, int number2, ref int average);
}
[DataContract]
public class Result
{
    [DataMember]
    public int Sum { getset; }
    [DataMember]
    public int Average{ getset; }
}

WCF Service Implementation

Simple implementation as below

public class MathServiceWithRef : IMathServiceWithRef        
{
    int IMathServiceWithRef.GetSum(int number1, int number2)
    {
        return number1 + number2;
    }
    Result IMathServiceWithRef.GetSumAndAverageAsResultObject(int number1, int number2)
    {
        return new Result() { Sum = number1 + number2, Average = (number1 + number2) / 2 };
    }
    int IMathServiceWithRef.GetSumAndAverage(int number1, int number2,ref int average)
    {
        average = (number1 + number2) / 2;
        return number1 + number2;
    }
}

WCF Client proxy & Test class

Right clicked on the service reference section and added using the svc url. Used simple test project which already have service reference

        [TestMethod()]
public void WithNormalValues_ShouldSuccess()
{
    WCFService1.MathServiceWithRefClient client = new WCFService1.MathServiceWithRefClient();
    int average=0,sum = 0;
    //////////////////////////////////////////
    sum = client.GetSum(20, 10);
    Assert.IsTrue(sum == 30);
    //////////////////////////////////////////
    Result result = client.GetSumAndAverageAsResultObject(20, 10);
    Assert.IsTrue(result.Average == 15 && result.Sum == 30);
    /////////////////////////////////////////
    sum=client.GetSumAndAverage(20, 10, ref average);
    Assert.IsTrue(average == 15 && sum==30);            
}

And this worked as expected. Fiddler results below

Fiddler results

Simple Sum() call

Request XML
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Body>
    <GetSum xmlns="http://tempuri.org/">
      <number1>20</number1>
      <number2>10</number2>
    </GetSum>
  </s:Body>
</s:Envelope>
Response XML
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Body>
    <GetSumResponse xmlns="http://tempuri.org/">
      <GetSumResult>30</GetSumResult>
    </GetSumResponse>
  </s:Body>
</s:Envelope>

With ref keyword

Request XML
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Body>
    <GetSumAndAverage xmlns="http://tempuri.org/">
      <number1>20</number1>
      <number2>10</number2>
      <average>0</average>
    </GetSumAndAverage>
  </s:Body>
</s:Envelope>
ResponseXML
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
      <GetSumAndAverageResponse xmlns="http://tempuri.org/">
        <GetSumAndAverageResult>30</GetSumAndAverageResult>
        <average>15</average>
      </GetSumAndAverageResponse>
    </s:Body>
  </s:Envelope>

As we can see in the fiddler the response xml just contains the ref value. But not as result. 

WCF communication protocols tested

Obviously there is no need to test using various protocols as WCF is more towards protocol independent in all most all the cases.

Still we tested in net.pipe and net.tcp protocols and it worked.

Why someone should use ref keyword in WCF operation contact

If someone is really lazy to create a DataContract class they can use this method. I don't see any other benefits.

Happy coding. 

Tuesday, March 3, 2015

Is my code running inside virtual machine?

This is a small tip which rarely, we as programmers need to apply. Its about how to determine whether our code is running inside a virtual machine or not.Actually it doesn't matter if we are writing business applications. But in some cases especially when we are dealing with hardware devices we may need to disable some features by checking whether the code is under virtual machine. Also this is not a first thing any programmer is looking for. So I am aggregating links as well.

Native C++

Hope everyone will be aware of this as people who are in unmanaged world are talking directly to the computer.

.Net (C# or VB.Net)

In .Net the easiest way if to query the WMI objects. We can get easily get example code snippets from internet. Some are below.

http://stackoverflow.com/questions/498371/how-to-detect-if-my-application-is-running-in-a-virtual-machine

SQL

Query the DMV sys.dm_os_sys_info as follows

SELECT virtual_machine_type,virtual_machine_type_desc 
FROM sys.dm_os_sys_info

The sys.dm_os_sys_info gives much more information about the host system such as last SQL Server restart time, memory information etc...

There are different virtualization softwares available to create virtual machines. The method to determine that detail may vary from type to type. Need to invest some time to determine that.

Tuesday, February 24, 2015

Group by test class name in VS2012 Test Explorer

Background

Writing unit test case is always a good practice in programming. It gives confidence to re-factor code and make sure each and every piece is working properly. If individual pieces are working good mostly the application will work after integration.

Basics of Visual studio unit test framework

Visual Studio has out of the box unit testing framework. We can create test classes and those classes can contain test methods. Normally if we follow proper naming convention to relate production code with test projects, it is easy to see how the tests are performing. One of the naming convention for unit test class and methods names can be as follows
  • Let the test project name be same as production project but suffix the test project with .Tests
  • For each production code method, create a unit test class.
  • Each scenario in the production method can be one test method in test class.
  • Test class name can be <Scenario>_<Production Method Name>{}
  • Method name can be <Test case specification>_<Expected behavior>()
eg: Production code

    public static class ObjectFactory
    {
        public static object CreateObject(string fullyQualifiedClassAndAssemblyName, object[] constructorParameters)
        {
//Production code which uses reflection to create object.
        }
    }

Unit test class

[TestClassExcludeFromCodeCoverage]
public class ObjectFactory_CreateObject
{
        [TestMethodTimeout(1000)]
        public void WhenFullyQualifiedClassAndAssemblyNamePassedAndClassIsInternal_ShouldSucceed()
        {
//Test code
        }
        [TestMethod,ExpectedException(typeof(FileNotFoundException))]
        public void WhenFullyQualifiedClassAndNonExistingAssemblyNamePassed_ThrowFileNotFoundException()
        {
//Test code
        }
}

The main benefit of this naming convention is, it does not need any additional documentation to describe the what the test does. When we avoid one more item from maintenance, we are increasing productivity.

Running test cases from visual studio

We can easily trigger the unit test execution and see the results in "Test Results" window in VS 2010. The results will be shown with test name by default. The same functionality is there in VS2012 as well, but the window name is changed from "Test Results" to "Test explorer".

Problem

When the project grows, there are high chance that programmers forget to write test cases. Or even if there is strict enforcement, they will tend to write one test method like below for each and every production method.

[TestClass]
public class EmailSender_Send
{
        [TestMethod]
        public void WithProperparameter_ShouldSucceed()
        {
//Test code
        }
}
[TestClass]
public class Scheduler_CreateSchedule
{
        [TestMethod(), Timeout(30000)]
        public void WithProperParameters_ShouldSuccessed()
        {
//Test code
        }
}

Normally the test results will be displayed only with the test name and the test name is the test method name. That ends up in the test result window with all tests with same name. Its very difficult to understand from the review perspective.

If we want to see the test class name we can group by the test class name as below. This works well in VS2010.
[Photo taken from http://www.ladislavmrnka.com/wp-content/uploads/2012/09/TestView.png]

The same functionality in VS2012 named Test Explorer doesn't have Group by Class name in its earlier production releases. The group by popup is shown as below when try to group by class name.
This is really useless for somebody who want to see the test results. There is no idea why Microsoft removed a useful feature from earlier version in new release. But fortunately its back in later VS2012 updates. When this post is written, the latest update was Update 4.

Happy coding. Yes unit testing is part of coding:)

Tuesday, February 17, 2015

Standalone tools for production debugging

Below are some tools which are standalone and can help in production debugging. Don't confuse this works if we as developers have access to production servers and able to run standalone tools but don't have rights to install applications.

If we have rights to install applications in production server, we can use better tools than these such as Visual studio
  • .Net tools
    • Perfview - to catch exceptions
      • This is useful if we suspect that there are some exceptions eaten in the application and those are causing some other side effects.
  • Database
This is list is growing list. Hopefully I can add more later.


Tuesday, February 10, 2015

WCF DataServices - Date Time comparison in OData query

The concept of WCF data service which uses OData is really great. But I doubt whether that is ready to be used in an enterprise production applications. The main reason is its immaturity and lack of support on some Linq queries and operators. One of the major item is lack of joins. I believe OData specification itself doesn't have join. Below is another scenario which is not supported by WCF Data Services.

Requirement - Compare date in database against current date

Lets take and scenario where a MessageProcessAttempt table is exposed via WCF Data service. There is a field called DequeuedTime which tells when a message is dequeued. We need to list down all the messages which dequeued in last 1 hour.

In normal sense we can obtain the list by subtracting the de-queued time from current time and checking if the difference < 1 hour. The query can be as follows.

List<MessageProcessingAttempt> msgs = (from msg
                           in qProxy.MessageProcessingAttempts
                           where (DateTime.UtcNow- msg.DequeuedTime.Value).TotalHours<1
                           select msg).ToList();

But it will fail with below error message
"No property 'TotalHours' exists in type 'Edm.DateTime' at position 56."
Position will change based on the code.There will be no compile time errors. It just fails at runtime. Tried to find the details about Edm.DateTime class in msdn but able to find out except one EdmDateTimeConstant class.So what can we do

Solution - Calculate target date and compare

Find the target date and do comparison rather than finding the difference in the OData query.


DateTime timeToFilter = DateTime.UtcNow.Subtract(TimeSpan.FromHours(1));
List<MessageProcessingAttempt> msgs = (from msg
                           in qProxy.MessageProcessingAttempts
                           where msg.DequeuedTime.Value >= timeToFilter
                           select msg).ToList();

Here the major difference we can notice that the date subtraction is not happening via operator and its not producing the TimeSpan obejct. We can have DateTime.UtcNow.Subtract(TimeSpan.FromHours(1)) in OData as well instead of variable and it will work. This points to the fact that TotalHours property is not there in a special type called Edm.TotalHours which is produced as part of subtraction operator.

This was found by our development team in India and I didn't agree to them in the beginning. I spend around 2-3 hours and could see WCF Data Service is still not suitable for enterprise apps. Thanks to the team for finding the workaround.

Happy coding...