Service Bus for Windows Server - Getting started

Background

Recently we got a big web project which will be public facing unlike most of the other projects. The security becomes really important as its dealing with students and as always students are more brilliant in hacking than the professional hackers. So it is decided to isolate the modules even physically at the DB level itself. Wow...Yes it really means the database machines of each module can be switched off when that module is not in use.

The architecture is really nice. But that doesn't really solve all the problems. The modules share some data among them. Since the database or each module is different, some tables or columns needs to be replicated in to other databases, if corresponding module needs same data. It raised the old famous question about syncing.

How do we sync the data between modules as they are physically separate and even can be switched off individually?

I am not directly part of the project but whenever I am in reviews of the project, and asks about the syncing, I hear same answer. "It will be done by the service bus."

What is this service bus? Are you referring to the same Windows Azure Service bus? Are you referring to any back end database technology which will sync the data? Or is that any magic software which will read the developers mind and copy the data from selected table columns of one database and copy to another table in a different database? Almost every time there will be a big silence.

I had tried Windows Azure service bus in the early days of Azure and its mainly for integrating independent systems using messaging and of course we can leverage messaging for syncing data.

When we develop huge systems especially for web like FB or Google, we cannot develop it as single application. It must be split into different independent sub systems and they should work together to serve the customer needs. To achieve smooth integration and communication among these sub systems there should be a strong messaging mechanism. Service Bus for windows is such a strong messaging mechanism.

If we take a logging module which is an essential feature of big systems, it should be built as independent system. So that all the other modules such as front end web servers and back end processing servers can use this logging without much overheads. To get a better view of importance of subsystems, think in the other way where the logging is integrated in web server. In such a situation web server needs to prepare the html page response as it's primary duty after that, it needs to log. The logging taskcan be split into 2. Prepare the log message and write the log. Obviously the time to prepare the log message cannot be avoided as web server is the only one which knows what it did and what to log. But what about the time to write log. If the web server tries to open a file or DB connection and write itself, it will add some time to the overall turn around time of web request. Instead of that what about just publishing a log message to the service bus, so that it can be processed later by the logging sub system? To develop such a messaging system, first let us look at how to get a hands on experience with Service Bus for Windows server

What is ServiceBus

There are already so many definitions available. The general architecture term for these kind of messaging systems is  Enterprise Service Bus. There are commercial as well as open source implementations of ESB such as NServiceBus, Oracle ESB etc...Microsoft implemented the concept of ESB as 'Windows Azure Service Bus' and they released the same product to stand alone servers (can be an Azure VM or privately hosted) named 'Service Bus for Windows Server'. Are the Service Bus implementations for Azure and Windows Server same? No. You can check the comparison here.

Service Bus for Windows uses SQL Server for keeping the messages and other contextual information.

Installing Service Bus for Windows Server in Windows 7 development machine

Service Bus has 2 parts .One is the infrastructure and services and next is the APIs to interact with the Service Bus. The current version is 1.0. You may download both from 

http://www.microsoft.com/en-in/download/confirmation.aspx?id=35374

This contains 3 files as of now. We are interested in 'ServiceBus.exe'. Install by double clicking. The installer uses web plat form installer so that it will install all the dependencies as well. You may avoid the above download by using web plat form installer directly for installing service bus.

Successful installation mostly asks for windows restart.After restarting you can see the below folder created.

<Install drive>:\Program Files\Service Bus\1.0 - This the path to service bus APIs dlls.

Also you can locate so many .sql files as well. Those are required to configure the SQL databases needed
for Service Bus operations.

Configuring Service Bus

Next task is configuring the service bus. For that either you can navigate via start menu and start "Service Bus Configuration" program or directly execute the exe located at 

<install drive>:\Program Files\Service Bus\1.0\Microsoft.ServiceBus.ConfigWizard.exe

This will ask for some basic things about Service Bus farm. Those mainly include your windows domain password, a security key etc...It will create the service bus farm and add the computer to that then create the service namespace etc...Once you configure the farm you can see a dialog which tells your computer is already in the farm along with a button for leaving the farm.

Better copy the details and keep in a text file .The details looks like below

---

Starting

Created and configured Service Bus farm management database.

Created and configured Service Bus gateway database.

Creating default container.

Microsoft.ServiceBus.Commands.SBFarmInfo

Processing completed

Validating input and configuration parameters.

Installing auto-generated certificate.

Granting 'Log on as Service' privilege to the run as account.

Windows Fabric configuration started.

Running Windows Fabric deployment.

Windows Fabric starting.

Service Bus configuration started.

Updating database.

Service Bus services starting.

Updating local registry.

Successfully added this host to the farm.

Microsoft.ServiceBus.Commands.SBFarmInfo

Processing completed

Name: ServiceBusDefaultNamespace

AddressingScheme: Path

CreatedTime: 6/21/2013 1:00:07 PM

IssuerName: ServiceBusDefaultNamespace

IssuerUri: ServiceBusDefaultNamespace

ManageUsers: <**username**>@<**domain**>

DnsEntry:

PrimarySymmetricKey: <**encryped key ending =**>

SecondarySymmetricKey:

Processing completed

Endpoint=sb://<**computer name**>.<**domain**>/ServiceBusDefaultNamespace;StsEndpoint=https://<**computer name**>.<**domain**>:9355/ServiceBusDefaultNamespace;RuntimePort=9354;ManagementPort=9355 

Processing completed

---

First Service Bus Application

The first application which we are going to see is about messaging in topics and subscriptions world. There are always 2 ways to have our first application. Obviously first one is create ourselves from scratch. Next download applications of others which can be sophisticated or their first application itself. I selected to take existing application from service bus samples project in codeplex. Downloaded the samples zip and started with ..\CSharp\GettingStarted\BrokeredMessaging\MessagingWithTopics on the assumption that I can change the sample easily to fit to the stand alone machine environment.

Error / Challenge - 1 - Terminologies

I would like to call it as challenge rather than error. The Service Bus introduces so many terminologies. Some of those are already existing and related to other technologies but some are specific to service bus.

STS - This is a general term which deals with the service which serves security tokens

Namespace - Groups the service bus features. eg: one namespace can have queues and topics.

Topic - This is service bus specific term.Refers to the category of messages. Messages will be send using this topic.

Subscription - This is a virtual queue create on topic. Copy of each message will be available in all subscriptions of topic.*

Message - The atomic communication packet in the Service Bus which is assiciated to topic.

Error / Challenge - 2 - Service Bus for windows still refer Windows Azure

Got this error when running the sample after changing the ServiceNamespace and IssuerName  

The token provider was unable to provide a security token while accessing 'https://servicebusdefaultnamespace-sb.accesscontrol.windows.net/WRAPv0.9/'. Token provider returned message: 'Error:Code:401:SubCode:T0:Detail:ACS50009:

Seems like there are still some references to Azure existing in the Service Bus code. When MSFT brought the same framework to standalone Windows Servers they might forgot this. 

I changed the code as follows.

Uri serviceUri = ServiceBusEnvironment.CreateServiceUri("sb", ServiceNamespace,"<full machinename>");

But this doesn't solve the issue .Still it refers Azure. The Uri create became "sb://servicebusdefaultnamespace.servicebus.windows.net/<full machine name>/"

---

Then I went to the extent of hard coding

Uri serviceUri = new Uri(@"sb://servicebusdefaultnamespace.<full machine name>/");

Now interesting error 

"uri provided servicebusdefaultnamespace.<full machine name> does not match Service Bus domain: servicebus.windows.net."

Oh its still in love with Azure. By default its pointing to windows azure namespace. 

---

As of my understanding there is no coding / API ways to redirect to our Service Bus installation. So relay on the config way. But before that I did a reflection of the Microsoft.ServiceBus.dll to find out the hard coding and I succed in by exploring the constructor of Microsoft.ServiceBus.RelayEnvironment.ConfigSettings class. There the name of config file is hardcoded to servicebus.config. Inspecting the ReadConfigSettings() method in same ConfigSerrings class give insight to the location where we need to put the servicebus.config file.

The file should be either at the application root folder or in the config folder of .Net framework. Obviously this was identified by others and posted in their blogs. My bad google skills. The location is %Windir%\Microsoft.NET\Framework[64]\v4.0.30319\Config

Error / Challenge - 3 - Authentication and token issues

You will get so many network related issues if you try to execute the sample further. Some are below.

"The token provider was unable to provide a security token while accessing 'https://servicebusdefaultnamespace-sb.<full machine name>/WRAPv0.9/'. Token provider returned message: 'The remote name could not be resolved: 'servicebusdefaultnamespace-sb.<full machine name>'.

Where from that <namespace>-sb came? No idea.

---

You can try giving the STS Url to the token provider. 

TokenProvider tokenProvider1 = 

                TokenProvider.CreateSharedSecretTokenProvider(IssuerName, IssuerKey,new Uri(@"https://<full machine name>:9355/ServiceBusDefaultNamespace/"));

But then you will get

"The token provider service was not avaliable when obtaining a token for 'https://<machine name:9355/ServiceBusDefaultNamespace/WRAPv0.9/"

What is this WRAP?

---

If you are not familiar, you can try with WindowsTokenProvider as follows

TokenProvider tokenProvider1 = 
                TokenProvider.CreateWindowsTokenProvider(new List<Uri>() { new Uri(@"https://<full machine name>:9355/ServiceBusDefaultNamespace") });
            

Which will end up as

"The token provider was unable to provide a security token while accessing 'https://<full machine name>:9355/ServiceBusDefaultNamespace/$STS/Windows/'. Token provider returned message: '<Error><Code>400</Code><Detail>Cannot resolve a namespace for scope: http://servicebusdefaultnamespace.<full machine name>/IssueTrackingTopic/"

---

All these issues / exceptions will be raising when the execute the below statement.

if (namespaceManager1.TopicExists("IssueTrackingTopic"))

So better follow the below way which I collected by googling.

---

string stsUri = string.Format("sb://{0}.{1}:{2}/", Environment.MachineName, domain,port);
string namespaceUri = string.Format("https://{0}.{1}:{2}/{3}/", machineName, domain,port, ServiceNamespace);
 
TokenProvider tokenProvider = TokenProvider.CreateWindowsTokenProvider(new List<Uri>() { new Uri(stsUri) });
NamespaceManager namespaceManager = new NamespaceManager(namespaceUri, tokenProvider);
            
            // Delete if exists
if (namespaceManager.TopicExists("IssueTrackingTopic"))
            {
                namespaceManager.DeleteTopic("IssueTrackingTopic");
            }

Now its working till the namespace manager checking for topic. But if you debug further it will fail at another point which is nothing but during message sending

                myTopicClient.Send(message);

Error message will be
"The token provider was unable to provide a security token while accessing 'https://<full machine name>:9355/$STS/Windows/'. Token provider returned message: '<Error><Code>400</Code><Detail>Cannot resolve a namespace for scope: http://servicebusdefaultnamespace.<full machine name>/IssueTrackingTopic"


This is because the address which we gave during MessagingFactory creation is wrong. 


Correct as below.


string messagingUri = string.Format("sb://{0}.{1}/{2}/", machineName, domain, ServiceNamespace);
MessagingFactory factory = MessagingFactory.Create(messagingUri, tokenProvider); 

During this long exercise, at a bad moment I though of turning off the network connectivity. From then I started getting another error message 

The token provider was unable to provide a security token while accessing 'https://<full machine name>:9355/$STS/Windows/'. Token provider returned message: '<Error><Code>401</Code><Detail>The security token service cannot translate sid 'S-1-5-*1-3**18*34*9-23***35**9-21*****38-8**0' associated to user '<domain>\<user name>'

This because I am using my company laptop and my login is in the company DNS (companydomain\username). This shows that the Service Bus needs constant connection to the domain controller as well.

Service Bus sample walk through

The modified sample can be downloaded from sky drive.
 

The main assembly reference needed for this application is Microsoft.ServiceBus. The location of Microsoft.ServiceBus.dll is as follows.

<Install drive>:\Program Files\Service Bus\1.0\Microsoft.ServiceBus.dll

As I told this is a modified version of ..\CSharp\GettingStarted\BrokeredMessaging\MessagingWithTopics sample which is available in codeplex. This basically has 2 exes.

Working

First one sender.exe, will create a 'Topic' in the Service Bus installed locally. Then add 2 Subscriptions  and send 3 messages. Wait for key stroke. Upon key stroke, it will delete the created topic along with messages. Which means you need to run the receiver.exe when sender is waiting for key stroke.

Second the receiver.exe listen for the messages using 2 subscriptions in the same topic. One subscription just reads the message and other read and delete messages. Messages are same. That's why if we run the receiver again, it will not receive the messages.

Both the exes can be found in ..\binaries folder

Sample Configuration

Main thing you need to give the proper service name space. The default is - ServiceBusDefaultNamespace.

Next is the Port number. The default is 9355.

The windows domain user account, I used is same as my login. So it will automatically take the credentials.If you are using different token provider you need to change the code. So it is advised to run the sample in the same Windows 7 machine where the service bus is configured

Also this sample doesn't cover the scenarios where the Service Bus is installed in other server machine and different clients accessing from different domains with different authentication mechanisms.

Trouble shooting and monitoring Service Bus

As you seen above, the environment is really hard to maintain. There may be situations where the message may disappear or vanishing of subscriptions or topics etc... So its always good to have a general service bus explorer application to monitor the health of the system. Luckily there are 2 tools available to explore the service bus and they are.

Visual Studio integrated service bus explorer

Service Bus explorer - Download

References

* http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/
http://msdn.microsoft.com/en-us/library/windowsazure/jj193021(v=azure.10).aspx

http://msdn.microsoft.com/en-us/library/windowsazure/jj193022(v=azure.10).aspx

http://msdn.microsoft.com/en-us/library/windowsazure/jj193027(v=azure.10).aspx

http://msdn.microsoft.com/en-us/library/windowsazure/jj860550.aspx

http://blogs.msdn.com/b/windowsazure/archive/2012/10/24/announcing-public-availability-of-service-bus-for-windows-server-service-bus-1-0.aspx

http://blogs.neudesic.com/post/2012/07/18/Introducing-Service-Bus-for-Windows-Server-(Service-Bus-10-Beta).aspx

http://www.codeproject.com/Articles/276400/Using-Windows-Azure-Service-Bus-Messaging

http://soa-thoughts.blogspot.in/2012/06/visual-studio-service-bus-explorer.html

http://social.msdn.microsoft.com/Forums/windowsazure/en-US/af6da122-e766-4f1e-bd1e-507126a890e8/service-bus-security-101