Tuesday, September 21, 2021

Azure @ Enterprise - PowerShell to send message to Azure Service Bus Queue

We all know that Azure and PowerShell are friends. This post discusses how can we use PowerShell to work with the data place of Azure Service Bus. The example here is about queueing a message into the Service Bus queue.

Problem

We did implement a messaging architecture using Azure Service Bus Queue. The consumer is a .Net app that connects to the Service Bus Queue using Service Principal + Certificate. It works fine in lower environments such as dev, QA, etc... but stopped working in higher environments. Below was the message.

"An existing connection was forcibly closed by the remote host ErrorCode: ConnectionReset (ServiceCommunicationProblem);"

As it is a higher environment, there is no way we can install troubleshooting tools or even don't have access to the Azure portal.

Troubleshooting

Since there is no portal access and cannot install tools we decided to simulate how the application consumes Service Bus using PowerShell.

The first step was easy to connect to Azure using cmdlets using the Service Principal+Certificate and generate JWT. But the next step was not easy.

By the way, this requires the installation of Az.Accounts PowerShell module in the production machine. :)

Since PowerShell is considered an admin thing than a dev tool, admins normally allow it.

Challenge

The problem is that there is no official PowerShell package from Microsoft to work with the data place of ServiceBus. The official PowerShell module supports control plane functions such as create service bus namespace(New-ServiceBusNamespace), create queues (New-AzServiceBusQueue), etc. Even interacting with the data plane is not there in Azure CLI. We were not able to find any community-built PowerShell cmdlets too except for some tutorials on how to send messages.  

Solution

Finally, we had to get our hands dirty by making HTTP calls from PowerShell. The REST endpoints are properly documented to easily craft the requests. The PowerShell source code to send messages is available on GitHub.

https://github.com/ms-azure-demos/servicebus-queues-powershell

The root cause of the problem

When we run the PowerShell script from the same machine the app is running, it worked!!!. That eliminated the questions of what ports are open etc. The remaining difference between the .Net application sending messages v/s PowerShell was the AMQP v/s simple HTTP. .Net application uses AMQP by default and we didn't put an option in the application to fall back to HTTP. We checked whether the ports are open and they are. Hence concluded the higher environment network is not allowing AMQP traffic. Some network devices are on the way that doesn't like the AMQP traffic. 

2 comments:

Abdul Azeez Tk said...

Interesting, What else does it require for AMQP communication other than Port Open.

Joymon said...

Normally in enterprises, the port opening is fine. But in enterprises that work in regulated industries, it is not enough. There may be network devices in between that inspect the traffic going and may not like AMQP traffic. I personally encountered such a scenario but not sure what and how it was blocked. Just raising tickets to unknown teams solved it.