We all know that Azure and PowerShell are friends. This post discusses how can we use PowerShell to work with the data place of Azure Service Bus. The example here is about queueing a message into the Service Bus queue.
Problem
We did implement a messaging architecture using Azure Service Bus Queue. The consumer is a .Net app that connects to the Service Bus Queue using Service Principal + Certificate. It works fine in lower environments such as dev, QA, etc... but stopped working in higher environments. Below was the message.
"An existing connection was forcibly closed by the remote host ErrorCode: ConnectionReset (ServiceCommunicationProblem);"
As it is a higher environment, there is no way we can install troubleshooting tools or even don't have access to the Azure portal.
Troubleshooting
Since there is no portal access and cannot install tools we decided to simulate how the application consumes Service Bus using PowerShell.
The first step was easy to connect to Azure using cmdlets using the Service Principal+Certificate and generate JWT. But the next step was not easy.
By the way, this requires the installation of Az.Accounts PowerShell module in the production machine. :)
Since PowerShell is considered an admin thing than a dev tool, admins normally allow it.
Challenge
The problem is that there is no official PowerShell package from Microsoft to work with the data place of ServiceBus. The official PowerShell module supports control plane functions such as create service bus namespace(New-ServiceBusNamespace), create queues (New-AzServiceBusQueue), etc. Even interacting with the data plane is not there in Azure CLI. We were not able to find any community-built PowerShell cmdlets too except for some tutorials on how to send messages.
Solution
Finally, we had to get our hands dirty by making HTTP calls from PowerShell. The REST endpoints are properly documented to easily craft the requests. The PowerShell source code to send messages is available on GitHub.
https://github.com/ms-azure-demos/servicebus-queues-powershell
The root cause of the problem
When we run the PowerShell script from the same machine the app is running, it worked!!!. That eliminated the questions of what ports are open etc. The remaining difference between the .Net application sending messages v/s PowerShell was the AMQP v/s simple HTTP. .Net application uses AMQP by default and we didn't put an option in the application to fall back to HTTP. We checked whether the ports are open and they are. Hence concluded the higher environment network is not allowing AMQP traffic. Some network devices are on the way that doesn't like the AMQP traffic.
2 comments:
Interesting, What else does it require for AMQP communication other than Port Open.
Normally in enterprises, the port opening is fine. But in enterprises that work in regulated industries, it is not enough. There may be network devices in between that inspect the traffic going and may not like AMQP traffic. I personally encountered such a scenario but not sure what and how it was blocked. Just raising tickets to unknown teams solved it.
Post a Comment