Tuesday, March 2, 2021

Azure @ Enterprise - Managed Identity formerly MSI to bulk insert from Azure Data Lake Gen2 (ADLS Gen2) to Azure SQL Database

Requirement

Bulk insert a CSV format file from Azure Data lake Gen2 to Azure SQL using the system-assigned managed identity (Managed Service Identity) as the authentication mechanism.

Official link

As per the Microsoft link, it is not supported. The most secure way seems the SAS token-based database scoped credential.

BULK INSERT and BACKUP/RESTORE statements cannot use Managed Identity to access Azure storage


Verifying the above

The first step is to create a system-assigned managed identity to the Azure SQL Server. As of writing this post, it seems there is no way to do this from the Azure portal unless we use PowerShell. Below goes the step.

The command is az sql server update -g <resource group> -s <sql server name> -i

The next step is to create Azure Data lake Gen 2 account. How to create that is skipped in this post as that is already available on the internet. The next step is to give the required permissions to the SQL Server's identity to access the Azure Data lake Gen 2 account.
The permissions can be more restricted based on the use case. Now lets us see how the SQL scripts look like.

Tuesday, February 23, 2021

[Video] Azure @ Enterprise - ROPC flow in Azure AD

Here is the next video about authentication using Microsoft Identity Platform formerly Azure Active Directory Developer Platform.

Here we discuss ROPC flow in OAuth that is not recommended but a use case for it. The sample uses C# .Net


Additional references

https://medium.com/@dany74q/service-to-service-auth-with-azure-ad-msi-oauth-2-0-step-by-step-a1aed196b1e1

https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-authentication-flows

https://docs.microsoft.com/en-us/azure/active-directory-b2c/add-ropc-policy?tabs=app-reg-ga&pivots=b2c-user-flow

Tuesday, February 16, 2021

Azure @ Enterprise - Dilemma of choosing .Net SDK for ServiceBus Queue

If you want to skip all my stories and want to understand what is the .Net SDK to use with Azure ServiceBus, please use the Azure.Messaging.ServiceBus nuget package. No more suspense.

Others welcome to the story. 

All we wanted was to dequeue Azure ServiceBus Queue messages in the synchronous fashion inside the .Net 4.6.1 application. 

WindowsAzure.ServiceBus Library and Microsoft.ServiceBus.Messaging namespace

The first nuget library we got was WindowsAzure.ServiceBus which has Microsoft.ServiceBus.Messaging. This library has a class called QueueClient. That has a synchronous dequeue mechanism using ReceiveBatchAsync() method. Unfortunately, this library needs .Net 4.6.2. If it is not an enterprise, upgrading the .Net framework version would not be a big deal. But upgrading an application having half a million lines of code in the enterprise that too working in regulated industry is a thing. So had to say bye to this technique.

Another reason to say bye is the new library named Microsoft.Azure.ServiceBus and the migration instructions from WindowsAzure.ServiceBus.

Microsoft.Azure.ServiceBus Library and Microsoft.Azure.ServiceBus namespace

This library has migration guidelines from the community at least. It appeared to be the way to go. It supports .Net 4.6.1. This also has a QueueClient class. But this QueueClient doesn't have a mechanism to synchronously dequeue messages. It accepts a callback via RegisterMessageHandler() and suitable for daemon apps. Our app is not a daemon.

We got stuck here and since we had a channel to MSFT we decided to activate it. They suggested a third library which comes next.

Later we figured out there is a way in Microsoft.Azure.ServiceBus library to dequeue messages in sync way. This link explains the same using MessageReceiver class. But we are planning to use the below one which Microsoft confirms the way forward.

Azure.Messaging.ServiceBus

As per Microsoft, this library is the way to go. They are making all SDK names starting with Azure prefix now. The good thing we are seeing is that now Microsoft is preparing the migration guide from older SDKs.

Migration into Azure.Messaging.ServiceBus

From WindowsAzure.ServiceBus to Azure.Messaging.ServiceBus migration guide issue. Note that this is not yet available. At least there is an issue to track it.

Samples

The below link will take us to the code to dequeue messages in sync using Azure.Messaging.ServiceBus library.

https://github.com/joymon/azure-demos/blob/master/intergation/service-bus/service-bus-queue-basic/AzureMessagingServiceBusDequeuer.cs

It also has the code snippets we tested using other libraries. Get the full project to run it.

Conclusion

Those libraries are still in development. That is the interesting thing. 

LibraryLast release
WindowsAzure.Messagingv6.0.3 - 5 months ago
Microsoft.Azure.ServiceBusv5.1.1 - 24 days ago
Azure.Messaging.ServiceBusv7.0.1 - 25 days ago

The above table shows the recent releases as of writing this blog post.

Due to NDA, I am not supposed to disclose the names. But big thanks MSFT guy(s) for your support. Innovation and changes are always welcome from Microsoft. But one humble request is to slow down innovation a little bit and consider stability. That helps people like us who develop apps for the regulated enterprises.

Happy coding...