Azure @ Enterprise - App Service Deployment Error: ENOENT: no such file or directory, open

I was fiddling with deployment via the Kudo console in the Azure app service. Removed one file as part of testing. The expectation was that the next deployment would put the file back.

The next deployment from GitHub actions failed with the below message.

33      Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/5b7989fd-9794-4ed0-8bfc-dbf1f18c686b.zip (4.31 MB) to /tmp/zipdeploy/extracted

34

Error: Failed to deploy web package to App Service.

35

Error: Deployment Failed, Package deployment using ZIP Deploy failed. Refer logs for more details.

36

App Service Application URL: https://<app service web app name>.azurewebsites.net

test

No more details on failure in GH Actions logs. There was an error as shown below in the Azure AppService Deployment center.

Error: ENOENT: no such file or directory, open '/home/site/wwwroot/Azure.Core.dll

That was the exact file I removed earlier.

Debugging

Hypothesis 1

The file may be open in a Kudu console, hence unable to access it. 

But even after closing, the error persisted.

Hypothesis 2

The App Service deployment action in GHActions requires the file to be present there as that may be keeping track of files.

Even after manually uploading the file, the issue persists

Solution

Finally attempted the Microsoft-certified solution of Restarting the App service. It worked.

The order is to stop the App service, do the deployment, and start again.

Azure @ Enterprise - Upload certificate to App registration using PowerShell

Warning

All characters and events in this post are fictional. Any resemblance or similarity to any actual events entities or persons is entirely coincidental

Developer: Hi there, as you know we are excited to release the next version that uses Azure AD-based authentication. All the 100 development, testing till prod deployment instances require their own Azure app registrations. Also, require certificates added for authentication to outside services. Certificates can be shared across lower environments.

Install Manager: Oh. It is not possible to have one app registration per application instance. It is not manageable.

Developer: What? Are you not paid for managing instances? Hope you are not doing charity.

Install Manager: Yeah I know you are funny. We cannot manage that many app registrations and certificates.

Developer: What do you mean by managing? 

Install Manager: Oh boy, higher environments are not like your toy dev instances. It is serious business. Do you know the certificates are not forever? On that day they expire, we need to upload a new certificate to all your dev and test app registrations.

Developer: Hey, we are in 2023. Don't you guys automate?

Install Manager: We don't do automation here.

Developer: It is not huge code. Just use the simple PowerShell commands that any system admin knows.

Install Manager: Enough boy. We don't have skilled people here. If you want, you code it and give it to us.

Developer: Oh ok..here you go.

Azure @ Enterprise - PowerShell to send message to Azure Service Bus Queue

We all know that Azure and PowerShell are friends. This post discusses how can we use PowerShell to work with the data place of Azure Service Bus. The example here is about queueing a message into the Service Bus queue.

Problem

We did implement a messaging architecture using Azure Service Bus Queue. The consumer is a .Net app that connects to the Service Bus Queue using Service Principal + Certificate. It works fine in lower environments such as dev, QA, etc... but stopped working in higher environments. Below was the message.

"An existing connection was forcibly closed by the remote host ErrorCode: ConnectionReset (ServiceCommunicationProblem);"

As it is a higher environment, there is no way we can install troubleshooting tools or even don't have access to the Azure portal.

Troubleshooting

Since there is no portal access and cannot install tools we decided to simulate how the application consumes Service Bus using PowerShell.

The first step was easy to connect to Azure using cmdlets using the Service Principal+Certificate and generate JWT. But the next step was not easy.

By the way, this requires the installation of Az.Accounts PowerShell module in the production machine. :)

Since PowerShell is considered an admin thing than a dev tool, admins normally allow it.

Challenge

The problem is that there is no official PowerShell package from Microsoft to work with the data place of ServiceBus. The official PowerShell module supports control plane functions such as create service bus namespace(New-ServiceBusNamespace), create queues (New-AzServiceBusQueue), etc. Even interacting with the data plane is not there in Azure CLI. We were not able to find any community-built PowerShell cmdlets too except for some tutorials on how to send messages.  

Solution

Finally, we had to get our hands dirty by making HTTP calls from PowerShell. The REST endpoints are properly documented to easily craft the requests. The PowerShell source code to send messages is available on GitHub.

https://github.com/ms-azure-demos/servicebus-queues-powershell

The root cause of the problem

When we run the PowerShell script from the same machine the app is running, it worked!!!. That eliminated the questions of what ports are open etc. The remaining difference between the .Net application sending messages v/s PowerShell was the AMQP v/s simple HTTP. .Net application uses AMQP by default and we didn't put an option in the application to fall back to HTTP. We checked whether the ports are open and they are. Hence concluded the higher environment network is not allowing AMQP traffic. Some network devices are on the way that doesn't like the AMQP traffic. 

Azure @ Enterprise - PowerShell log in as service principal + certificate and generate JWT access token

The enterprise always loves to increase the security posture. One authentication approach enterprise takes in Azure is App registration with a service principal. The advantage here is that the service principal can use certificates to authenticate instead of passwords. Certificates are secure than passwords as those can be centrally managed.

Problem

If our application uses the service principal + certificate and it is working fine, there is no issue. But the problem starts when something goes wrong. If we log in to the portal using our credentials and try scenarios that are failing, we may see everything works fine. But things go wrong when the application logs in using service principal. It may be a permission issue, expired certificates or passwords, etc...

What if the problems appeared first in production where enterprises don't allow any changes to the environment. ie no debugging tools are allowed to install etc...

Solution

The solution is to troubleshoot the scenario as close as how the application works. We have to log in as the service principal and try the application scenarios. 

As always in enterprise the best method to troubleshoot in production environments is PowerShell. Below goes the code to log into Azure from PowerShell using service principal and generating a JWT access token.

The Connect-AzAccount cmdlet provides different ways to log in. Using the service principal is one of the methods.

Please note that the Az.Accounts need a minimum of Windows PowerShell 5.1 or PowerShell 7 version.

Once the PowerShell session is authenticated, we can perform various operations as the service principal. 

In most cases, the access token is not really required. There are PowerShell SDKs for most of the Azure services. We can directly use them. But in some cases like interacting with the data plane of Service Bus, we may need to use the access token and embed it in the Authorization header of the HTTP request.

Limitations

It reads the certificate from the personal store unless loaded from a file. We cannot pass an X509Certificate object to the Connect-AzAccount cmdlet. There is already an issue in GitHub to track it.

Update : 2021-10-30

What if we don't have permission to install Az.Accounts module?

Recently, I came to one production debugging situation where there is no permission to install the Az.Accounts module and even no connectivity to the internet to get the module. The only way is to write everything ourselves. Fortunately, someone had already done that and it is available publicly. If interested read the official docs.

Close all overdue Azure DevOps work items with Python

As software engineers, we need to track our work items even if we are working as freelancers or for an enterprise. Those work items constitute the health of the project also helping the management to understand the high-level view. Every company has its own ALM software to track the work. One of the famous ALM is Azure DevOps (ADO hereafter) from Microsoft.

Problem

In an ideal situation, we should plan our tasks for every day. Good if we can plan in advance and close the work items as soon as it's done. But sometimes we may not be able to make the ADO work items sync with reality. It may be due to long release days. Unexpected support issues, shortage of team members, etc. At the end of the billing cycle finance department will be chasing us to close work items. At that time closing each and every item manually is time-consuming. There are many ways to tackle it such as import to Excel close all and sync. We may use any no-code low-code platforms such as Power Automate to automate the task. Developers who use those platforms normally called 'citizen developers'. I strongly recommend trying any of these methods to automate this type of work. There is no need to code.

The no-code, low-code platforms generally offer building blocks to do simple day-to-day operations then provide extensibility via plug-ins or by invoking web requests / Web APIs. The plug-ins require coding efforts. Invoking web requests needs deep knowledge of Web APIs and how the service is structured.

Potential solution

As developers, we are proud of our programming ability and may not want to become citizen developers. For those, there are SDKs available to automate Azure DevOps work items. Also when the no-code, low-code platform doesn't have required customizations, we have to use the SDKs or make direct Web API calls to Azure Dev Ops.

We can use the Python SDK given by Azure DevOps as one of the mediums to interact with Azure Dev Ops to close all the OverDue work items.

Code is available on GitHub.

How to use is already documented in the readme.md files. Please follow that.

Code walkthrough

The Git Repo has enough comments. I am also planning a YouTube video explaining the same.