Linux tips - Running shell scripts in parallel and wait

Continuing the journey to improve my shell scripting skills from beginner to intermediate. 

Problem

I was working to improve the performance of my NAS daily sync mechanism. Currently, it uses rsync commands at the folder level. There are around 5 folders to sync right now. It issues the rsync commands one by one starting at 1 AM and outputs the results to a log file. After completing all rsync commands it sends a summary email. Obviously, it takes time. The easy quick fix is to parallelize.

Once all the rsync commands are complete in parallel execute the script to send the summary email

Running shell commands in parallel

If it was PowerShell we need to use Jobs or the ForEach-Object -Parallel iteration. But in shell, it is as easy as combining commands with &. Below is the code to demo the concept using a simple number-counting scenario.

Code to count from the numbers passed into the script

When it runs, it outputs the numbers to numbers.txt.

Below is the code to invoke the counter.sh by passing start and end numbers

This will be adding 2 additional lines to the numbers.txt file

The magic is done by the ampersand & symbol at the end of each command. & executes in the background

Executing & results

When we run without the & we can see the numbers.txt will have numbers in order. But with & the numbers will be saved out of order

Also, note the "End counting" line is saved only after all the numbers as it waits for the parallel invocations to complete.

References

• https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/start-job?view=powershell-7.2
• https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/
• 
  

pip install packages where multiple python versions installed

This is mainly targeted to developers like me who are coming to Python from other technologies. For seasoned Python developers managing packages from multiple versions would be muscle memory.

The main concern of this video is to address how we can manage the packages in a machine having multiple python versions installed.

Azure @ Enterprise - App Service Deployment Error: ENOENT: no such file or directory, open

I was fiddling with deployment via the Kudo console in the Azure app service. Removed one file as part of testing. The expectation was that the next deployment would put the file back.

The next deployment from GitHub actions failed with the below message.

33      Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/5b7989fd-9794-4ed0-8bfc-dbf1f18c686b.zip (4.31 MB) to /tmp/zipdeploy/extracted

34

Error: Failed to deploy web package to App Service.

35

Error: Deployment Failed, Package deployment using ZIP Deploy failed. Refer logs for more details.

36

App Service Application URL: https://<app service web app name>.azurewebsites.net

test

No more details on failure in GH Actions logs. There was an error as shown below in the Azure AppService Deployment center.

Error: ENOENT: no such file or directory, open '/home/site/wwwroot/Azure.Core.dll

That was the exact file I removed earlier.

Debugging

Hypothesis 1

The file may be open in a Kudu console, hence unable to access it. 

But even after closing, the error persisted.

Hypothesis 2

The App Service deployment action in GHActions requires the file to be present there as that may be keeping track of files.

Even after manually uploading the file, the issue persists

Solution

Finally attempted the Microsoft-certified solution of Restarting the App service. It worked.

The order is to stop the App service, do the deployment, and start again.