Linux tips - Running shell scripts in parallel and wait

Continuing the journey to improve my shell scripting skills from beginner to intermediate. 

Problem

I was working to improve the performance of my NAS daily sync mechanism. Currently, it uses rsync commands at the folder level. There are around 5 folders to sync right now. It issues the rsync commands one by one starting at 1 AM and outputs the results to a log file. After completing all rsync commands it sends a summary email. Obviously, it takes time. The easy quick fix is to parallelize.

Once all the rsync commands are complete in parallel execute the script to send the summary email

Running shell commands in parallel

If it was PowerShell we need to use Jobs or the ForEach-Object -Parallel iteration. But in shell, it is as easy as combining commands with &. Below is the code to demo the concept using a simple number-counting scenario.

Code to count from the numbers passed into the script

When it runs, it outputs the numbers to numbers.txt.

Below is the code to invoke the counter.sh by passing start and end numbers

This will be adding 2 additional lines to the numbers.txt file

The magic is done by the ampersand & symbol at the end of each command. & executes in the background

Executing & results

When we run without the & we can see the numbers.txt will have numbers in order. But with & the numbers will be saved out of order

Also, note the "End counting" line is saved only after all the numbers as it waits for the parallel invocations to complete.

References

• https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/start-job?view=powershell-7.2
• https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/
•