Linux tips - Running shell scripts in parallel and wait

Continuing the journey to improve my shell scripting skills from beginner to intermediate. 

Problem

I was working to improve the performance of my NAS daily sync mechanism. Currently, it uses rsync commands at the folder level. There are around 5 folders to sync right now. It issues the rsync commands one by one starting at 1 AM and outputs the results to a log file. After completing all rsync commands it sends a summary email. Obviously, it takes time. The easy quick fix is to parallelize.

Once all the rsync commands are complete in parallel execute the script to send the summary email

Running shell commands in parallel

If it was PowerShell we need to use Jobs or the ForEach-Object -Parallel iteration. But in shell, it is as easy as combining commands with &. Below is the code to demo the concept using a simple number-counting scenario.

Code to count from the numbers passed into the script

When it runs, it outputs the numbers to numbers.txt.

Below is the code to invoke the counter.sh by passing start and end numbers

This will be adding 2 additional lines to the numbers.txt file

The magic is done by the ampersand & symbol at the end of each command. & executes in the background

Executing & results

When we run without the & we can see the numbers.txt will have numbers in order. But with & the numbers will be saved out of order

Also, note the "End counting" line is saved only after all the numbers as it waits for the parallel invocations to complete.

References

• https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/start-job?view=powershell-7.2
• https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/
• 
  

How to automate the setup of Win 10 development machine by scripting?

Problem

As developers, many of us get a new upgraded machine every 3 years. Sometimes we may get earlier due to project change, machine complaints, or change of company. Every time we get a new machine we are not only getting a new machine but a boring task as well. 

Some software would already be present in the machine, some may not. Also, the versions may be different than what we need.

More than the software, there are some settings also we would like to be in or machine. eg: Mouse pointer speed, showing the extension of known files. etc...

Provided we have the admin privileges in the dev box, it is very difficult to convince managers that we need 6-8 hours to set up a new machine. Forget about developers who don't have admin privilege in their machines and raising tickets to get a change. They really should consider updating their resume.  

Below are some settings and software I prefer to be in my dev box 

Environment

These depend on person to person.

• Show all files, extensions, protected OS files, the full path in the title bar
• Increase the mouse speed
• Disable sleep on AC Power
• Remove preinstalled UWP apps- Windows maps, Microsoft Tips

Software inventory

Again this list also changes per developer depends on what technology they are working.

• Git
• Chrome
• PowerShell 7
• Node.JS
• VS 2019
• Java
• GraphViz
• Fiddler
• Notepad++
• VS Code
• VS Code Extensions
• LogParser 2.2
• PowerBI desktop
• Firefox
• Remote Desktop Manager Free
• Postman
• Telerik Just decompile
• UiPath studio
• VLC Player
• OBS Studio

Approaches

Below are some approaches other than the standard answer dockerize your environment. Docker helps to standardize the application runtime. Oh yes, we can install VSCode in docker and access remotely. Another option is to use GitHub Codespaces when they become available. The containerized option may not work for all the developers. Especially if they want to work on legacy technologies. Hence considering the below options

Developer machine image+preinstalled tools

This is what the first idea comes to us especially if we are from old age. This was a revolution one time. But nowadays people don't feel this as modern. Why this is not a good idea?

• No idea what will be in the image.
• No version control.
• Every update has to go via the infrastructure team or someone who reimage machines
• Has to reimage the existing machine when a new tool is introduced also after a version upgrade.

Winget+PowerShell

Microsoft recently introduced a new package manager for Windows. That space was already occupied by other package managers such as Chocolatey. WinGet has a public repository of package manifests. It is really in its infancy. So better wait for how it evolves than just jumping on it.

Obviously, PowerShell is required to change configurations that the package manager don't provide such as changing power options,

Choco+PowerShell

Choco is the short form for Chocolatey mentioned above. This package manager is there for some time. No idea why Microsoft didn't acquire them and set as the default package manager for Windows. Along with Boxstarter, Chocolatey+PowerShell is a good team.

Ansible+PowerShell

Ansible is an agentless configuration management platform to simplify deployment. It is mainly on Linux but supports Windows as well. As per their Windows strategy, it uses PowerShell to get things executed. Also, it can integrate with Chocolatey to install packages. Even it can execute any custom PowerShell scripts. More than all these it supports YAML which feels more declarative and modern.

Solution

Choco+PowerShell with Boxstarter seems the easiest approach to start with. But definitely, Ansible is promising. 

The next decision we need to take is whether we should fork an existing GitHub repo or start our own. I reviewed some repositories but finally decided to start my own by taking ideas from them. The rationale is that most of the repositories are personal. Moreover, every developer will have their own preferences.

 

Below is my repository. It currently has Choco+PowerShell approach to install and set up a dev environment on a fresh Windows machine. It assumes the required rights on the machine.

win10-dev-box-setup

It starts with installing Chocolatey itself. The Boxstarter provides a web-based install by a click of URL.

Since the list of software evolves over a period of time, I am not listing what are the capabilities added in this post. Please refer the repo

If statement in SQLCMD

SQLCMD is a nice utility to automate the SQL scripts execution.I became the fan of it years ago when I first used it for automating database creation and version upgrade of databases. The past sql automations using sqlcmd.exe were relatively simple. Most of those were using tokens and those tokens will be replaced with actual values when the execution starts. The values for tokens will be supplied by initiating programs and those programs may be of .bat, .vbs or .ps1 files at the most it will be invoked from the installers.

Recently we came to a situation where a branching is required in SQLCMD scripting. Basically we need to change the value of the token at runtime based no a database SELECT query. At that moment I realized that sqlcmd doesn't have branching statements. Team's initial decision to mitigate it was to call the database from the initiator programs.But those will make the .bat files really heavy or may need to write another PowerShell script or exe to do the job and get the value for sqlcmd token.

But a tight google exercise unveiled the below link which explains how to imitate the branching in sqlcmd. Its nothing but creating dynamic .sql file based on query result and executing the same from its creator sql file. I don't think I need to replicate the details again. So just open the link and have fun.

http://blogs.msdn.com/b/sql_server_appendix_z/archive/2013/02/23/sql-server-management-studio-sqlcmd-meets-the-richter-scale.aspx

Passing .Net objects as arguements to PowerShell

In the previous posts about PowerShell to .Net integration, the samples were simply invoking PowerShell scripts from .Net code. In the first post, the sample don't have any parameter at all. It just executed a get-alias command from .Net code. The second which I wrote in relation with DSL, passes the arguments as parameters to a dynamic block which uses '&' for script block execution. The below code snippet is just for  a reference regarding dynamic script block execution.

---

         private  static  void  RunPowerShellScriptDynamicallyWithParameter(int  inputNumber)

         {

             PowerShell  ps = PowerShell .Create();

             //The format needed for PS to execute dynamic scipr block is 

             //&{param ($paramsAsCommaSeparated);statements;} inputParameters as string 

             string  script = "&{param ($number);if (($number % 2)-eq 0)'}} " +inputNumber.ToString();

             ps.AddScript(script);

             ps.AddParameter("number" , inputNumber);

             Collection <PSObject > result = ps.Invoke();

             //Process logic for resultant PSObjects. 

         }

---

C#/VB.Net objects to PowerShell

There was a small limitation that we can append only string variables / primary datatypes as it uses string concatenation for passing the arguments.To achieve actual object parameter passing in a simple manner you need to define the parameters at the beginning of the script block you want to execute. Then from the .Net code add those parameters.Here is a small code block which validates an .Net FixedDeposit object in powershell code.

---

         FixedDeposit  fdAccount = new  FixedDeposit () { Customer = new  Customer  { Name = "George" , Age = 70 }, Interest = 9.0 };

         private  void  RunPowerShellScriptWithParameter()

         {

             PowerShell  ps = PowerShell .Create();

             string  script = "param($fd); if ($fd.Customer.Age -ge 60) {$fd.Interest=$fd.Interest+.5}" ;

             ps.AddScript(script);

             ps.AddParameter("fd" , fdAccount);

             Collection <PSObject > result = ps.Invoke();

             //Inspect the fdAccount object for changes 

         }

---

In India all the banks give additional .5 interest to senior citisons (above 60) for their fixed deposits. If this rule comes via a script it can be modified at any time.PowerShell scripts can be also applied to the UI elements ie we can modify / control the .Net UI elements from PowerShell

DSL Demo : First C# application which uses PowerShell as DSL

As I promised in my previous post, this post is dedicated to fully cover a practical application of DSL by combining 2 of my earlier posts.

http://joymonscode.blogspot.in/2013/01/what-is-domain-specific-language-dsl-in.html
http://joymonscode.blogspot.in/2012/12/running-powershell-script-from-c-vbnet.html

The idea was to give a kick start sample for C# developers into the world of DSL. But before going into the details I would like to add couple of theory titles.

How to select the DSL (Domain Specific Language)

This is a big question before you select the implementation method of DSL. According to Martin Fowler there are 2 categories of DSL. External and Internal DSL.

Internal DSL means adjusting our existing programming language practice to look more like business friendly. ie using plain English constructs to express the business cases. Linq is a best example for that. Also people are using fluent api for achieving internal DSLs. But here there is a considerable limitation. It is little difficult to add support for runtime changes. You can create the framework as a DSL and the tracks ie other developers can use that to achieve their tasks.

External DSL employees same programming language or different language altogether along with its own execution mechanism to run DSL program. SQL is a good example for that. We write C# code and for communicating with DB we need to write entirely different program called SQL queries. The execution engine and the model of SQL execution itself different from C#.

According to me if you are planning to select a DSL to conquer the constant requirements change better go for external DSL. This allows the end users, ability to change the program or behavior at run time. There is another psychological factor. If we ask your users write in C# or any other main stream programming language, they will protest first. But if you ask them to write in javascript or PowerShell they will be OK.This is because according to the end users, the responsibility of writing main stream code always lies with programmers. Can't believe? 

This is proved last week in our office.We have an installation system which expects so many parameters to configure the application such as DB name, IIS application pool name etc...Earlier this has to be entered by the installation guy (QA people in case of testing) at the beginning of an .vbs file.That same vb script file contains the processing logic after the initial declaration section. They usually complain about this process of giving values in .vbs file. According to them its complicated. Now the process changed little bit. We leveraged MS Excel to enter parameter values and wrote vbs to read the values from there. They became very happy. The complication is washed out by introduction of excel. But did we do anything? Yes we removed a mental roadblock of modifying a program file by QA and other installation people. Still they need to enter DB name, application pool name etc...But at a different place.Now they are happy with the installation procedure.

Another thing to remember is, never go for creating your own parser or lexers to execute that external DSL. Of course if you have research time and lavish time you can go.But as an Architect / Technologist the best way is to leverage an existing well proven mechanism to achieve execution of DSL.

In the book "Wings of fire", former Indian president Dr. A.P.J Abdul Kalam is clearly defining the 'difference between science and technology'. According to him, if we are doing experiments to create something really new and we are not sure whether we can produce positive result, it is called as science. In the other hand, if we are doing something with limited resource and we must produce the positive output in time, its called technology.A scientist can go up to any limit to experiment what he wants. But technologist mostly works in constraints, so he needs to be aware of all the available proven options and use those wisely to achieve the goal.

Creating a new DSL and writing its parser and related things is a science. But leveraging the power of PowerShell, Javascript or any other language which can be embedded in our program to achieve the DSL execution is technology. Now you might have understood why the heading says PowerShell as DSL.

C# Salary calculation using PowerShell as DSL

In the previous post I had told about a salary calculation scenario where we can apply DSL to solve the requirement change issue. Basically requirement change is the nature of software engineering rather than  issue. I am not good in explaining theories so I managed to create a C# program which uses PowerShell to accomplish the salary calculation based on dynamic logic written in PowerShell.

Why PowerShell

There is no special interest on PowerShell. You can select any scripting language such as Javascript, Lua etc..Only constraint is the scripting language should support embedding into C# application or there should be some methods to execute the script from C# and communicate back and forth.

The reasons for selecting PowerShell are

• Its becoming more and more popular like .bat file
• Easy to write programs using the pipline support.
• Recently I took a PowerShell hands on training session to a selected audience in our company :)

About the sample

Few words about the embedded sample. Its a simple program to demonstrate how to visualize the working of DSL. This is because most of the C# / VB.Net programmers might have heard about DSL but didn't experienced. It uses simple PowerShell invocation method from C#. In the formula textbox you can enter any PowerShell script snippet which returns a double value. The values for Basic, HRA & DA are available to the PowerShell context as variables $basic, $hra & $da respectively. Try changing the PowerShell code to calculate total salary by considering a 5% basic as metro allowance.

Happy coding...

PowerShell to start Lync group conversation and send message

In previous blog post I had described how I wrote code for myself to achieve a personal task. As people says “Code for yourself before you code for others”, I took that opportunity to improve my life, before I go to solve problems and improve other’s life with my code. But did my problem solved with that small C# code?

I would say 50/50 .Its really difficult to add one more person on the go. I had to open the solution in VSTS then modify the list and run again. I can add users manually with less time than the time taken for Visual Studio to open. I love VS as developer.But as a user I hate as a user. Again keeping an app.config and editing it is equally time consuming like keeping a script which can also be edited on the fly. So what is best.

I would say script. More general PowerShell which has the capability to seamlessly interact with .net.In simple words its very easy to convert a C# / VB.Net code PS. Here goes the PS equivalent of what I posted last week in C#.

cls
$lyncDLLPath = “C:\Program Files\Microsoft Lync\SDK\Assemblies\Desktop\Microsoft.Lync.Model.DLL”
Import-Module $lyncDLLPath
$contactUris="sip:<email1>","sip:<email2>"
 
$client = [Microsoft.Lync.Model.LyncClient]::GetClient()

$conv = $client.ConversationManager.AddConversation()

foreach ($contactUri in $contactUris)
{
    $conv.AddParticipant($client.ContactManager.GetContactByUri($contactUri))
}
$m = $conv.Modalities[[Microsoft.Lync.Model.Conversation.ModalityTypes]::InstantMessage]
$m.BeginSendMessage("Hi guys this is Joy from Powershell,,,Shall we have lunch....", $null, $null);

Running PowerShell script from C# / VB.Net

Recently I had to take series of PowerShell sessions to one of the non-programmers team in my current organization. Actually its the build and installation team who wanted to automate their installation using PowerShell. It was a new experience to me as most of them are non programmers or haven't coded for years. So I started with very basics such as what is programming, conditional branching statements if ... else,switch case ,looping constructs such as while,do ... while, for, for each, functions, passing arguments to functions etc to creating and running dynamic scripts.Actually I would not have been chosen for teaching  basics of programming as I am working as Junior architect. By the by I got promoted as Junior Architect after the last performance review which is a rare designation with very less people around the world. Since the technology is not familiar to most of the programmers I had to be the trainer.

Whether I took session or not has nothing to do with this post. But there is a small relation. When I explained one of my .Net colleague about new PowerShell training to installation team and the power of PowerShell, he told me one statement. "Yes bringing PowerShell into installation is good so that, we can call that PowerShell code from our .Net code". I said "why we need to call PowerShell from .Net because PowerShell is from .Net and it uses .Net code to accomplish the tasks. Why should we go back again to C# .Net to call the script. We can execute the PS scripts directly like an exe". Then he told. "We are .Net devs and PS needs to be started from our code.What is stopping us?"

It was a partial joke .But I started thinking about it seriously .Will there be any benefits, if we call PowerShell from our code? It seems yes. We can leverage scripting to our application. But before that I need to confirm whether it is callable from C# / VB.Net code. Initial google gave me somewhat big code block which is   using RunSpace and all. But it seems we just need to create PowerShell object to execute simple scripts.

Adding reference to PowerShell assembly

You need to add reference to the below assembly to create the object of PowerShell class.

<InstallDrive>:\Program Files\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0\System.Management.Automation.dll

Invoke PowerShell script from C#

It is again a matter of calling the invoke method in PowerShell class. But be sure that you had added the script using the AddScript method available in the same class.

The RunPowerShellScript method

         private  static  IEnumerable  RunPowerShellScript(string  script)

         {

                 PowerShell  ps = PowerShell.Create();

                 ps.AddScript(script);

                 Collection <PSObject > result= ps.Invoke();

                 foreach  (PSObject  psObj in  result)

                 {

                     yield  return  (psObj);

                 }

         }

Simple isn't it.This uses the LocalRunspace to execute the script .ie the local machine where this C# .Net code is running

Calling RunPowerShellScript method

             foreach  (var  v in  RunPowerShellScript("get-alias" ))

                 Console .WriteLine(v);

I am looking for some practical scenarios where we can use this. Hopefully can find one soon.

Merging files

Some time back we faced a simple situation. Its nothing but we need to merge a good number of files into one file. It was not a one time process. We may need to do it when ever any of the files change because those files are residing in the TFS source control.

We wanted to do it simply .ie without writing a .Net program.Normally developers have a tendency to write program even for small things .The first method was to use the copy command.Yes the same DOS command copy. We were able to do that as follows till the files grew beyond MBs.

Copy *.* c:\combined.sql

When the individual files grew bigger ie to MBs this command started appending some unwanted characters into the out put file. This prevent us from executing the sql files using SQL Server Management Studio.

Obviously the next solution was powershell.We got a script to combine files with in no time.Its simple,open source to modify,no need to compile but when we look at that it’s almost the same .net classes and code except some syntax changes .

Uploaded the script here.I am not sure from which site I got this..

Welcome to SQLCMD

Last couple of weeks I was mainly working in the backend side.Its nothing but SQL server 2008.We had to develop a bunch of scripts which will help the production DBAs to accomplish their tasks with less effort.Since the production environment is unknown and will vary to country to country we cannot put dbo.<table name> in any of our scripts.So we decided to put some tokens in the scripts like #schema#,#DB# etc…the DBA needs to replace these tokens in his environment before running the scripts.
Things went fine till the scripts were reviewed by Microsoft consultant Dimitri Furman .He suggested to use SQLCMD variables.
What is this SQLCMD variables? Do we really need to use that? But after having a script file of 100MB we automatically preferred the sqlcmd mode because SQL Server Management Studio is not suitable for executing such a big sql file.It will definitely crash.

Reasons why we selected SQLCMD

• We have more files which needs to share the variables.Replacing in all the files is tedious.
• We have large files which cannot be executed in SSMS.

SQLCMD is the command prompt type interface to execute sql queries.You can start the shell using the executable located in the below location.

<drive>:\Program Files\Microsoft SQL Server\90\Tools\Binn\SQLCMD.EXE

In the first look there is no change.But when we come to authoring sql files there can be changes.Main advantage is you can have scripting variables which will replace their values at run time both in the command mode and in the sql files.You can compare this to classic ASP or PHP programming which replaces the server script before sending to the browser.

Ok Lets see what is this scripting variable by looking at an example.

:setvar PersonName Joy
:setvar email joymon@gmail.com
INSERT INTO Person ([Name] ,[Id] ,[EMail])
     VALUES('$(PersonName)', newId(), '$(email)')
GO

PersonName & email are variables and their current values are Joy & joymon@gmail.com respectively.When we refer them using the $() they will get replaced with the values.So here the values got replaced between the single quotes ‘’ and the query run smoothly.
Another advantage is we can call another script file from this using the :r keyword.See the example below.

r: "c:\queries\SQLQuery2.sql"

If we set a variable in one file and that file is calling another file the variables will be available in the second sql script also.If we took the above example,the variable $(PersonName) can be used in SQLQuery2.sql as well.There are lot many advantages, if you explore the possibilities of SQLCMD.Refer the below links for more details.

http://msdn.microsoft.com/en-us/library/ms188714.aspx

http://www.sqlbook.com/SQL-Server/SQLCMD-command-line-utility-13.aspx
Running the sql files with SQLCMD variables in SSMS
You can even run the sql files with SQLCMD variables in SSMS.For that change the query mode to SQLCMD.See the below screenshot.

Powershell to find version of .net assemblies

Las week we upgraded our application version from 2010 to 2011.This involved a lot of changes in the configuration files along with UI changes such as logos ,images etc...One of the change was in the assembly version and correction of config files to reflect the version. Most of our code is relying on .net reflection so the config files should be updated properly to get the fully qualified name of classes.

Till then everything was fine. We got problems when the build came. The build script failed to build the dlls in the new assembly version. The product have around 400 DLLs and we are not using GAC. How to find out which dll is in wrong version?

One solution is to put the assemblies into GAC and see the version.Another is to open the DLL in reflector and check the version. Since the number of DLLs is somewhat huge ,it is better to write a program to find out the assemblies which are in wrong version.The next question came C# or VB.Net?

But keeping mind the principle of architecture finally decided to write a small windows power shell to find out the version.Code below.

Windows Power shell code to find out the .Net assembly / .dll version

#--------Script to find out .net assembly version mismatches---------
$curDir=$("C:\Binaries")
$pattern=($curDir + $("\*.dll"))
Foreach ($file in Get-Childitem $pattern)
{
    $fp=($curDir+"\"+$file.name)
    
    #Load the assembly 
    $asm=[Reflection.Assembly]::LoadFrom($fp)
    $asmName=$asm.GetName()
    $ver=$asmName.Version
    
    #Create required version object to compare
    $reqVer=new-object Version -arg "4.0.0.0"
    
    #if the version is not matching show a warning
    if($ver -ne $reqVer) {
        Write-Warning $fp
        Write-Warning $ver.ToString(4)
    }
} 

Providing Solution v/s creating Architecture

You may feel that this is something similar to one of the post “Technology v/s concepts and architecture” in my WPF blog.This is just a story from which you can understand what I would like to convey by the post Solution v/s architecture.

Last week when I was almost alone in office ,one of the Infrastructure & network support guy called me and asked “Can you write a program for me?” Yes buddy of course that is why I am here.He explained me his need.He need to restart 4 windows services after installing one particular software in a considerable number of machines.If I give a program he can just run that to restart services in one machine immediately after he installs and he can go to another machine for installation.

Then the .Net architect in me stood up and said .I will write a C# program which is configurable to restart any number of services.He said “OK”.My next question was “Which format you prefer for the configuration? xml or csv?”.What?? he continued “I need a program to restart 4 services.Let me know when you finish”.Oh poor guy he don’t know what he wants.But it’s my duty to give him highly sophisticated program.

Fortunately or unfortunately I had to attend one production support issue.Since it was complex our lead architect had to involve in the issue on the day.It was a out of memory exception which was not reproducible in our dev machines which have 4GB RAM.So it was decided like client needs to upgrade their machine as the data pulled is very large.But our lead architect started tackling in another way.He started asking some basic questions such as what is the chunk size we used in WCF services,is it really OOME or exception raised due to LOH fragmentation etc… which gave a new perspective to the issue.After 15mins the diagnostics session ended and I came to the previous problem.

May be due to the new light I got from my lead,I started thinking about the requirement from the network guy? Do I really need to write C# program to restart 4 windows services?It needs a .Net runtime. If I target to .Net 4.0,can I assure him that it will work in all the machines?Now the answer comes ’no’.I should think about some thing simple.

When .Net seems over engineered the option normally goes to scripting.Yes that is enough.Then which scripting? .bat files or vb script which will read the config to restart services?Oh.Again I am complicating.There is one more scripting technique which is nothing but WindowsPowerShell.Seems I got an alternative solution.The next phase is to discuss with the client about the alternatives.I called him and asked one thing.”Are you going to use this tool in machines which have Windows power shell?” Yes he replied.OS is either Vista or Win7.

Yes.This is the solution.Create a RestartService.ps1 file and write the below 4 lines.Ask him to change the service name using notepad and run using Windows Powershell.

Restart-Service BthServ
Restart-Service BthServ
Restart-Service BthServ
Restart-Service BthServ

Architect for solution.Don’t solve for architecture.

Seems I taught somebody to do my job which may be a threat in future:)

For Loop in .Bat file

echo off
for %%v in (one two three) do (
echo %%v
)

This is simple usage of for loop in batch file.This can be extended to file reading and more.