Setting up home NAS - Part 7 - Automate using Ansible

This is the continuation of the "Setting up home NAS" series. Better read the below posts to get context.

Setting up home NAS - Part 0 - RAID v/s Backup rsync v/s Raspberry Pi v/s Cloud Storage for Youtuber in the year 2020

Setting up home NAS - Part 1 - Scientific debugging to get USB drive visible as \\share in network

Setting up home NAS - Part 2 - PowerShell arrange media date bases files into YYYY\MM\DD folders

Setting up home NAS - Part 3 - Dealing with duplicates

Setting up home NAS - Patt 4 - Raspberry Pi to build locally redundant NAS for home

Setting up home NAS - Part 5 - e-mailing rsync report

Setting up home NAS - Part 6 - Raspberry Pi as Media Server using Plex

Setting up home NAS - Part 7 - Automate using Ansible

Introduction

Previous posts have manual instructions to set up the Raspberry Pi to function as a NAS and Plex media server. As we all know manual steps are error-prone and difficult to replicate. This post is about automating the deployment and manual configuration steps.

Architecture

There are different approaches to automating the machine configuration and software deployment.

The easy way is to simply write scripts and get them executed in the target machine itself. Here target is Raspberry Pi and Ansible supports this method and lets us install it on Raspberry Pi.

Secondly, agent-based¹ where we first manually install an agent software that polls the central repository of configuration. In case there are any changes, it will apply those to the host machine where it is running.

Another way is the opposite of agent-based. A controller machine pushes the changes to the target machine using some remoting technology. Here the target is Raspberry Pi

Here we are using an agentless method using a tool called Red Hat Ansible². Ansible is a well-known tool in IT automation workflows.

SSH to Pi with keys 

Since there is no agent to listen in, the controller machine needs to use a way to remote into the target machine to be configured. Ansible uses SSH³ to do the same.

The default way is to run the Ansible scripts and it will ask for login details. For Raspberry to set up that is fine. But there are ways to avoid it by using the SSH public-private key pairs.

There are clearly documented ways to establish a passwordless login to RasPi⁴ using SSH keys. But there are chances of things going wrong and troubleshooting SSH keys to authenticate should be a separate post on its own.

Installing Ansible

It requires Python to be installed. The installation of Ansible is straightforward and well documented⁵.

Source code

Enough theory. Since it involves scripts that can be enhanced later, it is better to keep in a source control repository. Below is the source code repo link to GitHub.

https://github.com/joymon/ansible-raspberry-home-nas-plex

The readme.md is pretty much updated.

One thing pending is the setting up of the user password that is used to access the shared path from Raspberry.

It will be added soon and updates will be pushed to the same repo. 

Ansible in Windows v/s Linux

One drawback is that Ansible is not very compatible with Windows as a controller. So we have to use the WSL2 feature of Windows to manage the target Raspberry Pi.

Reference

• ¹ - https://dzone.com/articles/part-ii-choosing-the-right-infrastructure-as-code
• ² - https://www.ansible.com/
• ³ - https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html
• ⁴ - https://www.raspberrypi.com/documentation/computers/remote-access.html#secure-shell-from-windows-10
• ⁵ - https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
• ⁶ - https://docs.ansible.com/ansible/latest/os_guide/windows_faq.html#can-ansible-run-on-windows

PowerShell to list all the IIS Web Applications using IISAdministration

When it comes to automating Internet Information Services (IIS) Web server there are multiple choices. One with Appcmd command and the other with PowerShell. In PowerShell itself, there are different modules to manage. The WebAdministration is the legacy and IISAdministration is modern. More on that was written in a previous post about "Change recycle settings of IIS AppPools using PowerShell & appcmd.exe"

Let us get into the business of this post

Problem

There is an existing system that uses the on-premises model for hosting web applications. Windows Server 2019 + IIS is the hosting model. As the first step to the cloud, these servers were lifted and shifted to the Azure. Now to reduce the cost and time to market of course to improve the developer agility the IaaS model needs to be upgraded to containers. AKS is the preferred choice as it is in Azure. As part of the analysis and capacity planning the # of containers needs to be evaluated from the existing IIS-based deployment model.  ie count the web applications in IIS.

So this post is about listing all the web applications inside IIS web server that are spread across different IIS websites.

Solution

Since the last post about IIS automation was using WebAdministration, I thought of going with IISAdministration this time as it's the latest to replace the former.

Hope the code is self-explanatory. If the above embedding is not working please check the GitHub file.

Enjoy scripting...

Generate pdf from markdown files in a folder using markdown-pdf

I am a big fan of markdown language for writing documents. One of the scenarios we encounter using markdown is to generate a pdf file or convert the markdown files to some other format. This post is to introduce an NPM package that converts markdown files to pdf file. It is considerably old but still does the job.

markdown-pdf

markdown-pdf npm library is open source software with MIT license. They have just enough documentation as well. It can be used as a command and as a library to use from our nodejs applications.

The limitations I found are as follows.

There is no direct API to convert an entire folder of markdown (.md) files. We have to iterate the folder by ourselves and give it all the file paths. Another problem is to insert page breaks between the markdown files. But that also can be overcome by using pre-processing markdown interceptor and some css tricks.

Sample code

Enough talking. The sample code to perform the below use cases with this library is available in GitHub repo.

• How a folder can be given to markdown-pdf as there is no direct API
• How to preprocess markdown files
• How to give a custom CSS file

https://github.com/nodejs-demos/markdown-pdf-demos

How it works

Internally it converts the markdown to HTML by using the remarkable npm library. Once the HTML files are available, it uses the phantomjs to load the HTML in the browser then print as pdf from there. It supports custom CSS files as well. 

Now it might be clear how the line breaks are inserted into the PDF files.

Use cases

If our documentation is prepared using markdown, the pdf file can be generated using this library through the CI/CD pipelines. 

Other libraries

• md-to-pdf - This uses Marked library to convert .md files to HTML. Then Puppeteer to print pdf. It also supports code highlighting via highlight.js library.
• There is a node-pandoc library that tells, it converts to pdf using pandoc. This method is not tried.

References

• https://stackoverflow.com/questions/57278886/insert-page-break-when-using-markdown-pdf-nodejs-module
• http://forums.apricitysoftware.com/t/include-pdf-pagebreak-instructions-in-markdown/152/5

How to automate the setup of Win 10 development machine by scripting?

Problem

As developers, many of us get a new upgraded machine every 3 years. Sometimes we may get earlier due to project change, machine complaints, or change of company. Every time we get a new machine we are not only getting a new machine but a boring task as well. 

Some software would already be present in the machine, some may not. Also, the versions may be different than what we need.

More than the software, there are some settings also we would like to be in or machine. eg: Mouse pointer speed, showing the extension of known files. etc...

Provided we have the admin privileges in the dev box, it is very difficult to convince managers that we need 6-8 hours to set up a new machine. Forget about developers who don't have admin privilege in their machines and raising tickets to get a change. They really should consider updating their resume.  

Below are some settings and software I prefer to be in my dev box 

Environment

These depend on person to person.

• Show all files, extensions, protected OS files, the full path in the title bar
• Increase the mouse speed
• Disable sleep on AC Power
• Remove preinstalled UWP apps- Windows maps, Microsoft Tips

Software inventory

Again this list also changes per developer depends on what technology they are working.

• Git
• Chrome
• PowerShell 7
• Node.JS
• VS 2019
• Java
• GraphViz
• Fiddler
• Notepad++
• VS Code
• VS Code Extensions
• LogParser 2.2
• PowerBI desktop
• Firefox
• Remote Desktop Manager Free
• Postman
• Telerik Just decompile
• UiPath studio
• VLC Player
• OBS Studio

Approaches

Below are some approaches other than the standard answer dockerize your environment. Docker helps to standardize the application runtime. Oh yes, we can install VSCode in docker and access remotely. Another option is to use GitHub Codespaces when they become available. The containerized option may not work for all the developers. Especially if they want to work on legacy technologies. Hence considering the below options

Developer machine image+preinstalled tools

This is what the first idea comes to us especially if we are from old age. This was a revolution one time. But nowadays people don't feel this as modern. Why this is not a good idea?

• No idea what will be in the image.
• No version control.
• Every update has to go via the infrastructure team or someone who reimage machines
• Has to reimage the existing machine when a new tool is introduced also after a version upgrade.

Winget+PowerShell

Microsoft recently introduced a new package manager for Windows. That space was already occupied by other package managers such as Chocolatey. WinGet has a public repository of package manifests. It is really in its infancy. So better wait for how it evolves than just jumping on it.

Obviously, PowerShell is required to change configurations that the package manager don't provide such as changing power options,

Choco+PowerShell

Choco is the short form for Chocolatey mentioned above. This package manager is there for some time. No idea why Microsoft didn't acquire them and set as the default package manager for Windows. Along with Boxstarter, Chocolatey+PowerShell is a good team.

Ansible+PowerShell

Ansible is an agentless configuration management platform to simplify deployment. It is mainly on Linux but supports Windows as well. As per their Windows strategy, it uses PowerShell to get things executed. Also, it can integrate with Chocolatey to install packages. Even it can execute any custom PowerShell scripts. More than all these it supports YAML which feels more declarative and modern.

Solution

Choco+PowerShell with Boxstarter seems the easiest approach to start with. But definitely, Ansible is promising. 

The next decision we need to take is whether we should fork an existing GitHub repo or start our own. I reviewed some repositories but finally decided to start my own by taking ideas from them. The rationale is that most of the repositories are personal. Moreover, every developer will have their own preferences.

 

Below is my repository. It currently has Choco+PowerShell approach to install and set up a dev environment on a fresh Windows machine. It assumes the required rights on the machine.

win10-dev-box-setup

It starts with installing Chocolatey itself. The Boxstarter provides a web-based install by a click of URL.

Since the list of software evolves over a period of time, I am not listing what are the capabilities added in this post. Please refer the repo

Change recycle settings of IIS AppPools using PowerShell & appcmd.exe

Background

The app pool recycle is a hack to hide our or .Net's poor memory management. When we refer poor .Net memory management, it means the LOH fragmentation which was not compacted in earlier versions of .Net CLR. Hence we need to set up the app pool recycle in IIS.

Changing or 'recycling' setting of an app pool in one server one time is easy and can be manual. But what if there are many app pools, servers and had to do multiple times. Obviously, it should be automated. This post is about some snippets about automating the same using some of the provided mechanisms.

AppCmd

This is a command which we can use to manipulate IIS programmatically from IIS 7 onwards. Location of appcmd.exe is [install drive]:\Windows\System32\inetsrv\appcmd.exe. Direct commands to manipulate IIS entities are very easy but chaining the commands is a little different. If we are from PowerShell background, we feel it very odd. It outputs XML and subsequent commands to read XML as input.

Below is a mix of PowerShell and AppCmd.exe to disable periodic recycling of app pools.

---

Import-Module WebAdministration
$pools= dir IIS:\AppPools
$pools | Foreach-Object{
    $appPoolName=$_.name
    $cmd="C:\Windows\System32\inetsrv\appcmd.exe set apppool /apppool.name: '$appPoolName' /-recycling.periodicRestart.schedule"
    Invoke-expression $cmd
}

---

As clear in the above snippet, the PowerShell iterates thought the AppPool and invoke the AppCmd.exe for each app pool with its name

Another drawback of AppCmd.exe is that we cannot manage remote machines using this command only. We need to combine with PowerShell to execute this command in the remote machine.

PowerShell

There are 2 PowerShell modules.

WebAdministration

• This mainly gives a provider which we can query just like file system
• Microsoft considers WebAdministration PowerShell module legacy and not bringing to Nano server. Clearly means someday WebAdministration will be discontinued from support.

IISAdministration

• The successor of the WebAdministration module and claims it's better than WebAdministration
• Comes default with IIS in Windows 10 and Windows 2016

PowerShell Function to set Recycling

Though WebAdministration is considered legacy, this post uses that to achieve the job of managing AppPool recycling.

---

function Set-ApplicationPoolRecycleTimes {     
    param ([string]$ApplicationPoolName, [string[]]$RestartTimes)
    Import-Module WebAdministration       
    Clear-ItemProperty IIS:\AppPools\$ApplicationPoolName -Name Recycling.periodicRestart.schedule       
    foreach( $restartTime in $RestartTimes)
    {
        New-ItemProperty -Path "IIS:\AppPools\$ApplicationPoolName" -Name Recycling.periodicRestart.schedule -Value @{value=$restartTime}           
    }
}

---

It has 2 parameters. First it clears the existing settings and then applies the new settings.Below is the method to call function to clear recycling. Just skip the second parameter $RestartTimes to clear the property.

---

Set-ApplicationPoolRecycleTimes -ApplicationPoolName 'DefaultAppPool'

---

The code below shows how to call the function to set new times. The parameters can be given just comma-separated or as and string array.

---

Set-ApplicationPoolRecycleTimes -ApplicationPoolName "DefaultAppPool" -RestartTimes "1:00","2:00","14:00"

---

Please note that the times expected are in 24-hr clock format. If we use the AM or PM what the IIS manager shows, this script will error with cast exception.

Let's see how we can manage all the app pools in a machine.

Applying to all the pools

Just loop all the app pools and call the function.

---

$pools= dir IIS:\AppPools
#$pools | ForEach-Object { Set-ApplicationPoolRecycleTimes($_.name) }

---

PowerShell has Invoke-Command to run scripts in remote machines. The same method can be used to control IIS in remote machines.

References

https://www.habaneroconsulting.com/stories/insights/2013/set-the-specific-times-to-recycle-an-application-pool-with-powershell
https://forums.iis.net/t/1229302.aspx